smalinin/Step-5-Preview-GGUF

Step-5-Preview

Q3_K_M with restored sparse-attention tensors

This release combines the Q3_K_M text weights from TypeSafeAI/Step-5-Preview-GGUF with the missing sparse-attention tensors from NeuroSenko/Step-5-Preview-Int4-g128. The original quantized weight payloads are preserved; the restored tensors retain their BF16/F32 types.

Use the experimental smalinin/llama.cpp#my_step5 branch described below. This Step-5 runtime was developed and validated only in a CUDA GPU configuration, including CPU offload. Other GPU backends and CPU-only execution of the complete model have not been validated.

Full CSA/Sparse GQA is not implemented. Long-context inference uses an experimental dense fallback on the full-attention layers. Successful local tests cover prompts up to approximately 60K tokens with a 64,000-token context allocation. They do not establish equivalence to the original model or support for its advertised 1M-token context.

Model background

The TypeSafeAI BF16 model card describes Step-5-Preview as a StepFun MoE model with approximately 600B total parameters and 27B active parameters per token, aimed at coding, reasoning and agent workflows. It advertises text, image and video inputs, configurable reasoning effort, parallel tool calling, structured output and a 1M-token context using Sparse GQA with block-wise token merging. These are source-model claims; the experimental runtime capabilities are listed separately below.

The checkpoint configuration specifies:

Component	Configuration
Main text layers	92
Hidden size	4096
Routed experts / selected experts	352 / 8 per token
Attention heads / KV heads	64 / 4
Attention head dimension	192
Attention pattern	23 full-attention layers and 69 sliding-window layers
Sliding window	512 tokens
Rotary dimensions	64 for full attention; 192 for sliding attention
Embedded MTP layers	3, following the main text layers

The GGUF architecture identifier remains step35; the experimental branch handles the Step-5-specific configuration and tensors.

How the GGUF was assembled

1. Start with the seven Q3_K_M GGUF parts from TypeSafeAI/Step-5-Preview-GGUF, containing 1,599 tensors. Keep their tensor descriptors, offsets and weight payloads unchanged.
2. Identify the missing sparse_indexer_* and ssmax_s tensors on the 23 full-attention layers: eight tensors per layer, 184 tensors in total.
3. Extract only those tensors from NeuroSenko's safetensors shards using HTTP byte-range requests. The donor revision used was fbc1415911951ad94736c1720a97a607829adb7c. Downloading the donor's quantized MoE expert weights was unnecessary.
4. Map the extracted tensor names into GGUF and write them into an eighth part, preserving the original numerical types and payload bytes. In particular, the indexer normalization tensors are copied directly, without applying the Step35 normalization offset.
5. Update only split.count and split.tensors.count in the original seven headers, and rename the parts for the new eight-part set. No original tensor payload is requantized or replaced.
6. Verify restored tensor names, shapes, types and SHA-256 payload hashes after the safetensors-to-GGUF round trip. Check the resulting set for missing and duplicate tensors.

NeuroSenko quantizes its routed MoE experts to INT4 but leaves its sparse indexer in BF16/F32, as documented in its model card. The added tensors were already BF16/F32; no INT4-to-Q3_K_M conversion was performed.

For each full-attention layer N (3, 7, ..., 91), the restored tensors are:

Source suffix under model.layers.N.self_attn.	GGUF tensor	Type
sparse_indexer_q.weight	blk.N.indexer.q_proj.weight	BF16
sparse_indexer_k.weight	blk.N.indexer.k_proj.weight	BF16
sparse_indexer_z.weight	blk.N.indexer.z_proj.weight	BF16
sparse_indexer_w.weight	blk.N.indexer.proj.weight	F32
sparse_indexer_q_norm.weight	blk.N.indexer.q_norm.weight	F32
sparse_indexer_k_norm.weight	blk.N.indexer.k_norm.weight	F32
sparse_indexer_k_norm.bias	blk.N.indexer.k_norm.bias	F32
ssmax_s	blk.N.attn_ssmax_s	F32

The restored payload is 874,326,784 bytes, approximately 833.8 MiB: 69 BF16 tensors and 115 F32 tensors. All eight restored tensors in the first full-attention layer, layer 3, were also compared against TypeSafeAI's BF16 checkpoint at revision e7746ba676c6a0391fce0d54d0a4f392d293dc07 and matched exactly. This additional BF16-source comparison covers that layer, not all layers.

The resulting text model has eight GGUF parts and 1,783 tensors. It preserves all 1,782 mapped HF text/MTP tensors plus one generated RoPE tensor. The vision encoder is separate and requires an mmproj GGUF.

The eighth part, containing the restored tensors, has this verified SHA-256 checksum:

1abc449307ade5c39a40d19b78c8ad05c70a5f1f318cdef5c0edf9515fe8e6d4  Step5-Preview-Q3_K_M.gguf-00008-of-00008.gguf

This is a repaired version of the existing Q3_K_M quantization. A new quantization from the full BF16 checkpoint with importance-matrix calibration has not been performed.

Files and memory requirements

Keep all eight parts together in one directory:

Step5-Preview-Q3_K_M.gguf-00001-of-00008.gguf
Step5-Preview-Q3_K_M.gguf-00002-of-00008.gguf
Step5-Preview-Q3_K_M.gguf-00003-of-00008.gguf
Step5-Preview-Q3_K_M.gguf-00004-of-00008.gguf
Step5-Preview-Q3_K_M.gguf-00005-of-00008.gguf
Step5-Preview-Q3_K_M.gguf-00006-of-00008.gguf
Step5-Preview-Q3_K_M.gguf-00007-of-00008.gguf
Step5-Preview-Q3_K_M.gguf-00008-of-00008.gguf

Pass the first part to -m; llama.cpp discovers the others automatically. The model files total approximately 286.85 GB / 267.15 GiB. Allow additional disk space for optional vision weights and other files.

Use the complete restored eight-part release. The original seven-part TypeSafeAI set lacks the required tensors and is rejected by this experimental runtime. No --override-kv split-metadata workaround is needed for the restored release.

This is a large model even though only a subset of its experts is active for each token. Available VRAM and system RAM must accommodate the loaded weights, KV cache and compute buffers. Auto-fit chooses placement; it does not eliminate the memory requirements.

The performance tests used an Intel Core Ultra 7 265K and six NVIDIA GPUs with approximately 236 GiB aggregate VRAM, with some weights on CPU. Memory needs and performance depend on GPU placement, context length and microbatch size.

Experimental runtime support

The instructions below correspond to tested commit 95a265ab9 on my_step5.

Feature	Status
Text generation and chat	Implemented and tested with the restored Q3_K_M model
Step-5 attention and RoPE parameters	Per-layer rotary dimensions and frequency bases handled
Attention scaling	Restored ssmax_s applied per query head on full-attention layers
Experimental long context	Dense full attention plus SWA 512; requires explicit opt-in beyond 512 tokens
Full CSA / block compression / provider-group selection	Not implemented
Embedded MTP	All three heads supported through llama-server; no separate draft sidecar needed
Adaptive MTP draft length	Enabled by default; adjusts from 1 to 3 tokens with the example settings
Prefix cache with MTP	Repeated prompts and changed suffixes tested
Auto-fit	Target and MTP contexts fitted together to available device memory
CUDA Flash Attention	Added native support for 192-dimensional K/V heads
CPU offload optimizations	Q3_K/Q4_K MoE panel kernels on VNNI-capable CPUs, with dynamic scheduling
Tool calling	Template/parser support, including parallel calls; complete model-to-tool round trip with MTP has not been tested
Images	Experimental separate projector path; two synthetic color/shape tests passed with the vision encoder on CPU and the language model on CUDA
Video, real-image OCR, GPU vision encoder	Not validated for this release

Restoring indexer tensors makes the checkpoint available for future CSA work. The dense mode currently uses ssmax_s and does not execute the indexer selection/compression algorithm. It has not been compared layer by layer against an official CSA implementation.

The branch also contains an opt-in DSA-based sparse experiment. It failed three local long-context factual checks. Leave LLAMA_STEP5_EXPERIMENTAL_SPARSE unset or set it to 0; the examples below explicitly disable it.

Build with CUDA

The validated environment is Linux with NVIDIA CUDA. Install Git, CMake, a C/C++ build toolchain, a compatible NVIDIA driver and CUDA Toolkit with nvcc. See the branch's build documentation for platform prerequisites.

git clone --branch my_step5 --single-branch \
  https://github.com/smalinin/llama.cpp.git llama-step5
cd llama-step5

# Pin the version used for the validation described in this card.
git checkout 95a265ab9e01c3a0c2b730d6e0fd7bc275c52324

cmake -S . -B build \
  -DCMAKE_BUILD_TYPE=Release \
  -DGGML_CUDA=ON \
  -DGGML_CUDA_FA=ON \
  -DGGML_NATIVE=ON

cmake --build build --config Release -j 8 --target llama-server

GGML_NATIVE=ON builds for the local CPU. Build on the inference machine when possible. For Ampere/Ada GPUs, the validated build used -DCMAKE_CUDA_ARCHITECTURES="86;89"; add this to the CMake command only if those architectures match your GPUs. The default CMake selection is preferable for a different GPU set.

Confirm the server binary and its supported options:

./build/bin/llama-server --help

Download the restored model

From the llama-step5 directory, install the Hugging Face CLI in a virtual environment and download all eight parts:

python3 -m venv .venv-hf
source .venv-hf/bin/activate
python -m pip install -U huggingface_hub

hf download smalinin/Step-5-Preview-GGUF \
  --include '*Q3_K_M*of-00008.gguf' \
  --local-dir ./models/Step-5-Preview-GGUF

Run the OpenAI-compatible server

The following example starts text chat with an 8K context, CUDA Flash Attention, auto-fit and embedded MTP. Run it from the llama-step5 directory after downloading the model.

export CUDA_DEVICE_ORDER=PCI_BUS_ID
export LLAMA_STEP5_EXPERIMENTAL_DENSE=1
export LLAMA_STEP5_EXPERIMENTAL_SPARSE=0

STEP5_MODEL='./models/Step-5-Preview-GGUF/Step5-Preview-Q3_K_M.gguf-00001-of-00008.gguf'
STEP5_CTX=8192

./build/bin/llama-server \
  -m "$STEP5_MODEL" --alias step5 \
  -c "$STEP5_CTX" -b 512 -ub 256 \
  --fit on --fit-target 1024 --fit-ctx "$STEP5_CTX" \
  --load-mode none \
  -t 20 -tb 20 --no-op-offload -fa on \
  --no-context-shift --no-warmup --no-mmproj \
  --parallel 1 --host 127.0.0.1 --port 8085 \
  --jinja --chat-template-kwargs '{"reasoning_effort":"medium"}' \
  --temp 0.7 --top-p 0.95 \
  --spec-type draft-mtp --spec-draft-n-max 3

Wait for the server to finish loading. The web UI is available at http://127.0.0.1:8085; the API base URL is http://127.0.0.1:8085/v1.

Important settings:

• LLAMA_STEP5_EXPERIMENTAL_DENSE=1 opts into the approximate long-context mode. Without it, the restored model's dense mode is restricted to contexts of at most 512 tokens.
• --fit on selects GPU/CPU placement. --fit-target 1024 leaves a target margin of 1,024 MiB per GPU; --fit-ctx prevents auto-fit from reducing the context below the requested size. A configuration still needs enough memory to fit.
• --load-mode none eagerly loads the CPU-resident weights, reducing first-request page-fault delays seen in local mmap tests. It does not reduce weight memory.
• -t 20 -tb 20 matches the tested CPU. Adjust the thread counts for your machine.
• --spec-draft-n-max 3 sets a maximum, not a fixed draft length. MTP adaptation is enabled unless LLAMA_MTP_ADAPTIVE=0 is set. To run without MTP, replace the two speculation options with --spec-type none.
• Use --jinja with the embedded chat template. Reasoning effort is passed as a template setting; the example uses medium.
• --no-context-shift means the prompt plus generated output must fit within the selected context.

Experimental 64K context

For the settings used in the long-context performance tests, stop the previous server and run:

export CUDA_DEVICE_ORDER=PCI_BUS_ID
export LLAMA_STEP5_EXPERIMENTAL_DENSE=1
export LLAMA_STEP5_EXPERIMENTAL_SPARSE=0

STEP5_MODEL='./models/Step-5-Preview-GGUF/Step5-Preview-Q3_K_M.gguf-00001-of-00008.gguf'
STEP5_CTX=64000

./build/bin/llama-server \
  -m "$STEP5_MODEL" --alias step5 \
  -c "$STEP5_CTX" -b 2048 -ub 1024 \
  --fit on --fit-target 1024 --fit-ctx "$STEP5_CTX" \
  --load-mode none \
  -t 20 -tb 20 --no-op-offload -fa on \
  --no-context-shift --no-warmup --no-mmproj \
  --parallel 1 --host 127.0.0.1 --port 8085 \
  --jinja --chat-template-kwargs '{"reasoning_effort":"medium"}' \
  --temp 0.7 --top-p 0.95 \
  --spec-type draft-mtp --spec-draft-n-max 3

A longer context or larger microbatch changes KV-cache and compute-buffer requirements and may move more weights to CPU. Increasing the microbatch does not necessarily improve performance; a previous local comparison was slower at 2048 than at 1024.

Make a chat request

curl http://127.0.0.1:8085/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "step5",
    "messages": [
      {"role": "user", "content": "Explain why binary search takes O(log n) time."}
    ],
    "temperature": 0.7,
    "top_p": 0.95,
    "max_tokens": 512,
    "stream": false
  }'

For tools, send OpenAI-style tools definitions. The server formats the prompt and parses model-generated tool calls; the client must execute each tool and send its result back in a role: "tool" message. Parser/template support has been checked, but real end-to-end tool execution with this checkpoint and MTP remains to be tested. Source-model claims about strict JSON-schema reliability have not been independently validated here.

Optional images

Images require a separate projector such as mmproj-Step5-Preview-bf16.gguf and the runtime's image input path. Once you have that file locally, replace --no-mmproj in the server command with:

--mmproj ./models/Step-5-Preview-GGUF/mmproj-Step5-Preview-bf16.gguf \
--no-mmproj-offload

The projector adds approximately 4.40 GB on disk. Local image checks used the vision encoder on CPU and the language model on CUDA, recognizing a red square and a blue circle. MTP is suspended for image-containing requests and resumes for subsequent text-only requests. GPU vision, real-image OCR and video are not validated.

Validation and measured performance

Local validation included tensor mapping and checksum checks, RoPE checks, CUDA attention tests, CPU kernel tests, real text generation, embedded MTP, prefix-cache reuse and synthetic long-context retrieval. The latest Q3_K/Q4_K CPU changes passed 417 CPU backend tests. These checks do not constitute a full quality evaluation against BF16 or an official CSA runtime.

With the tested CUDA/CPU placement, a 64,000-token context and MTP enabled:

Local test	Result
31,028-token prompt after Q3_K/Q4_K optimizations	Prefill: 170.63 s, 181.84 tokens/s; generation: 18.40 tokens/s
Eight record/value pairs in a 15,503-token prompt	All eight pairs retrieved correctly
60,068-token prompt, before the latest Q4_K optimization	Both target facts retrieved; generation approximately 17.18 tokens/s
Prefix reuse at approximately 60K tokens	Repeated prompt and changed suffix passed

The 60K and cache measurements preceded the latest Q4_K optimization and have not yet been repeated at the pinned latest commit. Performance depends on routing, prompt content, MTP acceptance and hardware. The numbers above are measurements on one system, not expected throughput for every configuration.

Limitations and next steps

• Full CSA needs reference code or a sufficiently precise specification for indexer scoring, token/block merging, selection, normalization and cache behavior. Further CSA work is deferred until that information becomes available.
• Dense long-context inference is approximate. The tested synthetic retrieval tasks do not establish general long-context quality, official runtime equivalence or 1M-context support.
• Restored sparse tensors are preserved for future implementation, but tensor presence alone does not implement CSA.
• This release inherits the original Q3_K_M quantization. A full BF16-to-GGUF conversion with new calibration, broader quality evaluation and a new quantization remains future work.
• The tested complete inference configuration uses CUDA. ROCm, Vulkan, Metal, SYCL and CPU-only complete-model inference are unvalidated.
• Tool-use quality, real-image understanding, OCR and video need additional testing.

Sample

Q3_K_M

• prompt

create html with svg "pelican on a bicycle"

• used opencode, toolcall, vision
• prompt

Indian Railways WDP-4D dual-cab diesel locomotive using Three.js

This is an illustrative existing sample, not a benchmark of CSA correctness or multimodal support.

Attribution and license

• StepFun: original Step-5 model.
• TypeSafeAI/Step-5-Preview-BF16: source model information and BF16 reference checks.
• TypeSafeAI/Step-5-Preview-GGUF: original Q3_K_M weight payloads.
• NeuroSenko/Step-5-Preview-Int4-g128: preserved BF16/F32 sparse-attention tensors used for restoration.
• ggml-org/llama.cpp: underlying inference engine.
• smalinin/llama.cpp, my_step5: experimental Step-5 runtime and optimizations.

The source card identifies the weights' license as the StepFun Community License. Refer to its LICENSE for the terms governing these derived weights. The runtime has its own llama.cpp license.