ROCmFPX for llama.cpp

ROCmFPX adds experimental AMD-focused 2-, 3-, 4-, 6-, and 8-bit GGUF model-weight formats to llama.cpp, with CPU reference paths and accelerated HIP/ROCm and Vulkan kernels.

Status: ROCmFPX is an experimental feature family on the canonical main branch. APIs, tuning choices, and performance can change. Results depend on hardware, drivers, model, prompt, and quantization recipe; use BF16/F16 sources for quality comparisons.

Disclosure

#AD #AMDAI

A huge thank you to AMD for providing the hardware that powers the ongoing development of ROCmFPX.

The development system provided by AMD includes:

• AMD Ryzen™ Threadripper™ 9980X
• AMD Radeon™ AI PRO R9700 GPU
• 128 GB ECC DDR5 Memory
• 2 TB NVMe SSD

This hardware enables me to continue developing, optimizing, benchmarking, and testing ROCmFPX for the community. AMD provided this hardware as part of a creator partnership. All development, benchmarks, code, testing, documentation, and opinions shared in this repository are my own.

Thank you, AMD, for supporting the continued advancement of open AI development on Radeon hardware.

#AD #AMDAI

Developer Update — ROCmFP2 Lands On Main (July 2026)

ROCmFP2 is now available on the canonical main branch as Q2_0_ROCMFPX. It uses a 2.50-bpw block layout with an S40 {-4, -1, +1, +4} codebook and dual UE4M3 scales. The smaller blocks reduce model storage and memory traffic compared with ROCmFP4. The 2.50-bpw figure describes native ROCmFP2 weight blocks; complete GGUF BPW can be higher because files also contain metadata and tensors stored in other types. The latest Vulkan dense and routed/MoE Q8_1 decode kernels landed through PR #42.

Download the current main source (ZIP) or use the clone command in Quick Start. Older tags and previously built binaries do not receive these updates automatically.

Fresh matched Qwen3.6-35B-A3B tests used the same prompt and 256-token, non-speculative decode workload from an internal NVMe:

ROCmFP2 also passed exhaustive packed-codebook validation and dense/routed Vulkan kernel gates. These measurements demonstrate the benefit on the tested system, not a universal speed or quality guarantee. ROCmFP2 is more lossy than ROCmFP4, so compare important workloads against the BF16/F16 source.

Why ROCmFPX?

• AMD-first weight formats: ROCmFP2, ROCmFP3, ROCmFP4, ROCmFP6, and ROCmFP8 are real GGUF model-weight quants, not just K/V-cache compression.
• Native accelerated paths: HIP/ROCm and Vulkan kernels are backed by CPU reference implementations for correctness testing.
• Speed and size choices: ROCmFP2 is the smallest optimized family, while ROCmFP4 is the speed-first 4-bit family; existing Qwen comparisons put ROCmFP4 files about 12% below the matched Q4_K_M size.
• Agent-aware presets: coherent/agent recipes protect tensors that matter for code, JSON, tool calling, and structured output.
• Built-in MTP acceleration: models with an MTP/NextN head—including M-RoPE Qwen models—can use target-verified self-speculative decoding without loading a separate draft model.
• Broad validation: the promoted source was exercised through local CPU/Vulkan/ROCm tests and cross-platform CI covering Windows, macOS/Metal, WebUI provisioning, and Apple packaging.

Start with Quick Start, choose a format in Which Format Should I Pick?, or jump directly to MTP Speculative Decoding.

Verified MTP Results — Strix Halo, 2026-07-12

These are local command-line decode results from the promoted main source on Strix Halo (gfx1151). Throughput is the final Generation: rate from llama-cli; runs used full GPU offload, FlashAttention, -c 4096, greedy sampling (--temp 0), -b 512 -ub 512, the same prompt within each row, and one model at a time.

Qwable is a matched single 64-token pair. Qwen no-MTP is one 256-token run; the MTP values are the median and peak of three matched 256-token runs.

The promoted source and the pre-promotion experimental build were effectively tied on Qwen3.6: median differences were -0.7% on Vulkan and -0.3% on ROCm. On a longer 512-token Vulkan run, the promoted source reached 110.7 t/s versus 107.2 t/s for the experimental build. Qwable's 256-token branch comparison was also tied: promoted/experimental measured 32.9/33.0 t/s on Vulkan and 32.3/32.3 t/s on ROCm.

MTP gains are content-dependent: predictable code, JSON, and lists usually accept more draft tokens than creative prose. Treat the profiles above as tested starting points, not universal defaults.

Experimental ROCmI4 IU4 Acceleration (gfx1151)

ROCmFPX includes an opt-in HIP/ROCm path for Q4_0_ROCMI4 models on AMD Strix Halo. It keeps ROCmI4 weights packed as signed four-bit values and uses the native v_wmma_i32_16x16x16_iu4 instruction for batched matrix multiplication. The feature is intended for prompt processing and MTP target verification; ordinary one-token-at-a-time generation continues to use MMVQ.

The path is deliberately conservative:

• GGML_HIP_ROCMI4_W4A4 defaults to OFF.
• Device code is compiled only under the exact __gfx1151__ target.
• Runtime dispatch also requires the exact gfx1151 device identifier.
• Other GPUs and default builds use the existing exact int8 MMQ path.
• No backend-test tolerance is weakened when W4A4 is enabled.

Architecture and terminology

These names describe different layers of the system:

W4A4 means four-bit weights and four-bit activations during the accelerated matrix multiply. The ROCmI4 weights themselves are unchanged; the additional loss comes from quantizing float activations directly to a signed four-bit grid.

Why MTP decode becomes faster

Non-speculative decode evaluates one new token at a time and is mostly memory-bandwidth bound, so the W4A4 MMQ path does not materially change a plain tg128 result. MTP changes the workload:

1. The model's embedded NextN/MTP head proposes up to several future tokens.
2. The target model verifies those proposed tokens together in a batch.
3. Batched verification uses MMQ and therefore reaches the native IU4 path.
4. Accepted draft tokens become output tokens without separate serial target evaluations.

On the development Ryzen AI MAX+ 395 / Radeon 8060S (gfx1151) system, a matched 10-task Qwen3.8-27B HumanEval pilot with strict MTP-16 measured:

The W4A4 run improved mean end-to-end decode by 18.66%. A separate full 164-task run measured 44.39 tok/s mean and 45.23 tok/s median, with HumanEval pass@1 94.5% and HumanEval+ pass@1 91.5%. A 25-chunk perplexity sample was about 5.4% higher than the exact path, so W4A4 remains an explicit quality/ speed tradeoff rather than a default.

Build the opt-in path

Use a separate build directory so the exact build remains available:

cmake -S . -B build-rocmI4-w4a4 \
  -DGGML_HIP=ON \
  -DGGML_VULKAN=OFF \
  -DCMAKE_HIP_ARCHITECTURES=gfx1151 \
  -DGGML_HIP_ROCMI4_W4A4=ON \
  -DLLAMA_BUILD_SERVER=ON
cmake --build build-rocmI4-w4a4 --target llama-cli llama-server llama-bench -j 16

ROCm installations that do not identify Strix Halo natively may also need:

export HSA_OVERRIDE_GFX_VERSION=11.5.1

The server prints ROCmI4 W4A4: enabled when the accelerated path is active. If the compiled binary runs on an unsupported architecture, it reports the fallback and uses exact int8 MMQ.

Fast strict-MTP server profile

For a Qwen model that contains its embedded MTP/NextN tensors:

build-rocmI4-w4a4/bin/llama-server \
  -m model-Q4_0_ROCMI4.gguf \
  --host 127.0.0.1 --port 8116 \
  -dev ROCm0 -ngl 999 -np 1 -c 262144 \
  -b 512 -ub 256 -t 16 -tb 32 -fa on \
  -ctk f16 -ctv f16 --jinja \
  --spec-type draft-mtp --spec-mtp-strict-qwen \
  --spec-draft-device ROCm0 --spec-draft-ngl all \
  --spec-draft-type-k f16 --spec-draft-type-v f16 \
  --spec-draft-n-max 16 --spec-draft-n-min 0 \
  --spec-draft-p-min 0.60 --spec-draft-backend-sampling

MTP acceptance and speed depend on content. Start with n16 / p0.60 only for the qualified Qwen3.8 profile; reduce --spec-draft-n-max if another model or long-running workload shows lower acceptance or stability. Strict MTP preserves the selected target path's greedy decisions, but it does not remove the approximation introduced by W4A4 activation quantization.