When you type «nvfp4 vs» into Google it completes with «fp8», «gguf», «q4_k_m» and «bf16». The usual answer compares different models quantized by different people. On October 16 NVIDIA released Nemotron Labs Dory 1.0 4B in BF16, NVFP4, MXFP8 and GGUF at once, so I measured the same model in every format on an RTX 4070 Ti SUPER (Ada, 16 GB, no FP4 hardware), with the rules written down before the first run.
| Format | Engine | Generate (tok/s, batch 1) | Read 2048-token prompt (tok/s) | Weights |
|---|---|---|---|---|
| BF16 | vLLM | 51.0 | 7,003 | 7.69 GiB |
| FP8 (per tensor, W8A8) | vLLM | 78.5 | 10,118 | 4.67 GiB |
| MXFP8 | vLLM | 81.4 | 6,061 | 4.76 GiB |
| NVFP4 | vLLM | 116.8 | 6,978 | 3.28 GiB |
| GGUF Q5_K_M | llama.cpp | 58.8 | 1,710 | 3.42 GiB (file) |
- Among the vLLM formats, generation follows weight size. NVFP4 was 2.3 times faster than BF16. On Ada it runs through a weight-only Marlin kernel: the weights stay small, the math stays in 16 bits.
- Reading a long prompt, FP8 wins, because it is the only one of these formats whose multiplications Ada does in 8 bits.
- Quality, full GSM8K (1319 questions, 8192-token cap): BF16 1249, FP8 1241, MXFP8 1250, NVFP4 1238. All three tie with BF16 under the ±3-point rule; their 95 % intervals stay within ±2 points.
- Token by token it is another story. On fixed English and Spanish texts NVFP4 picks BF16's next token 89-90 % of the time, against 94-96 % for FP8 and MXFP8, while perplexity barely moves. MXFP8 is the most faithful to BF16.
The post also covers what broke before measuring (the llama.cpp converter rejects the public checkpoint, a prefix cache that inflated prefill, truncated reasoning being scored as answers) and a mistake the audit caught in my own first fidelity run.
Read the full measurements, the pre-registered sheet and the scripts → https://efraingaray.com/en/blog/nvfp4-vs-fp8-gguf-ada/
Top comments (6)
the 2.3x on NVFP4 is the Marlin kernel doing all the heavy lifting, and that's the part that doesn't transfer. on my box (4x 3090, 96 GB) i run models where the weights are the small part and the KV cache is the whole game, so the prompt-read column is what i'd actually look at first. FP8 winning that by a wide margin (10k vs 7k) is the only place where Ada's 8-bit math shows up, and that's a real advantage that survives on any model size, not just 4B.
question: did you try NVFP4 on a card without Marlin support, or is that kernel the only reason it's ahead? if the 4-bit path falls back to dequant-to-16-bit on non-Marlin hardware, the whole 2.3x probably collapses.
Thanks for the careful read, this is a fair question. To be clear on scope, the post says NVFP4 runs here through vLLM's weight-only Marlin kernel, and the load log confirms it (
Using MarlinNvFp4LinearKernel for NVFP4 GEMM). It's not a tuned setup; it's the path vLLM picks by default on this card. From vLLM's own check (is_fp4_marlin_supported, compute capability 7.5 and up), your 3090s should take that same Marlin path, and Blackwell runs FP4 natively, so the dequant-to-16-bit fallback you describe shouldn't come up on the cards vLLM supports. I only measured one Ada card, though, and the 2.3x is decode at batch 1, where speed tracks weight size (3.28 vs 7.69 GiB). With the KV cache dominating, as on your box, the gap may well shrink. On prefill we agree: FP8 leads at 10,118 vs 6,978 tok/s.If I get the time this weekend, I'll rerun NVFP4 with the Marlin kernel disabled to see how much of the 2.3x is the kernel, and publish it as a follow-up to this post. I'll let you know here if I do.
did the NVFP4 runs ever hit the 16 GB ceiling, and if so how did the scheduler handle swapping?
It never ran out of memory, by design: vLLM reserves memory up front (
gpu_memory_utilization0.85, 13.24 GiB) and gives whatever the weights don't use to the KV cache. With NVFP4 the weights took 3.28 GiB, which left 8.61 GiB of KV cache: 40,232 tokens, or 4.6 full-length requests at the 8,704-token limit. BF16 got 4.31 GiB (20,128 tokens, 2.3 requests). There's no swapping to CPU in this vLLM build: when KV blocks run out, the scheduler preempts a running request, frees its blocks and recomputes it later. I didn't log preemptions, so I can't tell you how often that happened in the 1,319-question run. The speed numbers in the post are at batch 1, where none of this comes into play.nice, that reservation strategy really squeezes the kv cache out of the leftover space, did you notice any latency impact when hitting the 4‑request ceiling?
I didn't measure latency under load; the speeds in the post are batch 1. But the ~4.6 ceiling is the worst case, every request at the full 8,704 tokens. In the GSM8K run, all 1,319 questions went in with up to 128 in flight, and vLLM logged zero preemptions in all four formats: the short requests never reached the ceiling. If they had, vLLM would preempt and recompute, and that's where latency would jump.