One level down: #002 the same compiler on a GPU

The previous article ended on a question. Herbert writes its machine code at startup, for the model and for the processor in front of it. Does that principle hold on a GPU, against the reference GPU engines?

On an RTX 4080, with Qwen3.5-0.8B and one request at a time, Herbert gives its first token in 6.5 ms on a short prompt, where each engine's fixed costs dominate. vLLM takes 19, llama.cpp in CUDA 12. On the 4-billion-parameter model, against llama.cpp at its best settings, it is a tie. And on the Arc GPU of an Intel laptop, in f16 against f16, Herbert is clearly slower at prefill.

All three results are true. For each one, I give the card, the version, and the setting.

One clarification before going further: no recent AMD graphics card is in this bench. On the AMD side, I only measured the integrated GPUs of two processors, a Ryzen 7 5700G and a Ryzen 9 9900X. I have ordered a Radeon RX 9060 XT with 16 GB, an RDNA 4 GPU: it will get its own article, which will also be the opportunity to compare against AMD's libraries.

In short

The essentials in four lines. The rest of the article explains how I measured it and in which cases Herbert does worse.

• First token on an RTX 4080, 0.8B, short prompt, one request at a time, where fixed costs dominate: 6.5 ms for Herbert, 12 for llama.cpp CUDA, 19 for vLLM, which is built to serve many requests at once. At 4,000 tokens, the gap with vLLM disappears; on the 4B, vLLM answers 3.7% sooner.
• Prefill of the 4B on the same card, in f16: a tie with llama.cpp CUDA at its best setting.
• On the Arc 140V of an Intel laptop, in f16: Herbert is slower at prefill, down to ×0.57. At decode, it stays faster than llama.cpp on every card.
• What gives the prefill lead on medium and long prompts: mostly cutting the prompt into large blocks, measured by turning them off. On a very short prompt, these settings account for only 15% of the time. The fusions are worth only 2% at decode.

After the first article, three readers picked up on the same thing, on X and on LinkedIn: what interested them were the cases unfavorable to Herbert, written into the text with their figures. One of them expected the Vulkan article to bring the references, the platforms, and the failures. I give them here. When I say an engine is ahead or behind, I am describing a measurement. I am not trying to win a race: a case where Herbert is slower shows me where to look to optimize it.

Emitting for a GPU

On the processor, Herbert writes x86 machine code. On the GPU, it writes SPIR-V, the intermediate language of Vulkan. One module per kernel, at startup, for the shape of the model and for the card in front of it. The subgroup size, the presence of cooperative matrices, native bf16 or not: it reads that on the card and writes it into the kernel. No vendor library, no shader compiled in advance, no external compiler.

Why Vulkan rather than a vendor's own language? Because Vulkan speaks to at least the three big ones, NVIDIA, Intel, and AMD, with the same binary and the same emitter. CUDA and ROCm each speak to a single vendor. OpenCL speaks to all of them, but I find Vulkan better implemented by the drivers and better able to make use of several kinds of hardware.

As in the previous article, an option writes to disk what the engine emitted. I ran it on a machine with two GPUs, an RTX 4080 and the AMD integrated GPU of its Ryzen 9 9900X, with two sizes of the same model.

target/release/herbert-vk-chat ~/models/Qwen3.5-0.8B --device NVIDIA --dump-spv spv/NVIDIA-0.8B
target/release/herbert-vk-chat ~/models/Qwen3.5-0.8B --device RADV --dump-spv spv/RADV-0.8B
target/release/herbert-vk-chat ~/models/Qwen3.5-4B --device NVIDIA --dump-spv spv/NVIDIA-4B

Take the projection that computes both halves of the feed-forward layer in one go, gate and up, at decode. On the 0.8B, it goes from 1,024 values to 7,168. Disassembled with spirv-dis, the module is 194 lines. Here are the only ones that change from one GPU to the other. On the RTX 4080:

     %uint_4 = OpConstant %uint 4
    %uint_32 = OpConstant %uint 32
         %36 = OpIMul %uint %31 %uint_4
        %125 = OpIAdd %uint %124 %uint_32

On the integrated GPU of the Ryzen 9 9900X:

     %uint_2 = OpConstant %uint 2
    %uint_64 = OpConstant %uint 64
         %36 = OpIMul %uint %31 %uint_2
        %125 = OpIAdd %uint %124 %uint_64

The 32 and the 64 are the subgroup size on each card, hence the loop stride; the 4 and the 2 follow from them, 128 invocations divided by 32 or by 64. From one model to the other, on the same 4080, it is the dimensions that change: 7,168, 128, and 7,167 on the 0.8B become 18,432, 320, and 18,431 on the 4B, the number of output rows, the length of a row of weights in 16-byte blocks (eight bf16 values), and the last index so as not to overrun. All of them are module constants: the kernel reads no bound while it runs.

The rest differs more. On the 4080, Herbert emits 38 kernels. The prefill matrix multiplications go through the card's cooperative matrices. At decode, two pairs of operations are fused into a single kernel: the normalization with the projection that follows it, the activation with the projection that follows it. On the integrated GPU, whose Vulkan driver does not expose cooperative matrices, it emits 32, with matrix multiplications written differently and without those fusions. One token step of the 0.8B fits in 192 operations on the 4080, in 265 on the integrated GPU. The difference is those two fusions: when I turn them off on the 4080, the step goes back up to 265 operations.

The numerical contract

The contract is the one from the previous article: weights, activations, and cache in bf16, accumulators in f32. The first article also set a condition: outputs identical token by token. On a GPU, it does not hold. The GPU follows the processor reference, then at some point it picks another word: depending on the prompt, at token 0, at token 8, or at token 172. On the 4080, 3 outputs out of 20 stay identical to the end.

I looked at each first divergence, recording the five best candidates on each side. The two words in the balance are one or two bf16 quanta apart, the smallest difference the format can write at that magnitude. Seven times out of seventeen, the GPU even sees them as a perfect tie. The GPU is not making a mistake: it does not add in the same order, so it does not round the same way. And it is stable: after an NVIDIA driver update, the divergences fell on exactly the same tokens. Same thing on an RTX 3060 Ti, under Windows. It is this contract that makes the figures comparable from one machine to another, not identical outputs.

The bench

Five GPUs, the same Herbert sources. An RTX 4080 with 16 GB under Linux. An RTX 3060 Ti with 8 GB under Windows 11. The Arc 140V GPU of a Core Ultra 7 258V laptop. The integrated GPUs of a Ryzen 7 5700G and of a Ryzen 9 9900X. Facing them, the versions of 12 October 2026: vLLM 0.28.0; llama.cpp in CUDA and in Vulkan, built at the same commit, 08ffba21e, checked on the binaries: the one from my first measurements was a build from 17 June, 8 to 16% slower at prefill than the right one. vLLM only runs on the 4080. The model is Qwen3.5, in 0.8B and 4B; the 4B does not fit in the 3060 Ti, which runs the 2B instead.

On the 4080, the clocks are locked, at 2,805 MHz for compute and 11,201 MHz for memory. On 7 October, the 4B seemed to have a weak spot on short prompts, 31% slower than vLLM at 128 tokens. Same binary, same prompt: a process ran either in 21 ms or in 34. It was the memory clock, which did not always ramp up, while NVIDIA's tool displayed the right frequency. Clocks locked with nvidia-smi, eight fast out of eight: the weak spot was gone. One caveat remains: under CUDA, the 4080 still lowers its memory clock to 10,801 MHz, 3.6% less bandwidth, whereas under Vulkan it stays at 11,201. I do not understand why yet, and I intend to study it.

The model is published in bf16. But for llama.cpp, bf16 is a secondary path: on the 4080, in Vulkan, its bf16 prefill is 2.3 to 3.6 times slower than in f16. Herbert in bf16 against llama.cpp in f16 would not be comparing the same thing: the two engines would not be counting the same way. So I wrote an f16 version of Herbert. Each comparison is made at equal format: bf16 against bf16, then f16 against f16, the format where llama.cpp is fastest. Both formats fit in 16 bits but do not split them the same way: bf16 keeps the exponent of a 32-bit float on 8 bits and 7 bits of mantissa, f16 gives 5 bits to the exponent and 10 to the mantissa. Not every card has good bf16 compute, whereas f16 is computed natively on every card of this bench. The trade-off: the outputs of the two versions cannot be compared token by token.

One request at a time, like someone sitting in front of their machine. vLLM is built to serve many requests at the same time: with one request, we are not measuring what it is built for. That is a boundary condition for everything that follows.

Two instruments. Each engine's own bench, llama-bench, vLLM's latency bench, Herbert's counter, tells the story of the computation, but does not measure the same thing from one engine to another: llama-bench counts the prefill alone, vLLM the whole request. The second instrument is a single client for all the engines, each run as a server, with the same API, the same prompts, and no prefix cache. It measures what the user sees: the time to first token, then the decode throughput. For this, Herbert got a compatible server, ported from the one in herbert-rs.

Decode: memory decides

At decode, each token rereads all the weights. From the very first version, the matrix-vector product emitted by Herbert read the weights at 590 to 690 GB/s on the 4080, against 694 measured by a pure-read probe. That is almost the maximum. As on the processor, the time left to win was elsewhere, in the recurrence of the Gated DeltaNet, the attention, and the round trip with the host.

Both instruments say the same thing. The format changes almost nothing there: in bf16 as in f16, each engine rereads the same number of bytes. Against llama.cpp, Herbert is faster everywhere. On the 4080, by 4 to 7% on the bench, by 5 to 22% on the common client. On the 3060 Ti, by 25 to 30% on the 0.8B, by 4 to 6% on the 2B. On the Arc and the integrated GPUs, by 1 to 20%.

Against vLLM, it depends on the model. On the 4B, they are tied: 1 to 3% in Herbert's favor, less than the advantage the memory clock gives it under Vulkan. On the 0.8B, it is faster as long as the context stays under about 500 tokens, then slower: 96% of vLLM around 1,000 tokens of context, 97% at 2,000, 94% at 4,000. Herbert's decode slows down faster than vLLM's as the context grows.

Prefill on the 4080

Prefill is computation. That is where an engine that writes its own code can show something.

In bf16 against bf16, on the bench, from 128 to 4,000 tokens, Herbert goes from 1.26 to 1.78 times llama.cpp CUDA on the 0.8B, from 1.09 to 1.24 times on the 4B; against llama.cpp Vulkan, from 3.3 to 5 times. These figures are correct, but they catch llama.cpp where it is at its worst. In f16 against f16, the gap narrows. It narrows further when llama.cpp is tuned. I swept, case by case, the batch size, the micro-batch size, and flash attention, each setting in its own process, five rounds, Herbert in the same rounds. llama.cpp CUDA gains up to 19%. I do not guarantee these are the best possible settings: they are the best this sweep found. If you know a better one, send it to me: I will measure it on the same card, in the same rounds.

On the 0.8B, Herbert stays one and a half times faster. On the 4B, it is a tie against llama.cpp CUDA: 1% faster over 759 tokens, 2% slower over 2,880.

vLLM is only measured in bf16, its reference format for this model. On the bench, on long prompts, Herbert in bf16 does 108 to 113% of vLLM on the 0.8B; on the 4B, they are tied within measurement noise, from 99 to 106% depending on the length. On short prompts, the bench puts Herbert at almost three times vLLM, at 128 tokens on the 0.8B. One might suspect the stopwatch, since vLLM counts the whole request. That is where the common client settles it.

On the 0.8B, Herbert's first token arrives earlier up to 500 tokens, then at the same time as vLLM's: 86.5 ms against 87.2 at 4,106 tokens. On the 4B, it arrives 7 to 15% earlier up to 500 tokens. Beyond that, vLLM's arrives 2.6 to 3.7% earlier. Personally, I like that: it shows me where I can still gain. I just wish days were longer than 24 hours.

In f16 against f16, with the same client, the result against llama.cpp CUDA does not change: 6.3 ms against 12.1 on a short prompt with the 0.8B, 22.8 against 38.4 on the 4B. The client also shows the gap between the bench and what the user sees: in bf16, served llama.cpp CUDA, on the 0.8B at 4,106 tokens, processes about 25,700 tokens per second, against 32,782 on its own bench at 4,000 tokens.

The other cards

On the RTX 3060 Ti, the comparison stops at bf16 against bf16. On 12 October, under load, the card dropped out four times, with three engines and two drivers, at 65 °C or less. I suspect its 500 W power supply: I bought a new one and will redo the measurement. Before that, on the bench, Herbert did 1.7 to 2 times llama.cpp CUDA at prefill on the 0.8B, 1.4 to 1.5 times on the 2B. Windows ignores the clock lock: the memory stayed at 6,801 MHz.

The other GPUs are not gaming cards. Herbert is compared there to llama.cpp in Vulkan.

In bf16 against bf16, Herbert is faster on the Arc, and on the 0.8B everywhere. The 4B is slower on both AMD integrated GPUs: 64 against 69 tokens per second on the 5700G, 20 against 26 on the 9900X. In f16 against f16, Herbert is slower almost everywhere. The AMD integrated GPUs compute in f16 twice as fast as in f32. Without cooperative matrices exposed by Vulkan, Herbert's bf16 version computes in f32 there; the f16 version computes in f16, which closes part of the gap on the 0.8B, not on the 4B. These are integrated GPUs: the Radeon I have ordered will get its own article.

On the Arc, there is something more interesting than a simple gap. Herbert runs at practically the same speed there in both formats: 333 tokens per second in bf16 on the 4B, 328 in f16. llama.cpp goes from 283 to 573. In bf16, Herbert is faster because llama.cpp does poorly there. In f16, it is slower because its own matrix multiplications do not get everything out of the card: llama.cpp's f16 path gets 1.75 times as much out of the card as Herbert does. On the common client, in f16, on the 4B at 4,106 tokens, Herbert gives its first token in 17.5 s, llama.cpp in 10.9 s. I have not had time to look into why yet, but I think I am going to enjoy it.

At decode, Herbert stays faster everywhere on these cards, in both formats.

I hoped for it, but not to this extent. On the processor, the way Herbert lays out the weights in memory had already paid off at decode: I hoped it would pay off on the GPU too. Still, I hadn't dared to make it a firm prediction :-D It is my working hypothesis, and I will measure it kernel by kernel.

A precise case

Two formats, two backends, micro-batches, flash attention, five cards, three model sizes, several prompt lengths: in a table that large, you can always find a cell where Herbert is the fastest. I am not fooled. Herbert is not faster than llama.cpp in absolute terms. It is in one precise case: a small model, one request at a time, a short or medium prompt, on an NVIDIA card. Elsewhere, at equal format, it is tied or slower: the 4B on the 4080 against llama.cpp's best setting, the Arc in f16, the 4B on the AMD integrated GPUs.

This precise case has a likely explanation. Herbert does not pick kernels from a library: it compiles the model. At prefill, it processes the prompt in large blocks, 2,048 rows on the 0.8B and 1,024 on the 4B, so as to read the weights only once per block. Each block goes to the card in one piece. At decode, it records the token step once and for all, with its fusions, then resubmits it at every token. On a small model, each kernel is short: what costs is everything around it, the launches, the synchronizations, the round trips. On the 4B, the matrix multiplications dominate, both engines approach what the card can do, and the gap closes. This is not specific to Herbert: beyond a hundred or so tokens, vLLM is also faster than llama.cpp on the 0.8B.

Part of this explanation can be measured without touching the code, since each of these choices is an engine setting. On 14 October, I turned these settings off one by one, then all at once, on the 4080, with the same binary, five alternating rounds. Here is the prefill time, relative to the time with the default settings:

The large blocks weigh the most: fewer passes, hence both fewer launches and fewer rereads of the weights, two effects this measurement does not separate. Submitting each block in one go and chaining the blocks without returning to the host account for 1 to 5%, mostly on short prompts. The fusions change nothing at prefill: they are worth 2% at decode. With everything off, the 0.8B prefill falls back to between 30,000 and 39,000 tokens per second depending on the length, in the range of llama.cpp CUDA. On a 24-token prompt, these settings account for only 15% of the time: the rest comes down to the kernels themselves, which this measurement does not turn off.

That is the heart of this series. Emitting your own code means being able to split and fuse the work for one precise card and one precise model. Why it pays off here and not there, I will keep checking kernel by kernel in the next articles.

The predictions, written beforehand

On the morning of 12 October 2026, before the definitive campaign, I wrote my predictions in a dated file, published as is with the data. I wrote them after a first series of measurements, from 6 to 8 October, whose results I knew. So they are not blind.

What I do not explain yet

I tried matrix multiplications split more finely, per subgroup, in 16 × 16 × 16 tiles. They are slower: 16 to 84% more time on the 4B at 2,048 tokens. The large tiles too, by 1 to 33%. Vulkan lacks two instructions to make this scheme work. I discovered them while writing a CUDA version of Herbert: they belong to the CUDA article.

The 4B prefill on the AMD integrated GPUs, the matrix multiplications on the Arc, and the 0.8B decode as the context grows remain without a complete explanation.

The limits

No B200, no H100, no FP8. No quantization either: this article compares weights in bf16 and in f16. Quantization will get its own articles, as announced in the previous article. Consumer GPUs, one card at a time, dense models, one request at a time: that is the niche I claim, not one I put up with. MoE and multi-GPU will come later.

These figures hold from 12 to 14 October 2026, with these versions and these drivers: NVIDIA 595.99.02 under Linux, 591.86 under Windows, Mesa 25.2.8 for the two AMD integrated GPUs, and 26.0.3 for the Arc. All these engines evolve. Some of these figures may no longer be true by the time you read this.

Reproducing at home

Three repositories are online, under the MIT license. The engine of this article, in bf16, with its server: github.com/xigh/herbert-vk-bf16, tag v0.1.0. Build, then run:

git clone --branch v0.1.0 https://github.com/xigh/herbert-vk-bf16.git
cd herbert-vk-bf16 && cargo build --release --locked --features serve
target/release/herbert-vk-chat ~/models/Qwen3.5-4B --no-think --ask 'What is the capital of France?'

The f16 version, without which the f16-against-f16 tables cannot be replayed: github.com/xigh/herbert-vk-f16, tag v0.1.0. It builds the same way. Without its f16 feature, it gives back the bf16 engine, bit for bit.

The campaign scripts, my predictions, the logbook, and the raw data, card by card: github.com/xigh/herbert-gpu-bench, tag v1-2026-10-14, folder campaign-v1/. If a figure looks wrong to you, you have what you need to check it.

What comes next

My predictions rested on one idea: CUDA gives access to instructions and features that Vulkan does not have. The question now is what CUDA could teach us that we do not know how to do in Vulkan. I know the answer. You will have to wait for the CUDA article.