One level down: #002.1 follow-up, the Arc in f16

This follow-up extends the article #002 the same compiler on a GPU, published on 18 October, which I am not changing. It had one case where Herbert clearly lost: the Arc 140V GPU of a Core Ultra 7 258V laptop, in f16 against f16. At prefill on 759 tokens, Herbert did ×0.89 of llama.cpp Vulkan on Qwen3.5-0.8B and ×0.57 on the 4B. I wrote that its matrix multiplications did not get everything out of the card, that I had not had time to look into why yet, but that I was going to enjoy it.

I did enjoy it.

What changed

Three things, all specific to this card.

The matrices the card can do. Under the Mesa ANV driver, the matrix units of the Arc 140V offer a single shape: 8 × 16 × 16 tiles in f16, on subgroups of 16 invocations. Herbert's kernels expected the 16 × 16 × 16 tiles on 32 invocations of the NVIDIA and AMD cards. Not finding them, the prefill attention fell back here on kernels that did not use the matrix units at all. It now works on tiles of 8 rows, the shape the card offers.

The size of the tiles. This is most of the gain. A matrix multiplication is cut into blocks: each workgroup computes one block of the output, and for that loads a band of each of the two input matrices. The larger the block, the more computation each loaded value serves. On this card, Herbert computed blocks of 32 × 64 outputs per workgroup of 64 invocations: 2,048 outputs for 96 rows and columns read at each step. It now computes blocks of 128 × 128 per workgroup of 512 invocations: 16,384 outputs for 256 rows and columns read, three times more computation per value read. The output block no longer fits in the workgroup's 48 KiB of shared memory, so the results are written straight from the accumulators.

I measured 31 tilings on the card before keeping this one. Each of its 32 subgroups carries 2 × 2 matrices. Beyond that, throughput falls to less than half. An even larger block, 256 × 128, gains 6 % on one model and loses 6 on another. The tiling kept has the same shape as llama.cpp's on this card.

A defect found on the way. The path that writes straight from the accumulators already existed, but it was not used by default. It hid a defect: Qwen3.5's Gated DeltaNet projection is 8,224 columns wide, not a multiple of 128, and the last matrices of each row landed on the next row. It is fixed: Qwen gives exactly the same tokens again as with the former write.

The figures

Same card, same prompts, same protocol as the October campaign, five alternating rounds, f16 against f16. Facing it, the llama.cpp of the article, 08ffba21e, at its best setting, and the one of 2 November, 4e910a86e, at its default setting.

On the 4B, Herbert goes from 328 to 804 tokens per second. The llama.cpp of the article gives back its October speeds, 573 tokens per second at its default setting. The one of 2 November has gained 8 to 19%: Herbert stays ahead.

On the common client, on the 4B at 4,105 tokens, in f16, Herbert now gives its first token in 7.75 s, against 17.5 s on 18 October. llama.cpp of 2 November gives it in 10.70 s, the one of the article in 11.14 s. These are medians of 15 requests. At decode, nothing changes.

A detail I had not written

On this card, for f16 weights, llama.cpp accumulates its matrix products in f16 (matmul_f16_f16acc). Herbert accumulates in f32. "f16 against f16" therefore means the same weights and the same activations, with a more precise accumulator on Herbert's side. Both GGUF files are F16, with no quantized tensor.

The limits

This measurement only holds for the Arc 140V, under Mesa 26.0.8 and Linux kernel 7.0.0. The code measured is version 0.2.0 of herbert-vk-f16 (tag v0.2.0), published with this follow-up. Before publishing it, I ran the non-regression again: on the RTX 4080 and the two AMD integrated GPUs, it gives the same tokens as the article's version, wherever the compute path did not change. The rest of article #002 stands as published, including the cases where Herbert is tied or slower: the 4B on the RTX 4080 against llama.cpp's best setting, and the 4B on the AMD integrated GPUs.