← back
Never miss a thing.
One email per new article, plus upcoming course announcements, nothing else. The full tech-watch news, figures and gifts are reserved for Subscriber Area members.

This article also exists in French: read the French version.

Discuss this article with your AI
The button copies the article with a short introduction for the model. Paste it all into Claude, ChatGPT, Le Chat or another AI, then ask it your questions.

One level down: #005 Is Herbert better than Llama.cpp?

Herbert vs Llama.cpp, 2 times faster? 4-bit quantization on a CPU.

In short

This article has two levels. First, the text of the video, which stands on its own (it has an English audio track, an AI clone of my own voice). Then the details, in sections you can read on their own: the tables, how it was measured, the limits and everything you need to replay it at home.

What I said in the video

Llama.cpp v1.0 is out! This incredible project is the reference for serving local, open-source AI. Many other projects, like Ollama, are built on it.

The question we are interested in today is whether it is possible to design a faster AI engine, in Rust, without a single line of assembly (or almost). Herbert, my inference engine, can do it, and on some tests it is even 2 times faster. Let me tell you about it and explain what is really going on.

Remember, a language model is billions of numbers stored in a few files. These numbers are called the model's weights. They take up a lot of space in your computer's memory, or in the graphics card's memory.

So we try to reduce that size: we store each number with less precision. That is what we call quantization. The idea does not come from language models: it was already done on neural networks in 2015. But Llama.cpp, released in March 2023 by Georgi Gerganov, made it common on our computers: its stated goal was to run a quantized model on a MacBook.

Let me explain it simply. Quantizing means storing a model's weights in a smaller encoding. Most models are encoded in 16 bits: each number can take 65,536 different values. Here we look at 4-bit quantization, where each number has only 16 possible values, which is 4,096 times fewer.

Needless to say, we are bound to lose something: we betray the original model a little. In return, it runs faster and we can run bigger ones on the same machine.

Well yes, to write a single token, the engine reads all the weights again, as I have been telling you for a few weeks. So, going from 16 bits to 4 bits, we should normally have 4 times fewer bytes to read. In practice it is a bit more than 3 times fewer, because not all the model's weights are quantized, and because we actually quantize around a reference value, block by block. As a result, the engine should, in principle, write the answer to a question 3 times faster.

And you are starting to know me: like everything I do in this series, it has to be checked and measured. With Herbert, on the same CPU, the Gemma4 E4B model, which weighs about 16 GB, goes from 5.65 tokens per second in 16 bits to 17.5 in 4 bits. The Qwen3.6 35B model, which weighs about 72 GB in 16 bits, does not even fit in the 67 GB available in my machine's memory. In 4 bits, it takes up 25.

So Herbert also runs these models in 4 bits, on the CPU, without a graphics card. I have tested it on several models: Qwen3, Gemma4 and even LFM from LiquidAI. It also works on graphics cards, but we will talk about that in the coming weeks.

To check whether Herbert is correct and fast, I compared it with Llama.cpp, whose version 1.0 came out on November 13, 2026. It is the engine I have been measuring myself against since the start of the series. When I compared the two, Herbert read the question 2 times faster and wrote the answer 1.5 times faster.

With such a gap, I told myself there had to be a mistake, even if the text seemed consistent. While digging, I realized I was not comparing quite the same thing. The file you usually download for Llama.cpp is called Q4_K_M. It says 4 bits, but part of the weights is kept in 8 bits. It is heavier and it stays closer to the original model. Herbert, on the other hand, stores everything in 4 bits, like Llama.cpp's simplest format, Q4_0. When writing the answer, part of its lead came from there: it was reading fewer bytes. And of course, it had to pay for it in fidelity.

So I also needed to measure that fidelity. We have the unquantized model read a text and note, at each word, what it would have written next. Then we do the same with the 4-bit version and see how far it drifts. It has to be said: mathematically, the Q4_K_M drifts almost 4 times less than the Q4_0. Scary, isn't it?

Yet, on whole answers, I asked 15 questions to the different versions. To "what is the largest city in Canada", the 16-bit model answers Toronto. All the 4-bit versions answer Montreal, the Q4_K_M too. On an economics question, Herbert and Llama.cpp in Q4_0 give the same answer word for word, slightly worse than the original model's, which the Q4_K_M reproduces exactly. And 3 times, it is the 16-bit model that gets it wrong and the 4-bit versions that get it right. It is almost a surprise, and rather odd in the end. But what we measure is not whether quantization gives more wrong answers: it is whether the quantized model drifts far from the original model.

In the end, what to remember is that for the tests I ran, quantization is not awful at all.

What we want to do here is see whether, with similar quantization, Herbert stays just as fast against Llama.cpp. So I redid the comparison, giving Llama.cpp the same format as Herbert, made from the same weights. The quality is then the same, within 1%.

And even then, Herbert still reads the question about 2 times faster on one CPU and 1.5 times faster on the other. It has barely moved. When writing the answer, however, its lead is only 8 to 13%, but it stays ahead. Except for Gemma4 E2B, a small model from Google, where Llama.cpp reads the question faster than Herbert. I have not had the chance to look into why yet.

To stay objective, I have to admit there is a fork of Llama.cpp, ik_llama.cpp, written by Iwan Kawrakow, the author of the Q4_K_M format. With the same format, it matches Herbert when writing the answer, within a few percent. When reading the question, it is behind Herbert on short prompts and ahead on long ones, by up to 30% on one of the 2 CPUs. I clearly still have work to do.

Four bits against sixteen

Same engine, same machine, same time window: an EPYC 4344P with 8 threads, Herbert in both formats, a prompt of about 630 tokens. Both formats use the same files: Herbert quantizes at load time, from the original weights. So there is no disk saving to expect from 4 bits, only memory and speed.

ModelFormatWriting (tokens/s)Bytes read per tokenMemory at rest
Qwen3.5-2B16 bits13.93.76 GB6.4 GB
Qwen3.5-2B4 bits43.21.18 GB3.4 GB
Gemma 4 E4B16 bits5.659.29 GB21.7 GB
Gemma 4 E4B4 bits17.52.90 GB15.6 GB

Writing is about 3 times faster in 4 bits: that is precisely the ratio of bytes read per token (0.31). Both formats read memory at the same speed: decoding follows the bytes, which matches #001. Memory at rest is roughly halved on the 2B, but only drops by a quarter on Gemma 4 E4B: a 9.3 GB table is read in place in both formats.

The price shows at startup: quantizing at load time takes time. Loading is about twice as long in 4 bits.

What a Q4_K_M file contains

A "Q4" file is not all 4 bits. I opened both files of Qwen3.6-35B-A3B, tensor by tensor. In both, the experts, which make up most of the model, are in 4 bits. The difference is elsewhere: in the Q4_K_M, everything that is not an expert is kept in 8 bits (the recurrent layers, the output head, the attention, the shared expert). The file is 20.5 GB, against 19.6 GB for the pure Q4_0.

The size difference looks small. It is not small for writing: for each token, only a few experts are read, whereas everything that is not an expert is read every time. Those are precisely the parts kept in 8 bits.

These formats have a history: Q4_0 dates from the very first day of Llama.cpp (the example file was already called ggml-model-q4_0.bin); the K formats, including Q4_K_M, arrived in June 2023 with Iwan Kawrakow's "k-quants" pull request.

Measuring fidelity

Each quantized version is compared with a reference: the same model in 16 bits, computed by transformers, the reference library. Every version is fed exactly the same tokens, a wikitext text in English and a text in French. At each position, we keep the whole distribution of the next token, then measure how far it is from the reference's (the Kullback-Leibler divergence, KL: 0 when both distributions are identical). We also count how often the first choice is the same.

ModelHerbert, Q4_0Llama.cpp, Q4_0Llama.cpp, Q4_K_M
Qwen3.60.072; 87.8%0.072; 87.6%0.019; 94.2%
Gemma 4 E4B0.041; 90.2%0.038; 90.3%
Gemma 4 E2B0.057; 88.2%0.053; 88.3%

KL on wikitext, then the share of identical first choices, measured on the Ryzen 9 9900X. With the same format, Herbert and Llama.cpp drift from the reference by the same amount. It is the Q4_0 format that is less faithful, for every engine; Q4_K_M is almost 4 times more faithful (0.019 against 0.072).

A different first choice is not a mistake. More than 80% of the switches happen where the reference was already hesitating between two almost equally likely tokens.

Whole answers

An average does not tell you what an answer looks like. I asked 15 questions to five versions of Qwen3.6, with identical inputs: the 16-bit reference, Herbert in Q4_0, Llama.cpp and ik_llama.cpp on the same Q4_0, then Llama.cpp in Q4_K_M.

So drifting from the reference does not always mean doing worse. And the clearest mistake, Canada, does not separate engines or formats.

The comparison with the same format

To compare what is comparable, I made a pure Q4_0 of Qwen3.6-35B-A3B from the same weights, then gave it to Llama.cpp and ik_llama.cpp. Herbert quantizes at load time with the same rule. Two exceptions: Herbert keeps the embedding table in 16 bits and two projections of the recurrent layers in 32 bits, where the file stores them in Q4_0.

Prefill / decode in tokens per second, medians of 3, 128 tokens generated.

EPYC 4344P, 8 threads:

Engine627 tokens6,131 tokens
Herbert335 / 25.9306 / 23.9
Llama.cpp, pure Q4_0171 / 23.9156 / 20.8
ik_llama.cpp, pure Q4_0312 / 26.6303 / 24.7
ik_llama.cpp -rtr, pure Q4_0368 / 26.8351 / 24.9
Llama.cpp, Q4_K_M157 / 16.6141 / 14.6

Ryzen 9 9900X, 12 threads:

Engine217 tokens621 tokens6,125 tokens
Herbert376 / 21.3371 / 21.1389 / 18.9
Llama.cpp, pure Q4_0239 / 18.7260 / 18.6248 / 17.2
ik_llama.cpp, pure Q4_0325 / 22.3407 / 22.1456 / 20.5
ik_llama.cpp -rtr, pure Q4_0350 / 22.3436 / 22.0500 / 20.5
Llama.cpp, Q4_K_M199 / 13.4221 / 13.3208 / 12.5

Against Llama.cpp on the same file, Herbert reads about 2 times faster on the EPYC and about 1.5 times faster on the 9900X. It writes 8 to 13% faster at about 630 tokens. Against the Q4_K_M, writing is about 50% faster (25.9 against 16.6), but most of that gap comes from the format: on the same Q4_0, Llama.cpp goes from 16.6 to 23.9.

Against ik_llama.cpp, it is closer. On the EPYC, Herbert sits between its two variants for prefill (335, between 312 and 368) and is 3% behind when writing. On the 9900X, Herbert is ahead on short prompts (376 against 350) and behind on long ones (389 against 500); when writing, it is 4 to 8% behind.

Gemma 4, QAT weights, EPYC 4344P, 8 threads, about 630 tokens:

ModelHerbertLlama.cppik_llama.cppik_llama.cpp -rtr
E4B398 / 17.5279 / 16.3
26B-A4B349 / 20.1140 / 18.7301 / 19.8358 / 19.9
E2B798 / 35.5969 / 30.91,908 / 35.12,004 / 35.0

On Gemma 4 E2B, Llama.cpp is 20% ahead on prefill and ik_llama.cpp is two and a half times faster than Herbert; Herbert stays 15% ahead of Llama.cpp when writing.

What I can't explain yet

Where the "2 times" in prefill over Llama.cpp comes from. I have run the profiles; I have not attributed the gap yet.

Why ik_llama.cpp's prefill rises with prompt length on the 9900X (325, then 456) while Herbert's stays flat (376, then 389).

Gemma 4 E2B: Llama.cpp 20% ahead, ik_llama.cpp two and a half times faster. It is a model to study in its own right.

Limits

Two CPUs, both with AVX-512: an EPYC 4344P and a Ryzen 9 9900X, on Linux. Quality is measured on AVX-512 only. A single benchmark prompt, one request at a time. The same-format comparison covers Qwen3.6 and Gemma 4 with QAT weights.

The engines I compare against move by 2 to 4% from one run to the next: that is why I give rounded ratios ("about 2 times") rather than decimals. The throughput benchmark of November 13 had no predictions written beforehand; those of November 14 did, and they are published with the code (5 right or partly right, 2 wrong).

Versions measured: Llama.cpp 5676dfbe0, 7 commits before its v1.0.0 tag (CI, removed examples, version numbers: no compute file touched); ik_llama.cpp 7aa7e014. Both are under the MIT license. These numbers hold at the time of publication: engines change, and some of them may no longer be true when you read this.

Replay it at home

The code is online, under the MIT license: github.com/xigh/herbert-jit-q4, tag v0.1.0. The engine, the server and the whole benchmark live in the same repository: the scripts as they ran, the raw data and the predictions written before measuring, in bench/.

git clone --branch v0.1.0 https://github.com/xigh/herbert-jit-q4.git
cd herbert-jit-q4 && cargo build --release --locked --features serve
target/release/herbert-jit-chat Qwen/Qwen3.5-0.8B --q4 --no-think --max-tokens 64 --ask "What is the capital of France?"

You need stable Rust and the system's linker. Models are downloaded with Hugging Face's tool, hf; the repository's README gives the exact command, the OpenAI-compatible server and how to replay a line of the benchmark. The default Q4_0 is the one in this article. If a number looks wrong to you, you have what you need to check it.

What's next

Herbert's next project is to get the fidelity of Q4_K_M while keeping as much of the performance as possible. Today this format costs writing speed in every engine I have measured: the open question is how to get that fidelity without reading more bytes.

I will also have to show you what happens when you handle several requests at once. Herbert serves only one for now. Reading the weights once to write several tokens should pay off, at least for writing the answer. Except for mixture-of-experts models, where two requests do not go through the same experts at the same moment. And Qwen3.6, the model in this article, happens to be one. On that front, for now, all the other engines in the series are much better.

To conclude: after several months of work, Herbert is finally going public. It is already in production at a few of our customers and it is now the heart of mAIstrow.

There is still a lot to do, but the foundations are solid.

One level down, or several...

  1. One level down: #000 an inference engine that writes its own code · 03-Oct-2026
  2. One level down: #001 a compiler for an LLM engine · 11-Oct-2026
  3. One level down: #002 the same compiler on a GPU · 18-Oct-2026
    1. One level down: #002.1 follow-up, the Arc in f16 · 02-Nov-2026
  4. One level down: #003 what CUDA has that Vulkan lacks · 25-Oct-2026
  5. One level down: #004 the same compiler on Apple's chips · 01-Nov-2026
  6. One level down: #005 Is Herbert better than Llama.cpp? · 15-Nov-2026