← back

This article also exists in French: read the French version.

Discuss this article with your AI

The button copies the article with a short introduction for the model. Paste it all into Claude, ChatGPT, Le Chat or another AI, then ask it your questions.

The two times of an answer

This page follows the script of the video, with its pictures, and ends with the exact conditions of every measurement. The pictures come from the French video and keep their French labels.

Over the past weeks, I have shown you how Herbert, my inference engine for local AI, runs on different kinds of hardware: processors, graphics cards, Apple chips.

But what exactly do we mean when we say an AI engine runs at 40 or 55 tokens per second? There are in fact two speeds, and they do not depend on the same thing. In the end, what really matters is how long you wait for an answer.

While building Herbert, I gave myself the means to measure every step precisely. I use them here to dig into the subject, without a single formula.

A simple example

I give an AI model, running with Herbert on a consumer processor, the beginning of a sentence: "My cat is purple. But my friends tell me there is no such thing as a". It continues it, and I let it write 100 tokens. On my machine, between sending the sentence and the last token written, 7.2 seconds go by.

The sentence "My cat is purple. But my friends tell me there is no such thing as a ...", the continuation written by the model ("purple cat. Is there a purple cat? There is no such thing as a purple cat...") and a stopwatch at 7.2 seconds, on a Ryzen 9 9900X processor.

During that time, two very different things happen. First, the model reads the sentence: this is called the prefill. Here it takes 8 hundredths of a second. Then it writes the continuation, one token after another: this is the decode. That phase lasts 7.1 seconds. Why is one almost instant and the other so long? To understand it, we have to look at what happens inside the engine.

What is an AI model?

An AI model is first of all a file, sometimes split into several. Inside, there is nothing but numbers, billions of them: what we call parameters. These parameters are arranged in layers, which the computation goes through one after the other.

The model's parameters arranged in 28 layers, crossed one after the other, from the input text to the next token.

They were tuned during training, once and for all. When you ask a question, they no longer change: they are only used to do the computation.

And that computation does only one thing: from the text it is given, it computes the piece of word that should come next. This is called a token. We add that token to the text, and start again. An answer is just this loop, one token after another.

The loop: the text gives a token ("purple", 92.5 %), we add it, then "cat" (90.5 %), then the period (68.8 %), then "Is" (12.8 %).

Herbert, vLLM, MLX or llama.cpp: the program that does this computation is called an inference engine.

Two operations

Everything that follows rests on two very simple operations. The first takes two lists of numbers of the same length, multiplies them cell by cell and adds up the results. That gives a single number: the dot product.

It tells you simply whether two lists point the same way. In mathematics, these lists are called vectors. When they point the same way, the dot product is positive; when they point in opposite directions, it is negative. The more aligned they are and the larger their numbers, the further it moves from zero.

Three pairs of arrows in three dimensions: same direction, dot product 10; opposite directions, -10; at a right angle, 0.

The second takes a series of scores and turns them into percentages that add up to 100 %. The largest score takes the largest share. This is the softmax.

Four scores (1.2; 3.5; 0.4; 2.1) turned into percentages: 7 %, 72 %, 3 %, 18 %, always positive, 100 % in total.

A language model uses many other operations, but these two are enough to understand most of what goes on.

The model we open

To look inside, I take a small model: Qwen2.5, in its 1.5 billion parameter version, the base version. I chose it because all its layers work the same way, unlike some more recent models, and I wanted to keep my examples simple.

Everything you see here inside this model is its real numbers, on real sentences. The timings and the model's choices come from Herbert. The internal values are extracted with Transformers, the reference library.

Tokens and numbers

Back to our sentence. The text is first cut into tokens: seventeen here, one per word, plus the period. This model knows more than 150,000 different ones.

The sentence cut into 17 tokens (My, cat, is, purple, the period, But, my, friends, tell, me, there, is, no, such, thing, as, a) and their numbers in the vocabulary.

Each token is replaced by a row of a large learned table. Each row, a vector, is made of 1,536 numbers: this is the embedding table. From then on, the model never sees words again, only these lists of numbers, one per token.

The token "purple" replaced by its row of the table: 0.017, -0.004, -0.008, -0.005, 0.026, 0.028... 1,536 numbers, one row of a table of 151,936 rows.

Each of these lists goes through 28 layers. No layer replaces the list: each one computes changes and adds them for the next layer. Layer after layer, the list is transformed, according to what the model has learned.

In each layer there are two steps, always in the same order. The first makes the tokens communicate with each other: this is attention, the only place in the model where a token receives information from the others. The second works on each token alone, without looking at the others: the MLP.

One layer: 1, the tokens communicate, attention (12 query heads, 2 key/value heads); 2, each token alone, the MLP (8,960 learned directions).

Attention

Attention is not done once, but several times in parallel, by what are called heads. Each head looks at the sentence in its own way and goes looking for something else. This model has 12 per layer.

From the last token "a", four of the 12 heads of layer 19: head 2 looks at "purple" at 91 %, head 7 at "My" at 75 %, head 9 at "as" at 73 %, head 11 at "a" itself at 95 %.

Let us follow just one. Arriving at the last token, "a", the model builds three lists from it, each one through dot products with learned parameter tables: a query, Q, a key, K, a value, V. In this model, 128 numbers each. Every earlier token also has its key and its value.

The engine then takes the dot product of the query of "a" with the key of every token up to it. That gives one score per token. Put simply, "a" asks each token before it: how useful are you to me?

With a softmax, these scores become percentages. Here are the real ones, in layer 19, for one of its heads: 91 % on "purple", 5 % on "My" and almost nothing on the rest.

The attention of "a", layer 19, head 2: 91 % on "purple", 5 % on "My", almost nothing elsewhere.

I picked a completely improbable example: a purple cat. The model cannot find my cat's colour in its memory; it is in my sentence. And precisely, when it is about to write the continuation, this head goes and fetches "purple", thirteen tokens earlier. Nobody told it to: it is what its parameters learned.

What comes back to "a" is the sum of the values of these tokens, each weighted by its percentage: 91 % of the value of "purple", 5 % of that of "My", and so on.

The weighted sum: 91 % of the value of "purple", 5 % of "My", 1 % of the period, 1 % of "But", 2 % for the other 13, added to "a".

The 12 heads do this computation at the same time, each with its own query. But they share the keys and the values: two sets for the twelve. If the cat is green, I find the same heads on the colour. If it has five legs, other heads react. And if I switch off the one in layer 19, the model still writes "purple": it is not indispensable, the information also flows elsewhere in the network.

We can also see where the model goes looking for "purple": up to layer 10, the average attention on that word stays under 5 %. Then it rises, up to 29 % at layer 17.

Left, the attention of "a" on "purple" for 28 layers x 12 heads; right, the average of the 12 heads layer by layer, rising up to 29 % at layer 17. Bottom: 43 % on average on "My", the resting point.

On average, the heads give 43 % of their attention to the very first token, "My". Researchers call this an attention sink: in many models, the first token draws a large share of the attention, even when it carries nothing particularly useful. It acts a bit like a resting point.

For order to matter, the position of each token is written into its query and its key, by rotating their numbers by an angle that depends on the position.

The KV cache

Once computed, the key and the value of a token never change again. So the engine keeps them, in a table that gains one row per token: this is the KV cache, K for keys, V for values.

When the model writes a new token, it only computes the query, key and value of that token. It adds its row to the table and compares its query to all the keys already stored. Without this table, everything would have to be recomputed from the start, for every token written.

The real KV cache of the sentence, layer 19: the keys and values (128 numbers each) of the 17 tokens. 14,336 numbers per token, about 29 KB, about 117 MB for 4,096 tokens.

Its size can be computed. In this model: 14,336 numbers per token, about 29 kilobytes. For 4,096 tokens, about 117 megabytes, still little next to the 3.1 gigabytes of parameters. But at 32,000 tokens, it is almost one more gigabyte to read again, for every token written. The longer the conversation, the more each new token costs. Have you been told to keep your conversations with an AI short? Here is one of the reasons.

The token alone: the MLP

The second step of each layer works on each token alone. Here it holds almost 9,000 pairs of learned directions. For each pair, the model takes two dot products with the token's list: the first acts as a valve, it decides how much of the second to let through. That gives one value per pair.

For "a", in layer 19, most of them are small, positive or negative: half do not exceed 0.08. Nearly 300 exceed 0.5, 70 exceed 1, and the largest is 3.3.

The 600 largest of the 8,960 MLP values for "a", layer 19, sorted, positive on top, negative below. In absolute value: half under 0.08, nearly 300 above 0.5, 70 above 1, maximum 3.3.

Each of these values sets a learned correction: the strong ones weigh a lot, the small ones almost nothing. The whole is brought back to 1,536 numbers and added to the token's list. This is where most of each layer's parameters live: 41 million out of 47.

Choosing the next token

After the 28th and last layer, the list of the last token is compared, by dot product, with every known token: 151,936 scores. I show them as percentages, but there is no need for that to choose. Here, I always take the highest score: this is the argmax. Most assistants do not choose that way. A part of the engine, the sampler, draws the token at random while favouring the most likely ones. That is where the percentages really matter.

The first token written after "...no such thing as a": the top 10 of 151,936. "purple" at 92.5 %, "blue" at 0.6 %, "cat" at 0.6 %.

For our purple cat: first token, "purple", 92.5 %, far ahead of "blue", under 1 %. Then "cat", 90.5 %. Then the period, 68.8 %. One token at a time.

I don't know about you, but to me this is mind-blowing: nobody told it my cat was purple, except my sentence, my prompt. And that is where it went to find it.

This last step weighs 233 million parameters out of 1.5 billion. In many models it is a second large table, the same shape as the one at the start, but learned for another purpose: no longer turning a token into a list, but giving a score to every token. This small model uses the same table both ways, to save space. And to write a single token, it has to go through all of it: 15 % of everything the model has to read again.

The output: the share of parameters read again to write one token. Qwen2.5-1.5B, 233 M out of 1.5 B, 15 %; Qwen3.5-2B, 509 M out of 1.9 B, 27 %.

First time: reading

The first time, the prefill, is reading the text given to the model. All the tokens of the text go through the layers together. In each layer, every parameter read from memory serves all the tokens at once: for a 3,000-token document, each parameter is used 3,000 times.

3,000 tokens go through a layer together, whose parameters are read once for all of them.

In attention, each token is compared with all those before it. That makes a triangle of scores that grows very fast with length. And the KV cache fills up.

Here, what counts is the amount of computation. The faster the processor computes, the faster the reading.

Reading 3,139 tokens of The Gold-Bug at once: about 9,100 billion operations in 2.1 s, about 4,300 billion per second.

Second time: writing

The second, the decode, is writing the continuation. There, one token at a time, because the next one depends on this one. Each parameter read from memory is now used only once.

For every token written, all the model's parameters must therefore be brought in from memory, 3.1 gigabytes, plus the KV cache. And again for the next token.

For a single token written, 1.5 billion parameters read again, 3.1 GB in bf16.

My PC's processor writes 14 tokens per second. 14 times 3.1 gigabytes is about 44 gigabytes read per second. Measured on its own, its memory delivers at best 46 to 48. We are very close to the maximum: the computation is the one waiting.

Ryzen 9 9900X processor: 14 tokens/s x 3.1 GB, about 44 GB/s used out of 46 to 48 measured on their own.

1 % or 22 %?

What counts is the total: reading plus writing. I tried it on my PC's processor, a Ryzen 9 9900X, with two texts.

Our short sentence, the purple cat: 17 tokens read in 8 hundredths of a second, 100 tokens written in 7.1 seconds, 7.2 seconds in all. Reading is only 1 % of the time.

A long text, a passage of a short story by Edgar Allan Poe, The Gold-Bug: 3,139 tokens read in 2.1 seconds, 100 tokens written in 7.5 seconds, 9.6 seconds in all. Reading is 22 % of the time.

Purple cat: 17 tokens read, 100 written, reading 0.08 s, writing 7.1 s, 7.2 s. The Gold-Bug: 3,139 tokens read, 100 written, reading 2.1 s, writing 7.5 s, 9.6 s.

The share of each time: purple cat, reading 1 %, writing 99 %; The Gold-Bug, reading 22 %, writing 78 %.

Even with a long text, writing makes up most of the wait, because every token written reads the whole model again.

The proof that the two times do not depend on the same thing: without the AVX-512 instructions, which compute on 512 bits at once, reading the story is almost 6 times slower. Writing does not move.

The Gold-Bug with and without AVX-512: reading 2.1 s against 12.2 s; writing 100 tokens 7.5 s against 7.3 s. Reading x5.9 without AVX-512, writing unchanged.

Same thing when I optimise my engine: reading gains 12 %, writing still does not move.

The Gold-Bug before and after the first optimisation: reading 2.1 s then 1.8 s (-12 %); writing 7.5 s in both cases.

On this processor, with a single conversation, reading is computation. Writing is memory, and it is already close to its maximum.

And when the model thinks?

Many recent models think before answering. In practice, they first write a draft, which you do not always see, then the real answer. This draft, the think, is tokens written one by one, exactly like the answer, at writing speed.

I asked another small model, Qwen3.5 in its 2 billion parameter version, which can think: "Why is the sky blue in the middle of the day, but orange-red at sunset?". Without thinking, it answers in 7.5 seconds, five times out of five. But its answer is wrong: it says the blue is absorbed by the air. It is a tiny model, no miracle here.

With thinking on, over several runs, it first writes between 3,600 and 5,100 tokens of draft. The full answer arrives after 5 to 8 minutes. And this time, every answer is right. Here, a good answer therefore cost 40 to 60 times more waiting, because every thinking token is paid at the price of writing.

Qwen3.5-2B, "Why is the sky blue...", with and without thinking: without, 7.5 s and a wrong answer; with, 3,600 to 5,100 tokens of draft and 5 to 8 minutes, a right answer every time, 40 to 60 times more waiting.

The model we opened earlier does not think: it is a base model, it completes the text, that is all. The assistants you use were trained a second time, to answer a question and stop at the end of their answer. This is called post-training, and the resulting model, the Instruct version. But underneath, it is the same loop: one token after another, at the same price.

What to remember

An answer is therefore two times. On my machine, reading is computation: all the tokens go through together, and a processor that computes faster reads faster. Writing is memory: one token at a time, and for each token, the whole model to read again.

Reading: computation, all the tokens together. Writing: memory, one token at a time, the whole model read again every time.

So the next time someone announces an engine at so many tokens per second, ask yourself two questions: how long does it take to read what I give it? And how long to write its answer? In the end, what matters is not the number of tokens per second, it is the time you spend waiting.

In a later video, I will talk about models that, to write, only read part of their parameters again: mixture-of-experts models.

Measurement conditions

Everything above holds for these conditions: one processor, a single conversation at a time, two small models, one question for the thinking test.

The machine. AMD Ryzen 9 9900X (12 cores, AVX-512 and AVX512-BF16), 12 threads. Memory bandwidth measured on its own, with a streaming read emitted by Herbert: 46.0 GB/s on 1 thread, 48.1 GB/s at best on 2 threads, 45.0 GB/s on 12 threads.

The engine. Herbert, JIT version for processors, in bf16 (no quantisation). The purple cat and The Gold-Bug timings are measured before optimisation; the "after" column of the optimisation is the first step of the same engine. Without AVX-512, the engine goes through its AVX2 kernels and also loses the dedicated bf16 instructions. Every timing is a median of 5 runs, context cleared between runs, 100 tokens written.

The model we opened. Qwen2.5-1.5B, base version (not Instruct), original weights in bf16: 28 layers, 1,536 numbers per token, 12 query heads and 2 key/value heads of 128 numbers, MLP of 8,960, vocabulary of 151,936 tokens, same table for input and output. Tokens are chosen by argmax (greedy decoding).

The internal values. Attention, KV cache, MLP and embedding are extracted with Transformers 5.14.1 (PyTorch 2.13), in bf16 on the same processor, attention in eager mode. There the model predicts "purple" at 90.3 %, against 92.5 % in Herbert: the two computations do not round at exactly the same places. Layer and head numbers are counted from 1 in the text (layer 19 is index 18).

The heads. The "green" and "five legs" variants use the same sentence with a single word changed. For the ablation, the output of a head is set to zero before the output projection of attention, at every position. Head 2 of layer 19 switched off: "purple" goes from 90.3 % to 92.5 %. The 10 heads that look most at the colour switched off together: 83.0 %. Ten heads drawn at random: 93.2 %. These observations show redundancy; on their own, they do not say which head "carries" the information.

The texts. The passage of The Gold-Bug is Legrand's explanation of the cryptogram, cut after "about four miles to the northward of the island." (3,139 tokens).

Thinking. Qwen3.5-2B in bf16 in Herbert (first optimisation step), context of 16,384 tokens. Without thinking: greedy decoding, one warm-up then 5 repetitions, 85 tokens written, done at 7.49 s all five times. With thinking: the sampling recommended by the model card (temperature 1.0, top_p 0.95, top_k 20, min_p 0, presence_penalty 1.5), seeds 1 to 5, cap of 8,192 tokens. Full answer in 320.6 to 464.6 s, at about 11.2 tokens/s. One limit to state: seeds 2 and 3 gave exactly the same text, and so did seeds 4 and 5. There are therefore only three distinct draws out of five; the seed handling in the server needs fixing.

The raw data. The measurement scripts, as they ran, with the raw outputs: repository to come.