← back
Never miss a thing.
One email per new article, plus upcoming course announcements, nothing else. The full tech-watch news, figures and gifts are reserved for Subscriber Area members.

This article also exists in French: read the French version.

Discuss this article with your AI
The button copies the article with a short introduction for the model. Paste it all into Claude, ChatGPT, Le Chat or another AI, then ask it your questions.

One level down: #006 I started porting Olala, OVHcloud's model

I started porting Olala, OVHcloud's model, to Herbert, my inference engine. Olala's 36 layers, annotated.

In short

On November 19, 2026, OVHcloud released Olala-v1.0-mini, a language model. A model designed in France, trained on European supercomputers, with its weights open under the Apache 2.0 open source license. It has 6.8 billion parameters and uses only about one billion of them to write each token.

And it has one peculiarity: none of the usual engines can run it in its official form. So OVH released its own engines for its model: a modified version of Llama.cpp, a version of vLLM for servers, and a port to MLX, Apple's framework.

That same evening, I started porting it to Herbert, my inference engine. It runs it now. But before showing you how it compares with OVH's engines, which will be the next issue, I wanted to stop on the model itself. This is a French model and, frankly, the incredible work our teams in France are doing deserves the time. All the more so because the technology they used is different.

Olala's diagram

To understand this model, you have to look at how it is built. I drew a diagram, with numbers, to find your way around.

Olala-v1.0-mini-7A1B (OVHcloud): 6.8B parameters, about 1B active per token Token IDs 1 Embedding 151,936 × 1536, normalized: h starts on the unit sphere Block × 36 2 Mixer Mamba-3 (M): 29 blocks 3 or attention (V): 7 blocks 4 Geodesic residual 5 Latent MoE 256 experts, 6 active, + 1 shared 4 Geodesic residual no norm before the mixer or the MoE no final norm 6 Head (lm_head) 1536 → 151,936, untied from the embedding logits 4 Geodesic residual (two per block) θ h h' The branch is not added to h: it rotates h on the sphere. tangent component θ ≤ π/4, divided by (block index + 1) the norm of h is preserved 2 Mamba-3 MIMO mixer (M) 48 heads of 64, state size 128, MIMO rank 4, no convolution 2.1 in_proj 1536 → 6288, per head: z[64] x[64] dt A trap 2.2 in_proj_dyn 1536 → 1056: B[4×128] C[4×128] angles[32] 2.3 B, C norm and rotation RMSNorm, then 32 pairs rotated by a cumulative angle 2.4 Per-head state S 64 × 128 in f32, trapezoidal update: α·S + β·(previous token) + γ·(current token) 2.5 Readout and gate o = S·q + D·v, then × silu(z) 2.6 mixer_proj 3072 → 1536 3 Differential TPA attention mixer (V) blocks 4, 9, 14, 19, 24, 29, 34; 48 heads of 128; no RoPE; 32,768-token window 3.1 Queries c_q: 1536 → 48 × 128 3.2 TPA keys and values rank-4 product: 12 heads of 128 3.3 Token shift k and v mixed with the previous token's 3.4 Norms and scalable softmax RMSNorm on q and k; q × ln(position + 1) 3.5 Scores ÷ 128, softcap 150, softmax 3.6 Differential per group of 4: 3 signal heads − λ × 1 noise head 3.7 Gate and mixer_proj × silu(gate_proj·h + 1.15), then 4608 → 1536 5 Latent MoE (in each of the 36 blocks) h 5.2 down_proj 1536 → 384: the latent space 5.1 Router (f32) sigmoid, top-6 of 256 (+ bias for selection) 5.3 6 experts 384 → 768 → 384, ReLU² picks the 6 and their weights 5.4 Weighted sum renormalized weights × 2.45 5.5 up_proj 384 → 1536 5.6 Shared expert fc_1 then fc_2: 1536 → 1536 → 1536, ReLU² + The 36 blocks M 0 M 1 M 2 M 3 V 4 M 5 M 6 M 7 M 8 V 9 M 10 M 11 M 12 M 13 V 14 M 15 M 16 M 17 M 18 V 19 M 20 M 21 M 22 M 23 V 24 M 25 M 26 M 27 M 28 V 29 M 30 M 31 M 32 M 33 V 34 M 35 Mamba-3 (M) attention (V) matrix that reads the outliers (up to ~1e14): kept in bf16 in Herbert's --q4 (all four on GPU, two on CPU) From config.json and modeling_olala.py, OVHaiLLM/Olala-v1.0-mini-7A1B on Hugging Face (revision b40f816)

The model reads from top to bottom. Tokens enter the embedding (marker 1), which places the stream on a sphere of radius 1. Then 36 blocks follow one another. Each has a mixer, either a Mamba-3 layer (marker 2, 29 blocks) or an attention layer (marker 3, blocks 4, 9, 14, 19, 24, 29 and 34), then a mixture of experts (marker 5). After the mixer and after the experts, the result is not added to the stream: it rotates it (marker 4). The step numbers in the diagram (2.4, 3.6, 5.1, etc.) are the ones I use below.

A summary instead of a full memory

A classic language model writes each token by looking at all the previous tokens. It keeps a trace of each of them; that is what we call the cache. The longer the text, the bigger the cache grows and the more each new token costs.

Olala does it differently. Of its 36 layers, 29 are Mamba-3 layers. Instead of keeping everything, they hold a fixed-size summary, updated at every token.

This technology comes from the work of two American researchers: Albert Gu, of Carnegie Mellon University, with Tri Dao, of Princeton University. Tri Dao is also the author of FlashAttention, a way of computing attention that almost every inference engine uses. Their first Mamba dates from late 2023.

Mamba-3, released in March 2026, adds three things.

The first: a finer update (step 2.4). At each token, the summary no longer takes into account only the incoming token but also the one before. It is the difference between measuring a slope at one point and measuring it between two points: the summary follows the text more closely.

The second: a state that can rotate (step 2.3). Some of the numbers in the summary are arranged in pairs, and each pair turns by a small angle at every token, like the hand of a watch. This lets the model track something that changes along the text, a task where models of this kind used to fail.

The third: instead of reading and writing its summary once per token, each part of the model does it 4 times. This matters for an engine: during decoding, the engine spends its time waiting for memory. So you can compute more on the same bytes without slowing down.

In Olala, each Mamba-3 layer has 48 heads, and the summary of one head holds 64 × 128 values.

One layer in five keeps everything

But a summary does not keep everything word for word. So one layer in five stays a classic attention layer, which keeps the exact past. And OVH stacked several recent ideas on it.

First, a so-called differential attention, an idea published by Microsoft researchers in 2024 (step 3.6). In a classic attention, each token spreads its attention over all the previous tokens, and each one gets at least a little, even those that have nothing to do with it. Over a long text, these crumbs add up and end up drowning the useful information. Differential attention computes two distributions and subtracts the second from the first, like headphones that cancel ambient noise. In Olala, the attention heads work in groups of 4: 3 heads pick up the signal and 1 head picks up the noise that gets subtracted.

Next, keys and values built from small vectors, a technique called TPA (step 3.2).

And no position information at all. An attention, by itself, does not know in what order the tokens come: to it, the text is a bag of words. In a classic transformer, the position is therefore added by hand: each key and each query is rotated by an angle that depends on its place in the text; that is what we call RoPE. Olala has none. Mamba, which reads the tokens in order, carries the order of the text.

Finally, over a long text, attention tends to spread itself too thin over too many tokens. So Olala multiplies the query by the logarithm of the token's position, to keep attention as sharp on a long text as on a short one (step 3.4).

256 experts and a stream that rotates

After each layer, there are 256 small networks, the experts. For each token, a small layer of neurons, the router, picks 6 of them (step 5.1). A 7th expert, called shared, works for every token. That is why the model rereads only one billion weights per token instead of almost 7.

And one last peculiarity: in most models, each layer adds its result to the stream that runs through the model. Here, the stream always keeps the same length. Each layer only rotates it by a small angle.

Where these ideas come from

OVH says so in its technical report: the team started from a hybrid skeleton of the Qwen3.5 kind, then reworked each element of the block (the mixer, the expert network, the normalization) by testing published improvements on a reduced bench, a 750-million-parameter model trained on 10 billion tokens. The novelty they claim is the combination. The bricks themselves all come from public papers.

Mamba-3 (marker 2). Mamba (Gu and Dao, December 2023) replaces the growing cache with a fixed-size state: at each token, the layer decides what it forgets and what it writes. Mamba-2 (May 2024) shows that this computation can be written as a form of attention, which makes it fast on a GPU during prefill. Mamba-3 (March 2026) adds the trapezoidal update, the rotating state and the so-called MIMO form, of rank 4 in Olala. Before Mamba-3, OVH's previous generation used Gated DeltaNet (Yang, Kautz and Hatamizadeh, December 2024).

Attention (marker 3). Differential attention (Microsoft Research, October 2024) subtracts a "noise" distribution from a "signal" distribution; Olala uses the grouped variant (Lim et al., October 2025), 3 signal heads for 1 noise head. TPA (Zhang et al., January 2025) builds keys and values as rank-4 products. NoPE (Yang et al., January 2025) removes any position encoding. Scalable softmax (Nakanishi, January 2025) multiplies the query by ln(position + 1); it is the only paper cited in OVH's code. Before the attention, each token's key and value are mixed with those of the previous token (step 3.3, after the Forgetting Transformer of Lin et al., March 2025). After it, a gate can almost shut off a head's output (step 3.7, Qiu et al., May 2025).

The experts (marker 5). The choice of the 6 experts follows DeepSeek-V3's recipe (December 2024): one score per expert between 0 and 1, and a small bias per expert, learned to balance the load, which is used for the choice without changing the weights. The experts are called latent, an idea from Nemotron 3 (NVIDIA, December 2025): they work on a version of the stream reduced from 1,536 to 384 values (steps 5.2 and 5.5). Inside each expert, the function is a squared ReLU (Primer, Google, September 2021).

The sphere (markers 1 and 4). nGPT (NVIDIA, October 2024) keeps the stream on the sphere of radius 1. GeoNorm (Zheng et al., January 2026) replaces addition with a rotation along the sphere; in Olala, the angle is capped at pi/4 and divided by the block number plus one.

What it asks of an engine

What interests me is what all this asks of an engine to make it work. And it is not trivial.

First, two kinds of layers to write instead of one. And neither of them is a layer you find in the usual engines.

Then, precision where you don't expect it. OVH says so in its documentation: the Mamba summary must stay in 32 bits. The router computes in 32 bits too. When it picks its 6 experts out of 256, the 6th and the 7th are sometimes almost tied, so the order in which the numbers are added can be enough to tip the choice.

Then there is this logarithm of the position in the attention. The model has no position, except there. So the engine has to carry the token's position all the way to the computation, at every token. If it forgets, the result is wrong.

And these rotations, the one in the Mamba-3 summary as well as the one in the stream, have to be computed with sines and cosines. They had a rather unexpected effect on Herbert, because of a quirk of Intel graphics cards I had not seen coming. I tell the story below.

And finally, OVH explains in its technical report that during training, three Mamba-3 layers ran away, with values up to 10 to the power of 16. And it was the rotation of the stream that hid everything: beyond a certain point, whether the value is 10 or 10 to the power of 16, the rotation is the same. The model doesn't care.

An engine that wants to compress it to 4 bits, on the other hand, does. That is why it was such a fight for me, and probably also why OVH does not offer a 4-bit version for Llama.cpp.

What I ran into while porting it

Token 18

To check that Herbert computes correctly, I compare it with the reference, OVH's code running in transformers 5.8.1. Very quickly, something was off at the 18th token of my first test question. Except that it was the reference that hesitated. Transformers has two paths for this model: without cache, everything in 32 bits; with cache, where part of the Mamba state is kept in 16 bits (bf16).

At token 18, the two paths do not write the same word. I took the all-32-bit path as the reference and kept the Mamba state in 32 bits in Herbert, as OVH asks. No optimization possible there: OVH warns that with a 16-bit state, long generations end up looping.

Next, I ported Olala to my CUDA and Vulkan engines. OVH does not offer a Vulkan version: with this port, Olala also runs on AMD and Intel graphics cards.

And there, on the NVIDIA card as with Vulkan, at the same token 18, Herbert no longer wrote my reference's word. I looked for a bug. On the RTX 4080, I looked at the router of the second-to-last layer, the one that picks the 6 experts. It gives a score of 0.246511 to one expert and 0.244879 to another: these are the 6th and the 7th, 0.0016 apart. Herbert saw them in the other order, so it did not pick the same 6th expert. There was no bug. The two experts are almost tied, and a rounding difference on the graphics card is enough to tip the choice. I only did this check on the NVIDIA card. With Vulkan, Herbert writes the same word at the same place, but I did not go and look at the router.

Values up to 3 × 10^16

I wanted to put the model in 4 bits, as I did for the others. On Metal, the text went wrong from the second token. By testing the families of matrices one by one, I found four that break the text in 4 bits: mixer_proj, fc_2, output_experts and up_proj. Then, by looking at the values flowing through the model, I found huge numbers. I measured them in 16 bits on two test questions, by two independent paths, Herbert's CPU engine and OVH's code in transformers, which agree within 6% at most:

WhereLayerMaximum value (Herbert / reference)
input of mixer_proj (output of Mamba-3)12.94e16 / 2.93e16
input of mixer_proj74.75e15 / 4.71e15
input of mixer_proj231.41e12 / 1.33e12
output of the experts' up_proj03.04e8 / 3.04e8
stream running through the modelall0.67 at most

Layers are numbered from 0. No infinite value. bf16 represents them all; f16, whose maximum is 65,504, represents none of the first four. These are the layers that OVH's report lists in its appendix C. OVH explains it with a lot of honesty: some Mamba-3 heads got stuck at the bound of their forgetting and no longer forget anything. Their readout becomes a sum over the whole text, which grows with the position. The rotation of the stream does the rest, as I explained above: the value costs nothing in training and never goes away.

For Llama.cpp, OVH actually withdrew its 4-bit version: the card of its Llama.cpp file says that a Q4_K_M file was built, tested, then withdrawn. On the MLX side, there is a 4-bit version. The huge values do not explain everything: fc_2, which serves the shared expert, only receives values below 2.4 and breaks the text anyway. I kept these four families wider, in 16 or 8 bits depending on the engine. The rest is quantized. This is not a simple model to quantize. OVH says so itself in the model card: generic quantization recipes produce NaN, values that are no longer numbers, on realistic inputs. It advises using its files or following its recipe.

The Intel driver's sine

On the Arc 140V of an Intel laptop, under Linux, a single one of my tests failed, by very little. It compares computing the Mamba-3 state in chunks with computing it token by token. The gap was 1.13e-3 for a bound of 1e-3. Everywhere else, it passed with room to spare.

I probed the driver's sine and cosine on 65,536 points between −pi and pi, against a 64-bit computation on the CPU. The maximum error is 3.40e-5 on the Arc, against 3.3e-7 to 3.8e-7 on the RTX 4080. That is a hundred times less precise, but it stays within the tolerance the Vulkan specification allows. With a sine computed by Herbert itself, the test passes on the Arc with a gap of 1.14e-4. So the bound was not too tight.

Now, at load time, Herbert measures the precision of the card's sine and cosine. If the error exceeds 1e-6, it computes them itself. The Arc takes that path; the other cards keep the driver's and give the same results as before.

A first duel lost

When everything finally ran correctly, I held a first duel against OVH's engines. They were faster almost everywhere. That is normal: they have been working on their model much longer than I have. So I optimized, engine by engine, measuring every attempt. Herbert does rather well, even if it is not ahead everywhere. The numbers will be in the next issue.

What it can do: first tests

I also started looking at what it is worth on two public tests. BFCL, the Berkeley bench, measures tool calls: calling the right function with the right arguments. IFEval, from Google, measures instruction following: write fewer than 100 words, without commas, with a title, etc.

OVH presents Olala as a small reasoning model made for conversation, instruction following and tool calls, specifically trained to call tools. So I wanted to see what it does against other models of its size. It is also a way of checking my implementation on real tasks, beyond my comparisons with the reference. OVH also publishes its own BFCL scores, on all 23 categories (4,981 questions), with high reasoning effort: 59.6 for Olala and 69.7 for LFM2.5-8B-A1B. My first 20 tests per category are not comparable with those numbers, but the gap goes in the same direction.

I put it against two models of similar size: Gemma 4 E4B from Google and LFM2.5-8B-A1B from Liquid AI, a mixture of experts like Olala (8 billion parameters, about 1 billion active).

To move fast, I focused on the first 20 tests of each category, the same for all three models. Each model is in its best conditions: in 16 bits (bf16), with the settings recommended by its maker, 3 different draws per test and enough room to finish reasoning (16,384 tokens). No answer reached that limit. All three run on the same Mac Studio (M3 Ultra), with Herbert, on November 22, 2026: Olala's server on one side, the server for the two other models on the other.

As a percentage of correct answers, mean of the 3 draws:

OlalaGemma 4 E4BLFM2.5-8B-A1B
Calling the right tool with the right arguments (4 BFCL categories)749192
Not calling a tool when none fits10010097
Real requests sent by users (2 "live" categories)717881
Following instructions (IFEval, strict)978892

The detail, in correct answers out of 20, mean of the 3 draws:

CategoryOlalaGemma 4 E4BLFM2.5-8B-A1B
a single call (simple_python)16.319.017.3
choosing among several tools (multiple)16.318.319.0
several calls at once (parallel)14.018.318.3
several calls among several tools (parallel_multiple)12.717.319.0
no tool fits (irrelevance)20.020.019.3
real request, one tool (live_simple)17.016.017.7
real request, several tools (live_multiple)11.315.314.7
IFEval19.317.718.3

Olala follows instructions very well and knows how to say no when no tool fits. On the other hand, it falls behind when it has to call several tools at once: that is where it thinks the longest, 1,671 tokens per answer on average on parallel and 1,429 on parallel_multiple, against 341 to 590 for the other two. Of its 420 BFCL answers, 22 could not be read as a tool call: 18 are text answers, where it solves the problem itself or asks a question back. The other 4 are calls whose arguments are not valid JSON.

And speed? This model is hard to run, so it matters. For whoever is waiting, the time of an answer is the number of tokens written multiplied by the time of each one:

OlalaGemma 4 E4BLFM2.5-8B-A1B
Tokens written per answer, reasoning included993397551
Decoding, in tokens per second158 to 16353 to 57110 to 148
Mean time per answer6.5 s7.4 s4.9 s

Olala writes 2.5 times more tokens than Gemma 4 E4B, but Herbert decodes it about 3 times faster. Its questions are also longer, 1.6 to 7 times depending on the category: its chat template adds a lot of text around the question, even more when it describes tools. It reads them at about 3,700 tokens per second, which keeps the first token between 0.1 and 0.4 seconds. These are the speeds of my engine on this machine, not those of the models in general.

Before this run, I had done a first one on the same questions, with Olala in 4 bits, in deterministic generation and with a limit of 2,048 tokens (4,096 for IFEval). 25 answers out of 160 were cut off at the limit. In the run above, none. Olala gains in every category except irrelevance, already at 20. Of those 25 questions, 16 to 18 now pass depending on the draw. But I changed three things at once (16 bits, the maker's settings and the room left for reasoning): I don't know yet which one matters. Gemma 4 E4B and LFM2.5 move by one or two answers per category.

Limits

20 tests per category is little: one test is worth 5 points. On IFEval and on live_simple, the ranges between draws overlap from one model to the next: these are not differences.

I do not apply the repetition penalty recommended for LFM2.5 (1.05): Herbert does not have it yet. Its 5 answers over 4,096 tokens might be shorter with it.

The Gemma 4 E4B tested is the checkpoint Google published in bf16 after training meant for 4 bits (QAT), with thinking enabled. LFM2.5 always reasons; its template has no switch.

Herbert's server reuses the start of the questions from one request to the next for Gemma 4 E4B and LFM2.5, not for Olala. The reading speeds leave those answers out; the time per answer benefits from it.

The scores come from the public graders: bfcl-eval 2026.3.23 for BFCL and Google Research's for IFEval, on the answer text only. Herbert's engine for Olala is not public for now.

The papers

The starting point is OVH's technical report, released with the model: Olala v1.0 mini Technical Report (OVHai LLM Research Team, 2026). It is fascinating and a bold bet, to say the least, by OVH's teams. I love it, and I wish I had more time to go over all of it with a scalpel.

Mamba-3 (marker 2):

Attention (marker 3):

The experts (marker 5):

The sphere (markers 1 and 4):

Beyond the architecture: Bruno Mlodozeniec, Pierre Ablin, Louis Béthune et al., Completed Hyperparameter Transfer across Modules, Width, Depth, Batch and Duration, December 26, 2025, for carrying the settings from the small bench to the full model.

The benches: BFCL (Berkeley) and IFEval (Google, 2023).

What's next

In the next issue, Herbert against OVH's engines, on their own model. The measurements engine by engine, and the code.

And in parallel, the tests go on: the first 100 tests of each category, with Liquid AI's LFM2 and LFM2.5 and Gemma 4 E4B, on several CPUs and graphics cards.

One level down, or several...

  1. One level down: #000 an inference engine that writes its own code · 03-Oct-2026
  2. One level down: #001 a compiler for an LLM engine · 11-Oct-2026
  3. One level down: #002 the same compiler on a GPU · 18-Oct-2026
    1. One level down: #002.1 follow-up, the Arc in f16 · 02-Nov-2026
  4. One level down: #003 what CUDA has that Vulkan lacks · 25-Oct-2026
  5. One level down: #004 the same compiler on Apple's chips · 01-Nov-2026
  6. One level down: #005 Is Herbert better than Llama.cpp? · 15-Nov-2026
  7. One level down: #006 I started porting Olala, OVHcloud's model · 22-Nov-2026