We were charging every model up to 7× too much for memory it never used

To run a language model on someone's laptop you have to answer a question the model cannot answer for you: how much context can this machine actually hold?

Get it wrong in one direction and you promise a window that runs out of memory halfway through loading. Get it wrong in the other and you quietly hand people a fraction of the machine they paid for. We were doing the second one, by up to a factor of seven, for every model we ship.

The formula was right in 2022

Our sizer priced the key-value cache like this:

kvCacheMB = nCtx * nLayers * nEmbd * 2 * bytesPerElement

Context length, times layers, times the embedding width, times two — one key and one value — times the size of each number. Read it out loud and it sounds obviously correct: every token needs a key and a value in every layer, and those are as wide as the model's embedding.

That is the pre-GQA shape, and it has been wrong for every model we ship for about two years.

Grouped-query attention changed the width. Many query heads now share a small number of key/value heads, and the cache is sized by the KV heads — not by the embedding. Qwen2.5-Coder 1.5B has 12 query heads and 2 KV heads at 128 wide. So the real per-token, per-layer width is:

nKvHeads * headDim  =  2 * 128  =  256

We were using nEmbd, which is 1536. Six times too wide, on that model, before multiplying by layers and context.

What it cost, measured

Priced at a 32k window against each model's published architecture:

Model	Old estimate	Actual	Over-count
Qwen2.5 Coder 0.5B	2819 MB	403 MB	7.0×
Qwen2.5 Coder 1.5B	5637 MB	940 MB	6.0×
Llama 3.2 1B	4295 MB	1074 MB	4.0×
SmolLM2 360M	4027 MB	1342 MB	3.0×

A 0.5B model — the one we offer to the weakest machines, the one whose entire appeal is that it fits anywhere — was being quoted 2.8 GB for a cache that needed 403 MB.

Over-counting is not the safe direction

This is the part that gets reasoned about backwards, including by us for longer than it should have been.

The instinct is that over-estimating memory is conservative. You reserve too much, you're cautious, nobody crashes. And it's true that nobody crashes. What happens instead is invisible, which is worse.

The sizer walks a ladder of context lengths and returns the largest one that fits the budget. Feed it an inflated cache cost and it steps down the ladder — and refuses models that would have run comfortably. On the shipped 1.5B:

Device	Window offered before	After
4 GB	6,144 tokens	32,768 tokens
8 GB	16,384 tokens	32,768 tokens

A 4 GB machine was being offered a quarter of the context it could actually hold. Nobody saw an error. They saw an agent that forgot the top of the file, or a model marked "won't fit" on a laptop that would have run it fine. Those read as the model is weak or this device is too small — conclusions about the product, caused by a bug in a spreadsheet.

A crash is a bug report. A silently smaller window is a worse product with no bug report attached.

What we deliberately did not fix

The type change alone would have shipped nothing — a formula that can use real dimensions still needs models that carry them. So the open GGUF rows now carry their true nKvHeads and headDim.

Our own three ternary models do not, on purpose. Their architectures aren't published here, and a guess dressed up as a measurement would be worse than an honest fallback. They keep the old nEmbd formula: it over-counts, which is wrong but safe in the sense that matters — it cannot promise a window that fails to load.

That leaves a fallback path sitting next to a precise path, which is exactly the shape that rots. Two guards keep it honest, and both were mutation-tested — we broke each one deliberately and confirmed the test went red:

A half-specified model falls back rather than defaulting the missing half to 1. If a row carries nKvHeads but not headDim, the tempting move is to substitute a 1 and carry on. That is arithmetic on an unknown, and it under-counts — which promises a window that runs out of memory on load. Half a specification is not a specification.

Zero or negative dimensions are ignored, not trusted. A catalog row arriving from the control plane with a 0 would otherwise price the cache at nothing and promise the model's full trained context on a machine that cannot hold it. Zero is not a measurement; it is a missing field wearing a number's clothes.

Both guards exist because the catalog is served live, so a future row we have never seen can reach this code. The failure has to be safe for rows that do not exist yet.

The general shape

The formula wasn't wrong when it was written. The field moved underneath it, and the code kept producing plausible numbers the whole time — never absurd, never zero, never large enough to look broken. Just three to seven times too big, quietly, for years.

Numbers that stay plausible while becoming wrong are the hardest kind to notice. The only thing that catches them is checking the arithmetic against the thing it claims to describe — in this case, reading the actual attention shapes out of the model files instead of trusting a constant that used to be right.

---

Shipped in 1.0.12. The over-count figures are computed against each model's published architecture at a 32k window.