The context window is not memory

A bigger window does not make the model remember more

Three years ago a context window was measured in thousands of tokens. Today the flagship model of every major vendor sits at a million. It sounds like the model grew a memory.

It did not. A bigger window means you may send more in each call. The model still keeps nothing between calls — that was the last post. The window only decides how much fits in one shot.

And the window is a budget, not a warehouse. Run out of budget and you lose data, in one of two ways. One of those two ways is far worse.

Four numbers, and one of them nobody can measure

No API key needed to see the problem. The script lives at docs/research/measure-context-budget.mjs in this repo.

The first measurement: the same content, written three ways.

Diacritics removed: 169 chars → 169 chars (unchanged), but 230 bytes → 169 bytes (down 27%).

{"labels": ["Vietnamese", "Vietnamese, no diacritics", "English"], "series": [{"name": "Characters", "values": [169, 169, 190]}, {"name": "UTF-8 bytes", "values": [230, 169, 190]}]}

The second measurement: how many turns fit in one context window. Each turn is a full request — system prompt, tool declarations, history and tool results.

The vi-no-diacritics row is the one to slow down on. It is the exact same Vietnamese sentence, with the diacritics removed. The character count does not move: 169 before, 169 after. The byte count drops from 230 to 169.

So the extra bytes do not come from a longer sentence. They come from UTF-8 encoding. Identical content, identical character count, 27% more bytes.

And the limit of this measurement has to be stated: it measures bytes, not tokens. A tokenizer does not divide by bytes.

Why 1.6 is not a discovery

The two token columns above come from two fixed ratios: the low column uses 4 characters per token, and the high column uses 2.5.

Because both are fixed ratios, dividing one by the other always gives 1.6 — for every text and language. That 1.6 tells you nothing about your data. It simply shows that the two most common estimates sit 60% apart.

What matters is not the number 1.6. It is that both estimates are real, and both run in production right now.

Where the two numbers come from:

• 4 characters per token is the heuristic from dsh-token-meter. Its own docs warn against it: “CJK text and JSON schemas underprice badly at four characters per token”. Google Gemini publishes the exact same rule: “a token is equivalent to about 4 characters”.
• 2.5 characters per token comes from Anthropic’s docs: “1M tokens is roughly 555k words or 2.5M Unicode characters”.

And OpenAI warns against this whole family of shortcuts: “estimates like characters / 4 are inaccurate”.

Three vendors give three different numbers, and none of them is wrong. That is the real puzzle.

Three layers of one budget

type: layers
title: Three layers of the context budget
desc: From the window size published by the vendor, through heuristic estimate gaps, to the actual budget you trust
nodes:
  - id: contract
    label: Contract layer
    sub: Window size published by vendor
    kind: external
  - id: estimate
    label: Estimation layer
    sub: 1.6x gap between ÷4 and ÷2.5
    kind: optional
    focal: true
  - id: decision
    label: Decision layer
    sub: Safe budget you actually trust
    focal: true
edges:
  - [contract, estimate]
  - [estimate, decision]

The contract layer: the window is a number set by the vendor

DeepSeek publishes a 1M window and 384K maximum output for deepseek-flash and deepseek-v4-pro. Anthropic keeps Haiku 4.5 at 200K while newer models have 1M. These are different limits for the exact same concept.

The practical impact: changing models changes your budget. A prompt that fits one model can easily overflow another, and no type checker will warn you.

The estimation layer: you cannot measure what you need

To get an accurate number, you must run the exact tokenizer of the model you call. There is no shortcut.

Even with a tokenizer, the number remains an estimate. Anthropic’s docs state clearly that “The token count is an estimate”. They also warn that a newer tokenizer creates “approximately 30 percent more tokens than on earlier models” for the exact same text. That is a 30% jump just from updating the tokenizer.

The decision layer: you must choose what to believe

This is where the real cost appears. If you estimate too low, your request exceeds the window and fails immediately. If you estimate too high, you trim context too early and throw away facts the model still needs.

These two failures are not equal: estimating too low fails loudly with clear errors; estimating too high fails quietly, and you only notice when the model gives a wrong answer because facts were missing.

Do this now

1. Pin one number per modelWrite down the real window of every model you call, taken from vendor documentation rather than memory. Update it when you change models.
2. Measure, do not guessRun the vendor’s count_tokens endpoint or tokenizer on the actual payload you send. OpenAI, Anthropic and Gemini all ship a counting endpoint.
3. Budget for what you did not writeTool declarations and the system prompt occupy space too. Measure the whole envelope, not just the user’s question.

The third step is the one people skip. A real request is not a question — it is a system prompt plus tool declarations plus history plus tool results. In my measurement that was 1,289 characters, and none of it was typed by a user.

The rule

A context window is a budget, and everything you put in it is taking space from something else.

A context window does not give the model more memory. It gives you more room to decide what is worth remembering.— tusgino

• Pin the real window of each model from vendor docs; changing models changes the budget and nothing in the type system warns you.
• Run a token-count endpoint over the whole payload, not just the user’s question — system prompt and tool declarations occupy space too.
• Never convert bytes to tokens by hand. If you must estimate, state which ratio you used, because the published ratios sit 1.6x apart.
• Prefer over-estimating to under-estimating: trimming early loses context, but overflowing loses the whole request.
• If your system trims context, log the event. An unlogged trim is a wrong answer nobody can trace.