Call the API, then forget

The paradox of the second call

Open a chat window and tell the model your name. It answers with the right name. Close the window, open a new one, ask again. It has no idea who you are.

Most people meet this paradox and explain it wrong. They assume the model remembers inside a session and forgets between sessions. The truth is less comfortable: the model remembers nothing at all, including inside a single session.

The illusion lives in the client. Every time you type another sentence, the client resends the entire conversation from the first message. The model re-reads all of it from zero, then answers the newest line.

Not guesswork. DeepSeek’s docs say it outright:

“The DeepSeek /chat/completions API is a ‘stateless’ API, meaning the server does not record the context of the user’s requests. Therefore, the user must concatenate all previous conversation history and pass it to the chat API with each request.”

Three sentences, and the second one is this entire post. The API reference describes the messages field in exactly one line: “A list of messages comprising the conversation so far.”

One exception needs stating: not every API is stateless.

Google’s Interactions API currently manages conversation state on the server. You turn it on with previous_interaction_id, and opt out with store=false. “LLM APIs have no memory” has become a design choice, not an absolute rule.

And this is a real architectural trade-off: the stateless approach costs more bandwidth on every turn, but gives you full control to inspect and replay history. The server-side stateful approach keeps payloads small, but the price is that the conversation state and its lifecycle sit outside your control.

// No API key here. This stub exists to expose the data structure.
type Message = { role: 'user' | 'assistant'; content: string };

declare function callModel(messages: Message[]): Promise<string>;

const history: Message[] = [];

async function ask(text: string): Promise<string> {
  history.push({ role: 'user', content: text });

  // Note this line: it sends the WHOLE array, not just the new sentence.
  const reply = await callModel(history);

  history.push({ role: 'assistant', content: reply });
  return reply;
}

Both history.push calls are required. Drop the second one and the next turn answers as if it never heard the previous reply.

type: sequence
title: Two API calls, the second one sending full history
desc: The first call sends one question; the second must resend the whole history so the model understands the context
nodes:
  - id: app
    label: Your App
    kind: input
    focal: true
  - id: model
    label: LLM API
    kind: external
edges:
  - [app, model]
  - [model, app]
  - [app, model]
  - [model, app]

Three numbers that quantify the paradox

You do not need an API key to see the price. Both measurements run on node, and the script lives at docs/research/measure-resend-cost.mjs in this repo.

The first measurement: the same content, written in two languages.

Vietnamese costs 36% more bytes for the same information.

The second measurement: resending history on every turn, assuming equal-length turns.

{"labels": ["3 turns", "6 turns", "10 turns", "20 turns"], "series": [{"name": "Total sent", "values": [6, 21, 55, 210]}, {"name": "If sent once", "values": [3, 6, 10, 20]}]}

The first and third figures are byte measurements: deterministic, independent of any model. The second is arithmetic, not a measurement — it is just the sum $1 + 2 + \dots + N$.

Its closed form is suspiciously tidy:

$\text{total sent} = \frac{N(N+1)}{2}$

At 6 turns you have paid 3.5x. At 20 turns, 10.5x. There is no threshold in that curve — cost grows quadratically while the value received does not.

What you call memory is a variable

The tightest definition of an LLM call: a pure function that turns an array of messages into a string. It reads no files, holds no session, and does not recall the previous request.

The consequence is that everything you call “memory” sits on your side. Specifically, in an array variable, a database table, or a log file.

type: flowchart
title: History is yours to own, not the model's
desc: The application holds the history array; every turn resends the whole array; the model only sees that array and keeps nothing after replying
nodes:
  - id: user
    label: User types
    kind: input
  - id: store
    label: History array
    kind: store
    focal: true
  - id: model
    label: Model replies
  - id: reply
    label: Write the reply
    kind: store
edges:
  - [user, store]
  - [store, model]
  - [model, reply]
  - [reply, store]

The second option is not free. It turns a UI detail into a data problem. How long may the array grow, what gets cut when it is too long, and where the original record lives.

Three layers of the same fact

The function layer: a request carries no state

A request has a model, an array of messages, and a few generation parameters. No session field, no conversation id, no memory field.

Two identical requests therefore produce near-identical output, no matter how many calls came before. The model accumulates nothing over time.

The application layer: the history array is your database

By now the shape is clear. The history array from the first snippet is not a convenience variable. It is the single source of truth for where the conversation has been.

And because it is a source of truth, it inherits every problem one has. Concurrent writes, crash recovery, schema changes, and the question of who may edit it.

The cost layer: language sets the price

The first measurement deserves a longer pause. For the same content, Vietnamese takes 1.36 bytes per character while ASCII English takes exactly 1.00.

Every byte-based budget is tighter for a Vietnamese writer. Not because Vietnamese is wordier, but because Vietnamese diacritics need more bytes in UTF-8.

I have to state this measurement’s limit plainly: it measures bytes, not tokens. A token is whatever the model’s tokenizer decides, and characters-per-token differs between models. A real token count needs that exact tokenizer. The 36% is evidence of direction, not a conversion factor.

Do this now: make history owned data

If the model is a pure function, the only place that can be wrong is the data you send in. Three steps, doable in one sitting.

1. Pick one source of truthDecide that exactly one place stores history. If the client and the server each keep a copy today, pick one and delete the other.
2. Write before you callPersist the user’s message to that store before calling the model. If the process dies mid-turn, you still know what was asked.
3. Count what you sendLog the size of every request. That is the one number that tells you whether resend cost is growing fast or slow.

The second step is the one people skip, and it is the cheapest. One write before the model call turns “why did it answer that” from a guess into a lookup.

One trap: user_id is not memory

The schema has a field named user_id, and it reads far too easily as “the handle that makes the model remember you”. The docs list three purposes: content safety review, KVCache isolation, and scheduling isolation. None of them is conversation memory.

The point matters because prompt caching is routinely mistaken for state. A cache is a performance optimisation for an already-computed prefix, not application state. It does not change the contract: you still send the full messages.

The response’s usage object says so. prompt_tokens equals prompt_cache_hit_tokens plus prompt_cache_miss_tokens, so the cached portion was still in the prompt you sent. A cache makes it cheaper; it does not remove the cost.

The rule

The model does not remember. You do. Everything you call context is data you chose to send, and you pay again for every resend of it.

The model remembers nothing at all. What you call memory is the data you choose to resend.— tusgino

• Persist the user’s message before calling the model — one write, and it changes how you debug.
• Keep history in one source of truth; two copies at two layers are two copies that will drift.
• Remember $\frac{N(N+1)}{2}$ when estimating cost: by 20 turns you have paid 10.5x the final history length.
• Do not convert bytes into tokens by hand. Byte measurement is deterministic; a real token count needs the model’s own tokenizer.
• For Vietnamese, the byte budget is 36% tighter than English for the same content — set your truncation threshold from your own measurement.