---
title: "Call the API, then forget"
description: "Every API call is a pure function: text in, text out, then nothing. The memory you think the model has is history you resend yourself, and you pay for it."
pubDate: 2026-09-19
author: tusgino
tags: [ai-engineering, architecture, llm]
series:
  id: tu-llm-den-harness
  title: "From LLM to Harness"
  order: 1
cover: /images/covers/mot-lan-goi-api-roi-quen-het.svg
coverAlt: "Two separate API calls, where the second one carries the entire history of the first"
lang: en
faq:
  - q: "If the API has no memory, why does a chat interface look like it remembers?"
    a: "Because the chat interface resends the whole conversation in every request. The model does not remember; the client remembers for it and sends the record along. That is why a longer conversation costs more tokens on every turn."
  - q: "How much does resending history cost?"
    a: "It grows quadratically. Over N equal-length turns the total sent is N(N+1)/2 units, so at 20 turns you have paid 10.5x the final history length. That figure is arithmetic, not a measurement, and it holds for any API that keeps no state."
  - q: "Does Vietnamese cost more tokens than English?"
    a: "For the same content, Vietnamese takes 1.36 bytes per character against 1.00 for English — 36% more bytes. That is a byte measurement, not a token measurement: real token counts need the tokenizer of the model you actually call."
---

<TLDR>Most chat APIs keep no memory between calls by default: every request is a pure function, and the memory you think the model has is history you resend yourself — at a cost that grows quadratically.</TLDR>

## The paradox of the second call

Open a chat window and tell the model your name. It answers with the right name. Close the window, open a new one, ask again. It has no idea who you are.

Most people meet this paradox and explain it wrong. They assume the model remembers inside a session and forgets between sessions. The truth is less comfortable: **the model remembers nothing at all, including inside a single session.**

The illusion lives in the client. Every time you type another sentence, the client resends the **entire** conversation from the first message. The model re-reads all of it from zero, then answers the newest line.

Not guesswork. [DeepSeek's docs](https://api-docs.deepseek.com/guides/multi_round_chat) say it outright:

> "The DeepSeek `/chat/completions` API is a 'stateless' API, meaning the server does not record the context of the user's requests. Therefore, the user must concatenate all previous conversation history and pass it to the chat API with each request."

Three sentences, and the second one is this entire post. The [API reference](https://api-docs.deepseek.com/api/create-chat-completion) describes the `messages` field in exactly one line: "A list of messages comprising the conversation so far."

One exception needs stating: **not every API is stateless.**

Google's Interactions API currently manages conversation state on the server. You turn it on with `previous_interaction_id`, and opt out with `store=false`. "LLM APIs have no memory" has become a **design choice**, not an absolute rule.

And this is a real architectural trade-off: the stateless approach costs more bandwidth on every turn, but gives you full control to inspect and replay history. The server-side stateful approach keeps payloads small, but the price is that **the conversation state and its lifecycle sit outside your control**.

```ts title="the-illusion.ts"
// No API key here. This stub exists to expose the data structure.
type Message = { role: 'user' | 'assistant'; content: string };

declare function callModel(messages: Message[]): Promise<string>;

const history: Message[] = [];

async function ask(text: string): Promise<string> {
  history.push({ role: 'user', content: text });

  // Note this line: it sends the WHOLE array, not just the new sentence.
  const reply = await callModel(history);

  history.push({ role: 'assistant', content: reply });
  return reply;
}
```

Both `history.push` calls are required. Drop the second one and the next turn answers as if it never heard the previous reply.

```diagram
type: sequence
title: Two API calls, the second one sending full history
desc: The first call sends one question; the second must resend the whole history so the model understands the context
nodes:
  - id: app
    label: Your App
    kind: input
    focal: true
  - id: model
    label: LLM API
    kind: external
edges:
  - [app, model]
  - [model, app]
  - [app, model]
  - [model, app]
```

## Three numbers that quantify the paradox

You do not need an API key to see the price. Both measurements run on `node`, and the script lives at `docs/research/measure-resend-cost.mjs` in this repo.

The first measurement: the same content, written in two languages.

| Language | Characters | Bytes | Bytes per character |
|---|---|---|---|
| Vietnamese | 169 | 230 | **1.361** |
| English | 190 | 190 | **1.000** |

Vietnamese costs **36%** more bytes for the same information.

The second measurement: resending history on every turn, assuming equal-length turns.

| Turns | History that turn | Total sent | Multiple |
|---|---|---|---|
| 3 | 3 | 6 | 2x |
| 6 | 6 | 21 | **3.5x** |
| 10 | 10 | 55 | 5.5x |
| 20 | 20 | 210 | **10.5x** |

```chart title="Resent history: what you send vs sending it once" type="bar"
{"labels": ["3 turns", "6 turns", "10 turns", "20 turns"], "series": [{"name": "Total sent", "values": [6, 21, 55, 210]}, {"name": "If sent once", "values": [3, 6, 10, 20]}]}
```

<StatRow stats={[
  { value: "10.5x", label: "Paid by turn 20", delta: "Against the final history length", trend: "up" },
  { value: "+36%", label: "Vietnamese bytes vs English", delta: "For the same content", trend: "flat" },
  { value: "1.00", label: "Bytes per ASCII character in English", delta: "The baseline", trend: "flat" }
]} />

The first and third figures are byte measurements: deterministic, independent of any model. The second is **arithmetic**, not a measurement — it is just the sum $1 + 2 + \dots + N$.

Its closed form is suspiciously tidy:

$$\text{total sent} = \frac{N(N+1)}{2}$$

At 6 turns you have paid 3.5x. At 20 turns, 10.5x. There is no threshold in that curve — cost grows quadratically while the value received does not.

## What you call memory is a variable

The tightest definition of an LLM call: **a pure function that turns an array of messages into a string.** It reads no files, holds no session, and does not recall the previous request.

The consequence is that everything you call "memory" sits on your side. Specifically, in an array variable, a database table, or a log file.

```diagram
type: flowchart
title: History is yours to own, not the model's
desc: The application holds the history array; every turn resends the whole array; the model only sees that array and keeps nothing after replying
nodes:
  - id: user
    label: User types
    kind: input
  - id: store
    label: History array
    kind: store
    focal: true
  - id: model
    label: Model replies
  - id: reply
    label: Write the reply
    kind: store
edges:
  - [user, store]
  - [store, model]
  - [model, reply]
  - [reply, store]
```

<Versus aTitle="Treat the model as a stateful entity" bTitle="Treat the model as a pure function">
  <div slot="a">
    <ul>
      <li>You believe the model remembers between turns.</li>
      <li>You cannot tell what is actually inside the context.</li>
      <li>When it answers wrong, there is no record to read back.</li>
      <li>History scatters across clients; nobody owns the truth.</li>
    </ul>
  </div>
  <div slot="b">
    <ul>
      <li>History is application data, with a schema and an owner.</li>
      <li>Each turn is a transform you can read and audit.</li>
      <li>Inserting or cutting a message is a controlled operation.</li>
      <li>The cost: you own the lifecycle of that array.</li>
    </ul>
  </div>
</Versus>

The second option is not free. It turns a UI detail into a data problem. How long may the array grow, what gets cut when it is too long, and where the original record lives.

## Three layers of the same fact

### The function layer: a request carries no state

A request has a model, an array of messages, and a few generation parameters. No session field, no conversation id, no memory field.

Two identical requests therefore produce near-identical output, no matter how many calls came before. The model accumulates nothing over time.

### The application layer: the history array is your database

By now the shape is clear. The `history` array from the first snippet is not a convenience variable. It is the **single source of truth** for where the conversation has been.

And because it is a source of truth, it inherits every problem one has. Concurrent writes, crash recovery, schema changes, and the question of who may edit it.

### The cost layer: language sets the price

The first measurement deserves a longer pause. For the same content, Vietnamese takes 1.36 bytes per character while ASCII English takes exactly 1.00.

**Every byte-based budget is tighter for a Vietnamese writer.** Not because Vietnamese is wordier, but because Vietnamese diacritics need more bytes in UTF-8.

I have to state this measurement's limit plainly: it measures **bytes**, not **tokens**. A token is whatever the model's tokenizer decides, and characters-per-token differs between models. A real token count needs that exact tokenizer. The 36% is evidence of direction, not a conversion factor.

## Do this now: make history owned data

If the model is a pure function, the only place that can be wrong is the data you send in. Three steps, doable in one sitting.

<Steps>
<Step title="Pick one source of truth">Decide that exactly one place stores history. If the client and the server each keep a copy today, pick one and delete the other.</Step>
<Step title="Write before you call">Persist the user's message to that store **before** calling the model. If the process dies mid-turn, you still know what was asked.</Step>
<Step title="Count what you send">Log the size of every request. That is the one number that tells you whether resend cost is growing fast or slow.</Step>
</Steps>

The second step is the one people skip, and it is the cheapest. One write before the model call turns "why did it answer that" from a guess into a lookup.

<Callout type="warn" title="The trap of a stateless design">
Because a request carries no state, the model cannot tell "I was never told the name" from "I was told, but the application forgot to send it". From the model's side those are the same situation. Every memory bug in your system is a data bug, never a model bug.
</Callout>

## One trap: `user_id` is not memory

The schema has a field named `user_id`, and it reads far too easily as "the handle that makes the model remember you". The docs list three purposes: content safety review, **KVCache isolation**, and scheduling isolation. None of them is conversation memory.

The point matters because prompt caching is routinely mistaken for state. **A cache is a performance optimisation for an already-computed prefix, not application state.** It does not change the contract: you still send the full `messages`.

The response's `usage` object says so. `prompt_tokens` equals `prompt_cache_hit_tokens` plus `prompt_cache_miss_tokens`, so the cached portion **was still in the prompt you sent**. A cache makes it cheaper; it does not remove the cost.

## The rule

The model does not remember. You do. Everything you call context is data you chose to send, and you pay again for every resend of it.

<Quote cite="tusgino">The model remembers nothing at all. What you call memory is the data you choose to resend.</Quote>

<KeyTakeaways>
- Persist the user's message **before** calling the model — one write, and it changes how you debug.
- Keep history in one source of truth; two copies at two layers are two copies that will drift.
- Remember $\frac{N(N+1)}{2}$ when estimating cost: by 20 turns you have paid 10.5x the final history length.
- Do not convert bytes into tokens by hand. Byte measurement is deterministic; a real token count needs the model's own tokenizer.
- For Vietnamese, the byte budget is 36% tighter than English for the same content — set your truncation threshold from your own measurement.
</KeyTakeaways>