Conversation State and Context Windows

A language model does not remember anything between calls. Every time an agent talks to the model, the agent must send everything the model needs to know. This lesson explains conversation state (what you send), the context window (how much you can send), and how to keep an agent working when a long task no longer fits.

Overview: how conversation state works

An LLM API is stateless. It takes an input, produces an output, and forgets both. The feeling of a “conversation” is an illusion created by your code. Your program stores a list of messages and sends the whole list again on every call.

This stored list is called the conversation state (or message history). A typical message has a role and some content:

• system – instructions that set the agent’s behaviour, rules and permissions.
• user – what the human (or your program) says.
• assistant – what the model replied, including any request to call a tool.
• tool (the exact name varies by provider) – the result of a tool call that you send back to the model.

An agent loop adds to this list at each step: the user’s goal, the model’s decision, the tool result, the next decision, and so on. So the state grows with every step. An agent that calls ten tools has ten or more extra messages in its history, and tool results (web pages, file contents, database rows) can be very large.

Tokens and the context window

Models do not read characters or words. They read tokens: small pieces of text, often part of a word. In English, a token is roughly four characters, but this differs by language and by model. Code, JSON and non-English text often use more tokens per character.

The context window is the maximum number of tokens the model can handle in one call. Both your input (system prompt, history, tool definitions, tool results) and the model’s output must fit inside it. Some providers also set a separate cap on output length. Window sizes differ between models and change over time, so always check your provider’s current documentation instead of memorising a number.

When the input is too large, one of three things happens: the API returns an error, the provider truncates your input, or the model gets worse at using information buried in the middle of a very long prompt. Long context also costs more money and adds latency, because you pay for the input tokens again on every call. This is why state management is a core agent skill.

Key concepts and syntax

Provider SDKs differ in small details, but the shape of the state is almost always a list of role/content objects:

[
  {"role": "system", "content": "You are a careful support agent."},
  {"role": "user", "content": "Where is my order 1234?"},
  {"role": "assistant", "content": "Let me look that up."}
]

Key ideas to remember:

• Append-only by default. New messages go at the end. Never edit past messages silently; the model trusts them as the true record.
• Token budget. Decide how many tokens of history you allow (well below the window), leaving room for the reply.
• Trimming. Drop the oldest messages when you exceed the budget.
• Summarising (compaction). Replace old messages with a short summary written by the model.
• Pinned content. Some messages, such as the system prompt, must never be dropped.
• External memory. Save facts outside the window (a file or database) and load them only when needed. Later lessons cover this in detail.

Examples

Example 1: state is just a list you resend

We use a fake call_llm() so the code runs without a network. To use a real model, replace the body with your provider’s SDK call, pass messages to it, and return the reply text.

def call_llm(messages):
    # Stub model. Swap this body for a real SDK call.
    last = messages[-1]['content']
    seen = [m['content'] for m in messages if m['role'] == 'user']
    return f'I have seen {len(seen)} user message(s). Last: {last}'


history = []
for text in ['My name is Asha.', 'What is my name?']:
    history.append({'role': 'user', 'content': text})
    reply = call_llm(history)
    history.append({'role': 'assistant', 'content': reply})
    print(reply)

Output:

I have seen 1 user message(s). Last: My name is Asha.
I have seen 2 user message(s). Last: What is my name?

The second call can “remember” the first only because we appended both the user message and the reply to history and sent the whole list. If you passed only the new message, the model would have no idea who Asha is.

Example 2: trimming history to a token budget

Real tokenizers are provider-specific, so this example uses a rough estimate of four characters per token. The function keeps system messages, drops the oldest other messages until the history fits, and then makes sure the history starts with a user message (some APIs reject a conversation that starts with an assistant turn).

def estimate_tokens(text):
    # Rough rule of thumb: about 4 characters per token in English.
    return max(1, len(text) // 4)


def count(messages):
    # +4 approximates the per-message overhead.
    return sum(estimate_tokens(m['content']) + 4 for m in messages)


def trim_history(messages, budget):
    system = [m for m in messages if m['role'] == 'system']
    rest = [m for m in messages if m['role'] != 'system']
    while rest and count(system + rest) > budget:
        rest.pop(0)
    while rest and rest[0]['role'] != 'user':
        rest.pop(0)
    return system + rest


messages = [{'role': 'system', 'content': 'You are a helpful tutor.'}]
for i in range(6):
    role = 'user' if i % 2 == 0 else 'assistant'
    messages.append({'role': role, 'content': 'x' * 40})

print(count(messages))
trimmed = trim_history(messages, 60)
print([m['role'] for m in trimmed])
print(count(trimmed))

Output:

94
['system', 'user', 'assistant']
38

The full history costs 94 estimated tokens, which is over the budget of 60. The loop removes old messages until three remain, then removes the leading assistant message, leaving one user/assistant pair plus the pinned system prompt (38 tokens).

Example 3: summarising instead of forgetting

Trimming loses information. A better approach for long tasks is to compact old turns into a summary and keep only the latest turns word for word. Here summarize() is a stub that truncates text; in a real agent it would call the model with an instruction such as “Summarise the key facts, decisions and open tasks.”

def estimate_tokens(text):
    return max(1, len(text) // 4)


def count(messages):
    return sum(estimate_tokens(m['content']) + 4 for m in messages)


def summarize(previous, new_text):
    # Stub. A real version asks the model to merge both into a short summary.
    return (previous + ' ' + new_text).strip()[:40]


class Conversation:
    def __init__(self, system, budget=60, keep_last=2):
        self.system = system
        self.summary = ''
        self.turns = []
        self.budget = budget
        self.keep_last = keep_last

    def messages(self):
        msgs = [{'role': 'system', 'content': self.system}]
        if self.summary:
            note = 'Summary of earlier conversation: ' + self.summary
            msgs.append({'role': 'system', 'content': note})
        return msgs + self.turns

    def add(self, role, content):
        self.turns.append({'role': role, 'content': content})
        too_big = count(self.messages()) > self.budget
        if too_big and len(self.turns) > self.keep_last:
            self.compact()

    def compact(self):
        old = self.turns[:-self.keep_last]
        self.turns = self.turns[-self.keep_last:]
        text = ' '.join(m['content'] for m in old)
        self.summary = summarize(self.summary, text)


c = Conversation('Be brief.')
for i in range(6):
    role = 'user' if i % 2 == 0 else 'assistant'
    c.add(role, 'x' * 40)

print(len(c.turns), len(c.summary), count(c.messages()))

Output:

2 40 56

After six turns the object holds only the last two turns plus a 40-character summary, and the total stays at 56 estimated tokens, under the budget of 60. The Conversation class owns the rules for state, so the rest of your agent never has to think about it.

Under the hood: what happens each agent step

1. The agent builds the request: system prompt, optional summary, recent turns, tool definitions.
2. It estimates (or counts with the provider’s tool) the input tokens.
3. If the total is over budget, it trims or compacts first.
4. It calls the model. The model reads every token again from scratch; nothing is cached in the model’s “mind”.
5. The reply is appended to the state. If the reply asks for a tool, the tool runs and its result is appended too.
6. The loop repeats until the task is done or a step limit is reached.

Some providers offer prompt caching, which makes repeated identical prefixes cheaper and faster. It still does not change the context-window limit, and the details vary by provider, so read their documentation.

Security and safety of state

Everything in the history is something the model may obey. Tool results such as web pages or emails can contain hidden instructions (prompt injection). When you summarise, you can accidentally turn an injected instruction into a “trusted” summary line. Keep tool output clearly labelled as data, never put it in the system role, and keep approval steps for risky actions (sending email, deleting files, spending money) outside the model’s control. Also avoid storing secrets or personal data in history that you log or send to a provider without need.

Common Mistakes

Mistake 1: trimming away the system prompt

history = history[-4:]  # keeps only the last 4 messages
reply = call_llm(history)

This looks harmless, but the system message is the first item, so it is dropped as soon as the list grows. The agent silently loses its rules and permission limits. Fix: keep system messages separate and pin them, as trim_history() does above.

Mistake 2: splitting a tool call from its result

history.pop(0)  # removes the oldest message, whatever it is
# If it was the assistant's tool request, its tool result is now orphaned.

Most APIs require that every tool result matches an earlier tool request. Removing only one half can cause an API error or confuse the model. Fix: trim or summarise in whole groups (a request plus its result), and never start the history in the middle of a tool exchange.

Mistake 3: sending only the newest message

Beginners often write call_llm([new_message]) and are surprised that the agent forgets everything. Remember: the model has no memory; your list is the memory.

Mistake 4: putting huge tool output straight into history

One large file or web page can use most of the window in a single step. Truncate, extract only the needed fields, or store the full output outside the history and put a short reference or excerpt in the conversation.

Best Practices

• Wrap state in one class or module with add(), messages() and compact(), so rules live in one place.
• Set a budget well below the real window to leave space for the reply and tool definitions.
• Use the provider’s token-counting feature in production; character estimates are only for teaching and rough checks.
• Pin the system prompt, safety rules and the user’s original goal. Never trim these.
• Summarise early rather than at the last moment, and keep the latest few turns verbatim.
• Ask summaries to keep decisions, user preferences, open tasks and important IDs, not chit-chat.
• Truncate or filter large tool results before they enter history.
• Treat tool output and summaries as untrusted data; require human approval for risky actions.
• Add a maximum step count so a looping agent cannot grow its history forever.
• Log token usage per run, so you can see cost and find runaway growth.
• Check your provider’s current window sizes and prices in its official docs; they change often.

Practice Exercises

1. Pinned goal. Extend Conversation so it stores the user’s first message as a pinned goal that is always included and never compacted. Hint: add it in messages() after the summary.
2. Safe grouping. Change the trimming code so messages with role tool are never kept without the assistant message that came right before them. Test with a list that has a tool exchange at the front.
3. Tool-output limit. Write clip(text, max_tokens) that shortens long tool output using estimate_tokens and adds the note [truncated]. Use it before appending tool results to the history.

Summary

• LLM APIs are stateless; conversation state is a message list your code stores and resends every call.
• The context window limits input plus output tokens, and long context also raises cost and latency.
• Agent histories grow quickly because every tool call and result is appended.
• Trim old messages, or better, summarise them, while pinning the system prompt and goal.
• Never split a tool request from its result, and never let huge tool outputs flood the window.
• Treat history as untrusted input: guard against prompt injection and keep human approval for risky actions.