A model's context window is finite, and every token costs money and latency. Production agents manage context deliberately instead of appending everything.
Where the budget goes
A typical request includes system instructions, tool definitions, conversation history, retrieved documents, tool outputs, and space for the model's reply. Long tool outputs or logs can easily crowd out the information that matters.
Allocate by purpose
Set a budget for each category. For example, reserve a fixed share for instructions and tool schemas, another for retrieved evidence, and enough room for the answer. Decide in advance what is trimmed first when space runs short.
Priorities
- Keep the current task and constraints.
- Keep recent, relevant observations.
- Summarize or drop old turns.
- Truncate bulky outputs, keeping the parts that decide the next step.
Measure it
Count tokens before sending requests, log the breakdown by category, and watch for requests near the limit. Compare quality when you shorten context, since cutting too aggressively can hurt performance.
More context is not always better
Irrelevant material can distract the model and raise cost. A smaller, well-curated context often beats a large unfiltered one.
The principle
Context is a scarce runtime resource, like memory. Plan it, monitor it, and make trimming decisions explicitly.
Continue with the book
Explore the full treatment, worked examples, and exercises in Building Production-Grade AI Agents.
Adapted from Building Production-Grade AI Agents, published by Choral Software. Article text and original examples © 2026 Choral Software. All rights reserved.