A single call to a language model is stateless: text goes in, text comes out. A production agent is a system that wraps that call with state, tools, and control logic.
What changes
An agent maintains information across steps: the goal, the plan, past observations, and results of tool calls. It decides what to do next, takes an action, and incorporates the outcome. That loop turns a text generator into software that accomplishes tasks.
The new responsibilities
Once state and actions exist, ordinary engineering concerns arrive:
- Persistence: what happens if the process restarts mid-task?
- Side effects: actions affect the outside world and cannot always be undone.
- Cost and latency: loops multiply model calls.
- Observability: you must be able to see what the agent did and why.
Why the model is not the whole system
The model proposes steps, but reliability comes from the runtime around it: validation of outputs, limits on actions, retries, and checks. A well-engineered runtime can make a modest model dependable, while a careless one makes even a strong model risky.
A helpful mental model
Treat the model as an unreliable but capable component, much like a network call or a user. You design for the cases where it is slow, wrong, or surprising.
Where to start
Define the agent's goal, the state it needs, the tools it may use, and the conditions under which it must stop or ask for help. Everything else builds on those boundaries.
Continue with the book
Explore the full treatment, worked examples, and exercises in Building Production-Grade AI Agents.
Adapted from Building Production-Grade AI Agents, published by Choral Software. Article text and original examples © 2026 Choral Software. All rights reserved.