An agent can reach the right answer through a dangerous path, or fail after doing most things right. Evaluating only final outcomes misses both.
What a trajectory is
A trajectory is the full record of an agent run: the model's reasoning steps, tool calls, arguments, results, and final response.
Why outcomes alone mislead
- An agent might get the correct result by luck, or by taking an action it should not have.
- A near miss may reveal a small fixable flaw that the outcome score hides.
- Cost, latency, and number of steps are part of quality too.
What to check along the path
- Tool selection: did it choose appropriate tools?
- Arguments: were the inputs valid and safe?
- Efficiency: did it loop or repeat work?
- Policy compliance: did it stay within permissions and ask for approval when required?
- Recovery: how did it respond to errors?
Methods
Use deterministic checks where possible, such as asserting that a forbidden tool was never called. Use rubric-based review, by humans or carefully validated model judges, for qualities that are harder to codify. Keep traces so failures can be replayed and inspected.
Build a test set
Collect real tasks, including difficult and adversarial ones, and rerun them whenever you change the model, prompts, or tools.
Takeaway
Measure how the agent gets there as well as where it ends up.
Continue with the book
Explore the full treatment, worked examples, and exercises in Building Production-Grade AI Agents.
Adapted from Building Production-Grade AI Agents, published by Choral Software. Article text and original examples © 2026 Choral Software. All rights reserved.