Public benchmarks do not tell you whether an agent will work on your codebase. A small evaluation built from your own tasks does.
Build a task set
Collect 10 to 30 representative tasks from your project history: a bug fix, a new screen, a refactor, a test-writing job. Write down the starting commit and a way to check success.
Define success
Use objective criteria where you can: the build passes, specific tests pass, a lint check is clean. Add a human review rubric for design quality and style.
Measure more than pass rate
Record time to completion, number of agent steps, tokens or cost, and how much human editing was needed afterward. A fast agent that creates review work may not save time.
Vary one thing at a time
When comparing models, prompts, or settings, change a single variable and keep everything else constant. Run each configuration more than once, since results vary between runs.
Watch for failure categories
Group failures: misunderstood requirement, wrong architecture, broken tests, unsafe action. Patterns tell you what to fix in instructions or tooling.
Re-run regularly
Models, tools, and your codebase change. Re-running the evaluation tells you whether a new version actually helps.
Takeaway
A small, repeatable evaluation on your own work gives better evidence than any general claim.
Continue with the book
Explore the full treatment, worked examples, and exercises in Agentic iOS App Development.
Adapted from Agentic iOS App Development, published by Choral Software. Article text and original examples © 2026 Choral Software. All rights reserved.