A local agent is only as good as the model you can run on your machines. Choosing well means balancing memory, speed, and quality.
Memory first
A model's weights need memory proportional to its parameter count and numeric precision. Quantization reduces memory by storing weights with fewer bits, usually with a modest quality loss. Context length adds more through the key-value cache, so long sessions need extra room.
Speed matters for agents
Agents make many requests, often with long prompts that include file contents. Measure prompt-processing speed and generation speed separately, using realistic inputs from your repository.
Runtimes
Inference runtimes differ in hardware support, features, and ease of operation. Evaluate how well each handles your hardware, tool calling, concurrency, and model formats.
Choose a model by testing
Benchmarks are only a starting point. Run candidate models on your own tasks, such as generating an Angular component, fixing a failing test, or refactoring a service, and compare results.
Pin everything
Record the exact model file, quantization, runtime version, and settings so that results are reproducible and the whole team works with the same stack.
Serving a team
For shared use, a central server can be more efficient than many laptops. Plan for concurrency, authentication, and monitoring.
Takeaway
Pick the largest model your hardware can run at a usable speed, verify it on your tasks, and lock the configuration.
Continue with the book
Explore the full treatment, worked examples, and exercises in Local AI Agents for Angular Teams.
Adapted from Local AI Agents for Angular Teams, published by Choral Software. Article text and original examples © 2026 Choral Software. All rights reserved.