Retrieval-augmented generation connects a language model to an external collection. Its quality depends on a sequence of design choices, any one of which can become the bottleneck.
Indexing and retrieval
Documents are cleaned, divided into useful chunks, represented for search, and stored with metadata. At query time, lexical, vector, or hybrid retrieval produces candidates. Chunk size, overlap, filters, and embedding choice all affect recall.
Ranking and generation
A reranker can reorder candidates before the prompt is assembled. The generator should distinguish supplied evidence from its own prior patterns and cite the supporting passages when the application requires traceability.
Evaluate each stage
End-to-end answer quality alone does not show where failure occurred. Measure retrieval recall, ranking quality, groundedness, citation accuracy, latency, and cost separately. Also test adversarial documents, stale sources, access controls, and queries with no supported answer.
Continue with the book
Explore the full treatment, worked examples, and exercises in The AI Algorithms Compendium.
Adapted from The AI Algorithms Compendium, published by Choral Software. Article text and original examples © 2026 Choral Software. All rights reserved.