Choral Software · English · Pre-publication draft
How can you tell whether an LLM evaluation measured an improvement or just picked up noise? This book explains statistical methods for designing experiments, comparing models, and monitoring deployed systems.
What the review edition covers
- Confidence intervals, paired comparisons, and sample size planning
- Leaderboard uncertainty, multiple comparisons, and model ranking
- LLM judge bias, human audits, and prediction-powered inference
- Data contamination, training variability, and scaling laws
- Calibration, conformal prediction, and drift monitoring
- A capstone protocol for reporting and reproducing an evaluation
The 20 chapters include derivations and worked exercises. Three case studies use public MT-Bench judgments to compare human preferences across turns, measure judge–human disagreement, and fit model rankings. All three analyze the same benchmark release.
Companion Python code
The archive includes the llmstats library, tests, six synthetic examples, and the MT-Bench reproduction script with pinned input checksums and a reference transcript. The code is available under the MIT License.
Download the companion source code (ZIP)