LLMs Derived
By Choral Software
Language: English
What does a large language model actually compute? Not what it might do one day, but what it does, expressed in the mathematics of the original papers, derived from first principles and connected end to end.
LLMs Derived is a complete, derivation-first treatment of the mathematics and engineering behind modern large language models, for readers who want the real thing rather than a product tour. Across eighteen chapters it derives the transformer architecture, attention, tokenization, pretraining, scaling laws, distributed training systems, fine-tuning, RLHF and DPO, and inference-time decoding. Every claim is backed by a derivation, a citation, or working code.
Inside you will find:
- Scaled dot-product attention, derived from first principles and implemented from raw tensors
- The full parameter-counting formula for a transformer, checked against GPT-2 and GPT-3 configurations
- Chinchilla’s compute-optimal scaling laws, re-derived and fit to synthetic data
- Distributed training: data, tensor and pipeline parallelism, and the ZeRO memory-partitioning stages
- LoRA and QLoRA, derived and implemented as working PyTorch modules
- The Direct Preference Optimization (DPO) loss, derived step by step from the RLHF objective
- Speculative decoding, with a proof that its output distribution matches the target model
- A grounded, citation-backed discussion of hallucination, evaluation contamination and mechanistic interpretability
Every chapter includes complete, runnable Python (PyTorch and NumPy), a summary of key results, and exercises that mix hand derivations with coding tasks.
Who it is for: graduate students, software engineers moving into ML roles, and researchers in adjacent fields who already know linear algebra, calculus, probability and basic Python/PyTorch, and want the specific mathematics of large language models rather than a general introduction to machine learning.
Apple Books ID: 6821756667
Download companion source code (ZIP)
Continue exploring
View this title on Apple Books
Related books
Browse the complete book catalog Explore ChoralSoft audiobooks