Speculative decoding uses a faster draft model to propose several tokens, then asks the target model to evaluate those proposals together. A correction rule preserves the target distribution.
Draft and verify
The draft model generates a short candidate continuation. The target model scores all proposed positions in one parallel pass. Tokens that pass an acceptance test are emitted; after a rejection, a corrected token is sampled and drafting begins again.
Where speed comes from
Ordinary autoregressive decoding invokes the large model once per token. Speculation can accept several draft tokens for one target-model pass. The gain is largest when the draft is cheap, its proposals often agree with the target, and verification uses hardware efficiently.
Exactness needs the correction step
Simply accepting matching argmax tokens changes sampling behavior. The acceptance probabilities and residual distribution are what make the resulting samples follow the target model. Performance claims should report acceptance rate, end-to-end latency, batch size, and hardware.
Continue with the book
Explore the full treatment, worked examples, and exercises in LLMs Derived.
Adapted from LLMs Derived, published by Choral Software. Article text and original examples © 2026 Choral Software. All rights reserved.