Speculative Decoding: Faster Generation Without Changing the Target
← Back to Blog & Articles Speculative decoding uses a faster draft model to propose several tokens, then asks the […]
LLMs Derived, Uncategorized
LLMs Derived
Original companion articles exploring the mathematics and engineering of large language models.
LLMs Derived, Uncategorized
DPO: Preference Optimization Without a Reward Model Loop
← Back to Blog & Articles Direct Preference Optimization turns paired preferences into a classification-like objective. It increases the relative […]
LLMs Derived, Uncategorized
LoRA: Fine-Tuning Through a Low-Rank Update
← Back to Blog & Articles Low-rank adaptation, or LoRA, keeps a pretrained weight matrix fixed and learns a compact […]
LLMs Derived, Uncategorized
How to Count Transformer Parameters Without Guessing
← Back to Blog & Articles Parameter counting reveals where model memory and compute are concentrated. The reliable method is […]
LLMs Derived, Uncategorized
Tokenization Changes What a Language Model Sees
← Back to Blog & Articles A language model does not receive words directly. It receives token identifiers produced by […]
LLMs Derived, Uncategorized
Why Scaled Dot-Product Attention Divides by Square Root of d
← Back to Blog & Articles The scaling factor in attention is not decorative. It keeps dot products in a […]