LLMs Derived Archives - ChoralSoft

Speculative Decoding: Faster Generation Without Changing the Target

alavingi — Wed, 21 Oct 2026 15:43:56 +0000

Speculative decoding uses a faster draft model to propose several tokens, then asks the target model to evaluate those proposals together. A correction rule preserves the target distribution.

Draft and verify

The draft model generates a short candidate continuation. The target model scores all proposed positions in one parallel pass. Tokens that pass an acceptance test are emitted; after a rejection, a corrected token is sampled and drafting begins again.

Where speed comes from

Ordinary autoregressive decoding invokes the large model once per token. Speculation can accept several draft tokens for one target-model pass. The gain is largest when the draft is cheap, its proposals often agree with the target, and verification uses hardware efficiently.

Exactness needs the correction step

Simply accepting matching argmax tokens changes sampling behavior. The acceptance probabilities and residual distribution are what make the resulting samples follow the target model. Performance claims should report acceptance rate, end-to-end latency, batch size, and hardware.

Continue with the book

Explore the full treatment, worked examples, and exercises in LLMs Derived.

The post Speculative Decoding: Faster Generation Without Changing the Target appeared first on ChoralSoft.

DPO: Preference Optimization Without a Reward Model Loop

alavingi — Wed, 21 Oct 2026 15:43:43 +0000

← Back to Blog & Articles

Direct Preference Optimization turns paired preferences into a classification-like objective. It increases the relative probability of a preferred response while controlling drift from a reference model.

The training signal

Each example contains a prompt, a chosen response, and a rejected response. The objective compares how strongly the current policy favors the chosen response relative to how strongly the reference policy favors it.

Why the reference matters

Without a reference, optimization could improve the pairwise objective by moving too far from the pretrained behavior. The reference supplies an implicit regularizing anchor, while a temperature-like parameter controls the strength of the preference separation.

Limits remain

DPO inherits the quality and coverage of the preference data. Inconsistent labels, narrow prompts, length bias, and distribution shift can produce misleading gains. Evaluation should include held-out preferences, task performance, safety checks, and measures of unwanted behavioral drift.

Continue with the book

Explore the full treatment, worked examples, and exercises in LLMs Derived.

The post DPO: Preference Optimization Without a Reward Model Loop appeared first on ChoralSoft.

LoRA: Fine-Tuning Through a Low-Rank Update

alavingi — Wed, 21 Oct 2026 15:43:32 +0000

← Back to Blog & Articles

Low-rank adaptation, or LoRA, keeps a pretrained weight matrix fixed and learns a compact update expressed as the product of two smaller matrices.

The parameter reduction

For a weight with shape m by n, full fine-tuning changes mn values. A rank-r update uses factors with roughly r times m plus n values. When r is much smaller than either dimension, the trainable parameter count drops sharply.

Why it can work

The method assumes that the task-specific change can be represented in a low-dimensional subspace even though the base model is large. The update is scaled and added during the forward pass; it can later be merged into the base weight for deployment.

Practical choices

Results depend on which modules receive adapters, the rank, scaling, dropout, data quality, and optimization settings. LoRA reduces trainable parameters and optimizer memory, but it does not eliminate the cost of running the frozen model itself.

Continue with the book

Explore the full treatment, worked examples, and exercises in LLMs Derived.

The post LoRA: Fine-Tuning Through a Low-Rank Update appeared first on ChoralSoft.

How to Count Transformer Parameters Without Guessing

alavingi — Wed, 21 Oct 2026 15:43:20 +0000

← Back to Blog & Articles

Parameter counting reveals where model memory and compute are concentrated. The reliable method is to count each matrix from its input and output dimensions, then add biases and normalization parameters if present.

Embeddings and attention

A token embedding contains vocabulary size times model width parameters. A standard attention block has query, key, value, and output projections. When all are square model-width matrices, their leading contribution is four times width squared per layer.

The feed-forward block

A two-layer multilayer perceptron expands from model width to an intermediate width and projects back. Its leading count is twice the product of those widths. Gated variants add another input projection and therefore change the familiar shortcut formula.

Avoid hidden assumptions

Check whether input and output embeddings are tied, whether projections include biases, how many key-value heads are used, and which normalization is implemented. State the architecture first; then a parameter total becomes an auditable calculation rather than a memorized estimate.

Continue with the book

Explore the full treatment, worked examples, and exercises in LLMs Derived.

The post How to Count Transformer Parameters Without Guessing appeared first on ChoralSoft.

Tokenization Changes What a Language Model Sees

alavingi — Wed, 21 Oct 2026 15:43:10 +0000

← Back to Blog & Articles

A language model does not receive words directly. It receives token identifiers produced by a tokenizer, and that representation shapes sequence length, cost, and the patterns available to learn.

Vocabulary versus sequence length

A larger vocabulary can represent common strings with fewer tokens but requires a larger embedding and output layer. A smaller vocabulary reuses pieces more widely but produces longer sequences. The best trade-off depends on languages, domains, and compute limits.

Uneven treatment of text

Names, source code, rare scripts, and morphologically rich languages may split into many pieces. Two sentences with similar character length can therefore consume very different context. Token-level metrics should be complemented with character-, byte-, or task-level analysis.

Reproducibility requires the tokenizer

Model weights alone are not enough. Normalization rules, special-token assignments, vocabulary files, and pre-tokenization behavior must match training. Changing them changes the input distribution even when the visible text looks identical.

Continue with the book

Explore the full treatment, worked examples, and exercises in LLMs Derived.

The post Tokenization Changes What a Language Model Sees appeared first on ChoralSoft.

Why Scaled Dot-Product Attention Divides by Square Root of d

alavingi — Wed, 21 Oct 2026 15:43:01 +0000

← Back to Blog & Articles

The scaling factor in attention is not decorative. It keeps dot products in a range where the softmax remains responsive as vector dimension grows.

What grows with dimension

If query and key components are roughly independent with mean zero and similar variance, their dot product sums d component-wise products. Its variance grows in proportion to d, so its typical magnitude grows like the square root of d.

What softmax does with large scores

Large-magnitude logits make softmax outputs extremely concentrated. In that saturated regime, most probabilities are near zero and gradients can become small or unstable. Dividing by the square root of d approximately normalizes the score scale.

It is a modeling assumption

Real learned vectors are not perfectly independent, but the derivation explains the design pressure. Implementations also subtract the maximum logit before exponentiation for numerical stability and apply masks before normalization.

Continue with the book

Explore the full treatment, worked examples, and exercises in LLMs Derived.

The post Why Scaled Dot-Product Attention Divides by Square Root of d appeared first on ChoralSoft.