Hyperparameters

Overview

Hyperparameter = anything you set before training that the model doesn't learn from data.

• Parameters — weights and biases. Learned by gradient descent. Live inside the model.
• Hyperparameters — knobs you choose. Live around the model.

There is no closed-form formula for picking them. Modern practice is: copy a known-good recipe, sweep the few that matter most, scale up. Frontier LLM teams do thousands of small runs to dial these in before a big run.

When you're picking hyperparameters

1. Start from a published recipe (ResNet, GPT-2, Llama, etc.)
2. Tune learning rate first — biggest payoff per unit effort
3. Sweep on a small proxy model, then scale up
4. Read about scaling laws (Kaplan 2020, Chinchilla 2022) and µP if you go big

Open questions

• What does µP actually do mechanically? Why does the optimal LR transfer across widths under µP but not standard parameterization?
• Why is the FFN ratio in transformers 4×? Has anyone systematically tested deviations?

01Initialization

How weights start before training.

Why not zero?

If every weight in a layer is 0 (or any same constant), every neuron in that layer computes the exact same thing on the forward pass and receives the exact same gradient on the backward pass. They update identically forever — the layer collapses to one effective neuron. This is the symmetry problem.

Biases are fine to start at 0 because each neuron has its own bias, so symmetry is broken by the (random) weights anyway.

Why not large random?

If W is too large, activations explode layer by layer. Too small, they vanish. Either way gradients break. The fix is scaled random init, where the scale depends on fan_in (number of inputs to the neuron).

The two you'll see most

# He / Kaiming — for ReLU and friends
W = randn(fan_in, fan_out) * sqrt(2 / fan_in)

# Xavier / Glorot — for tanh / sigmoid
W = randn(fan_in, fan_out) * sqrt(1 / fan_in)

The sqrt(2/fan_in) factor keeps the variance of activations roughly constant as signals propagate forward — derived by tracking the variance of Wx through one layer assuming inputs and weights are independent with mean 0.

What PyTorch does by default

nn.Linear uses Kaiming uniform with a = sqrt(5). Reasonable default but not always optimal — many codebases override it.

µP (Maximal Update Parameterization)

A reparameterization where the optimal learning rate stays constant as you scale model width. Tune hyperparameters on a tiny model, scale up with confidence. Used in production at Cerebras and reportedly parts of GPT-4 training.

In our forward-pass viz

The weights in forward-pass.html are hand-picked, not from any init scheme — chosen so the arithmetic is readable. A real nn.Linear(2, 3) in PyTorch would init those with Kaiming uniform.

02Width & depth

How many layers, how many neurons each. Pure hyperparameters — no formula.

Practical heuristics

• Powers of 2 (64, 128, 256, ...) — GPU-friendly memory layout
• Hidden wider than input/output — gives capacity for nonlinear features
• Underfitting (train loss too high) → wider or deeper
• Overfitting (train low, val high) → narrower, or add regularization

Real models

Folk formulas to ignore

hidden = (input + output) / 2 and sqrt(input × output) show up in tutorials. Not principled. Skip them.

Modern view

Width, depth, dataset size, and compute scale together — see scaling laws (Kaplan 2020, Chinchilla 2022). Chinchilla's claim: most LLMs of its era were too big for their training data. Optimal allocation is roughly 20 tokens per parameter.

03Learning rate

The step size in w ← w − lr · ∂L/∂w. Single most impactful hyperparameter. If you tune one thing, tune this.

Intuition

• Too high → loss explodes or oscillates wildly
• Too low → loss decreases but glacially; may stall in poor local minima
• "Just right" → loss curve drops smoothly and plateaus

Order-of-magnitude matters more than precision. 1e-3 vs 1e-4 is a meaningful choice; 1e-3 vs 1.2e-3 usually isn't.

How people pick it

1. LR range test (Smith 2017) — train for a few hundred steps while linearly increasing LR, plot loss, pick the LR where loss decreases fastest before diverging
2. Copy from a paper — for Adam on transformers, 1e-4 to 5e-4 is the common range
3. Sweep — train short runs at [1e-5, 1e-4, 1e-3, 1e-2], pick the best, narrow

Schedules

Constant LR is rarely optimal. Common patterns:

• Warmup — start near 0, ramp up linearly for the first ~1–10% of steps. Prevents early instability when the optimizer's running averages haven't stabilized.
• Cosine decay — after warmup, decay from peak to ~0 along a cosine curve. Standard for LLM pretraining.
• Step decay — drop LR by 10× at fixed milestones. Older, still used in vision.
• OneCycle — warmup then anneal in one symmetric arc. Smith's variant.

Llama-style recipe: linear warmup for ~2000 steps, cosine decay to 10% of peak.

04Regularization

Anything that fights overfitting — i.e. reduces the gap between train loss and val loss.

Dropout

During training, randomly zero out a fraction p of activations each forward pass. Forces the network to not rely on any single neuron. Disabled at inference.

• Typical: p = 0.1 to 0.5
• Less common in modern transformers (Llama uses 0)

Weight decay

Add λ · ‖W‖² to the loss, or equivalently shrink weights by a small factor every step. Discourages large weights.

• AdamW = Adam + weight decay done correctly (decoupled from the gradient update)
• Typical: 0.01 to 0.1

Label smoothing

Instead of one-hot targets [0, 1, 0], use soft targets like [0.05, 0.9, 0.05]. Prevents the model from becoming overconfident. Common in vision and NMT.

Data augmentation

Not a knob on the model — a knob on the data. Crops, flips, mixup for images; token dropout, span masking for text. Often the most effective regularizer in practice.

Early stopping

Stop training when val loss stops improving. The simplest regularizer.

When to use what

• Small dataset relative to model → all of the above matter
• Very large dataset (LLM pretraining) → most of these become irrelevant; you under-fit before you over-fit

05Cheat sheet

Categorized list of common hyperparameters and which ones actually matter.

Architecture (set once, expensive to change)

Optimization (most-tuned in practice)

Training loop

What to tune first

1. Learning rate
2. Batch size (if memory allows)
3. Width
4. Weight decay
5. Everything else