# Learning Rate

The step size in `w ← w - lr * ∂L/∂w`. **Single most impactful hyperparameter.** If you tune one thing, tune this.

## Intuition

- Too **high** → loss explodes or oscillates wildly
- Too **low** → loss decreases but glacially; may stall in poor local minima
- "Just right" → loss curve drops smoothly and plateaus

Order-of-magnitude matters more than precision. `1e-3` vs `1e-4` is a meaningful choice; `1e-3` vs `1.2e-3` usually isn't.

## How people pick it

1. **LR range test** (Smith 2017) — train for a few hundred steps while linearly increasing LR, plot loss, pick the LR where loss decreases fastest before diverging
2. **Copy from a paper** — for Adam on transformers, `1e-4` to `5e-4` is the common range
3. **Sweep** — train short runs at `[1e-5, 1e-4, 1e-3, 1e-2]`, pick the best, narrow

## Schedules

Constant LR is rarely optimal. Common patterns:

- **Warmup** — start near 0, ramp up linearly for the first ~1–10% of steps. Prevents early instability when the optimizer's running averages haven't stabilized.
- **Cosine decay** — after warmup, decay from peak to ~0 along a cosine curve. Standard for LLM pretraining.
- **Step decay** — drop LR by 10× at fixed milestones. Older, still used in vision.
- **OneCycle** — warmup then anneal in one symmetric arc. Smith's variant.

Llama-style recipe: linear warmup for ~2000 steps, cosine decay to 10% of peak.

## Linked

[[initialization]] — under µP, optimal LR transfers across model widths, which is why µP is interesting.