# Transformers

The architecture behind every modern LLM: attention, multi-head attention, the transformer block, positional encodings, layer norm, residual connections.

## What's here

- *(notes added as I work through Karpathy's GPT-from-scratch and the original paper)*

## Status

[~] In progress — going through alongside [`courses/karpathy/gpt-from-scratch/`](../../courses/karpathy/gpt-from-scratch/).

## Prerequisites

[`foundations/backprop/`](../backprop/) — you should be comfortable with how gradients flow before you stack 12 attention layers.

## Resources

- *Attention Is All You Need* — original paper. Notes: [`papers/notes/attention-is-all-you-need.md`](../../papers/notes/attention-is-all-you-need.md).
- Jay Alammar — *The Illustrated Transformer*.
- Karpathy — *Let's build GPT* lecture.