Attention: how words borrow meaning
The one mechanism that made modern AI work. Each word looks back at earlier words and pulls in what it needs.
Words mean different things in different company. “Bank” by a river is not “bank” on a statement. In “The server rejected the request because it was malformed”, you instantly know “it” means the request, not the server. A model has to work that out too, and the mechanism it uses is called attention.
After the previous chapter, each word is a point on the map of meaning, but each one only knows about itself. Attention lets every word look back at the words before it, decide which ones matter for its own meaning, and blend in information from them.
Click any word below to see where it looks. Click “it” first.
Click any word. Illustrative weights for one attention head; real heads are messier, and meaning is spread across many heads and layers.
Notice the dashed words: a word can only look backwards. When the model writes text, the future doesn’t exist yet, so it is never allowed to peek ahead.
Why this was the breakthrough
Older language models read text strictly word by word and had to squeeze everything they’d read into a small running summary, so they forgot the beginning of long sentences. Attention lets every word reach back to any earlier word directly. The 2017 paper that introduced the transformer built a whole model around this idea, and nearly every modern LLM descends from it.
There is a price, which comes back later: every word compares itself with every earlier word. Double the length of the text and the work roughly quadruples.
Engineer depthQuery, key, value: a soft hash-table lookup
Mechanically, attention is a differentiable key-value lookup, built from three matrix multiplications per token (query, key, value):
- Each token emits a query (“what am I looking for?”), a key (“what do I contain?”) and a value (“what do I hand over?”).
- Each earlier token is scored with query · key. Softmax turns the scores into weights that add up to 1.
- The output is the weighted sum of the values, added back onto the token’s own vector.
The causal mask sets the scores for future positions to minus infinity. Real models run dozens of these lookups in parallel per layer (attention heads); GQA lets many query heads share fewer key/value heads to save memory.
Position is injected. The dot product alone treats the input as a set. RoPE rotates each query and key by an angle proportional to its position, so the score depends on relative distance. Extending a context window mostly means stretching these rotations.
Induction heads. Researchers found a two-head circuit that implements “find where the current token appeared before, and predict what came after it last time”. These induction heads emerge abruptly in training and drive much of in-context learning. For coding, this is why models mirror patterns already in context so well: an existing test, a naming convention, a neighbouring handler.
Cost. Reading a prompt of n tokens costs about n²/2 scores per head per layer, all in parallel. Generating each new token costs work proportional to n, against cached keys. A huge prompt is affordable; a long conversation slowly gets slower.
Checkpoint. In “The server rejected the request because it was malformed”, what lets “it” pick up meaning from “request”?