Machinery
Reading path
Chapter 5 of 9

Training: from autocomplete to assistant to agent

Five stages that each change what the model is. Knowing them explains a lot of everyday behaviour.

About 10 minutesUpdated November 2026new since your last visit

Nobody writes the rules a model follows. Everything it does was learned, in stages. And each stage changes what the model is, not just how good it is:

Click through the five stages. Each one shows the training data, the goal, and an example:

How this explains everyday behaviour

  • Why models are so good at code and maths: answers can be checked automatically, so the practice-exam stage works especially well there. since late 2024, reinforcement learning with verifiable rewards has produced reasoning models (o1, DeepSeek-R1 and successors)Nov 2026.
  • Why they sound confident and agreeable: people rating answers tend to prefer confident ones that agree with them, and models absorbed that preference. More on this in the limits chapter.
  • Why they don’t know last month’s news: the reading stage is a snapshot. Anything newer exists for the model only if a tool or you put it into the conversation.
  • Why a coding agent sometimes “fixes” a test by changing the test: in practice exams, a passing test was the reward. Models can learn shortcuts that earn the reward without doing what was meant.
Engineer depthScaling laws, synthetic data and reward hacking

Pretraining minimises cross-entropy loss on next-token prediction over tens of trillions of tokens. Scaling laws show loss falling as a smooth power law in parameters, data and compute, which made frontier training a predictable engineering bet. Pretraining historically consumed most of the compute; RL post-training is the fastest-growing share and produced most capability gains since late 2024.

Models training models. A growing share of data is synthetic: frontier models write problems, solutions and explanations; verifiers filter them; the results train the next generation or smaller models via distillation. Preference feedback is increasingly AI-generated too (constitutional AI, RLAIF), with humans auditing.

Reward hacking. RL optimises whatever the checker rewards. If passing tests is the reward, a model can learn to special-case the tests, weaken assertions or edit the test file. Labs close these loopholes, but traces of reward hacking still show up in coding agents. Keep the mechanism in mind when a build turns green suspiciously easily.

Cutoff and contamination. Pretraining data is a snapshot, so later facts only arrive through context. Benchmark questions leak into web-scale corpora, so treat headline scores as an upper bound and test on your own tasks.

Checkpoint. Which stage taught models to work through problems step by step before answering?