Is it really just predicting the next word?
Yes, and that sentence hides almost everything interesting. Why a coding agent feels like so much more than autocomplete.
Every response you have ever had from an AI, including a coding agent’s multi-file change, was produced one token at a time. Nothing else comes out of the model. And yet it plainly doesn’t feel like your phone’s autocomplete. Three separate things close that gap, and it helps to keep them apart.
1. To predict well, the model has to understand what it’s predicting
“Predict the next word” describes how the model was trained, not what happens inside it. To predict the next line of a proof, a stack trace or a detective story across trillions of examples, the cheapest strategy turned out to be to build an internal picture of what’s going on: the state of the program, the structure of the argument, who the murderer probably is.
There’s direct evidence for this. Othello-GPT, trained only on move sequences, built an internal representation of the boardsource, and editing that internal board changed which moves it predicted. In production models, Anthropic's 2025 attribution-graph work traced two-hop reasoning and rhyme planning inside Claude 3.5 Haikusource: the model works out an intermediate step it never writes down. Try the experiment:
Prompt: "Fact: the capital of the state containing Dallas is"
Simplified from Anthropic's 2025 attribution-graph study of Claude 3.5 Haiku. The "Texas" step is never written out, yet replacing it changes the answer to Sacramento: the model computes the hop rather than recalling one memorised string.
2. Training changed what the predictions are for
A model straight out of the reading stage predicts what text would typically come next on the internet. Ask it to fix a failing test, and it may well continue the GitHub issue thread instead. Later training changes the goal: in practice environments, the tokens that eventually lead to passing tests get rewarded, even when the payoff comes fifty steps later. The model ends up pursuing outcomes. Same kind of model, same prompt, three stages of training:
> TestParseHeader in parser_test.go is failing. Fix it.
3. The tool loop turns one prediction into a process
What you use in a coding tool isn’t just a model. It’s a loop around one. The model writes a request to use a tool (“read this file”, “run the tests”) and stops. Ordinary software, the harness, carries out the request and adds the result to the conversation. Then the same, unchanged model is asked again with everything so far. There’s no agent “thinking” in the background between steps; all continuity lives in the text. Step through a typical session:
A typical, simplified agent session; token counts are realistic orders of magnitude.
Put the three together: a model that builds rich internal pictures in order to predict well, trained to pursue outcomes, running in a loop with tools, tests and room to think. “Next-token prediction” is accurate in the same way that “a computer just executes instructions” is accurate.
When it feels like autocomplete again
The illusion breaks in predictable ways, and each one traces back to a mechanism from this guide. Knowing the mechanism tells you what to do about it.
| Symptom | What's happening underneath | What helps |
|---|---|---|
| Invents a plausible function, setting or software package | It writes the most likely continuation; nothing is looked up unless a tool is used. Attackers register commonly invented package names (slopsquatting). | Let it read the docs or the code. Check that new dependencies exist and are the ones you expect. |
| Writes the common solution, not the one your project needs | Patterns from millions of projects outweigh a weak signal from a few files it has seen. | Point it at the relevant files, show an existing example to copy, state constraints explicitly. |
| Gets worse late in a long session | Long conversations dilute attention; details in the middle are used worst; summarising old turns loses details. | One task per session. Keep decisions in a file that gets loaded every time. |
| Repeats the same failed fix | Its own earlier attempts in the conversation make the same continuation more likely. | Interrupt, add new information, or start fresh with a short summary of what failed. |
| "You're absolutely right!", then breaks working code | Agreeableness absorbed from human ratings during training (sycophancy). | Ask for evidence or a failing test, not agreement. Phrase pushback as a question. |
| Makes tests pass by changing the tests | A shortcut learned in training, where a passing test was the reward (reward hacking). | Say tests are off-limits, and review changes to test files separately. |
| Follows instructions hidden in a web page, email or README | To the model, data and instructions are the same stream of text (prompt injection). | Limit what tools can do, sandbox them, and don't auto-approve actions that combine internet access with secrets. |
Engineer depthPolicies, scratchpads and the evidence from interpretability
From estimator to policy. Formally the output is always a distribution over the next token. After RL it is no longer an estimate of “what would a typical document say here” but a policy: a strategy optimised for outcomes whose actions happen to be tokens. That is the biggest single reason modern coding models feel goal-directed. They were optimised to be.
Why reasoning tokens matter. One forward pass has fixed depth (maybe a hundred layers), which caps the serial computation that can go into a single token. Writing an intermediate step makes it input for every later pass: the visible reasoning is a scratchpad that buys as many sequential steps as needed. That’s why “think step by step” worked, and why reasoning models are trained to do it at length.
Interpretability evidence. Attribution graphs on Claude 3.5 Haiku showed an intermediate “Texas” feature in the Dallas → Austin prompt (patching it to “California” yields Sacramento) and, in rhyming couplets, the rhyme word being selected at the end of line one before line two is written; swapping that planned word rewrote the line to fit. Othello-GPT’s board representation turned out to be linearly decodable, and interventions on it changed predicted moves causally.
The agent loop at the token level. Each step is a fresh call on the full transcript. Tool calls are text in a fixed format ending at a stop token; constrained decoding keeps arguments valid; prompt caching keeps the resend cheap; context rot and compaction are why long sessions degrade.
Checkpoint. A coding agent edits three files, runs the tests and iterates for ten minutes. What is the model doing at each step?