Machinery
Reading path
Chapter 4 of 9

Generating text, one token at a time

A weighted dice roll, repeated. Why answers differ each time, what “temperature” does, and why thinking takes time.

About 9 minutesUpdated November 2026new since your last visit

At the end of the stack, the model has a score for every possible next token. Those scores become probabilities: “connection” 30%, “request” 22%, “traffic” 18%, and so on down to “banana” at almost nothing. Then one token is picked at random, weighted by those probabilities: a weighted dice roll. The chosen token is added to the text, and the whole process runs again for the next one.

That’s it. Every answer you have ever received from a chatbot was written this way, one piece at a time. It explains several things you’ve probably noticed:

  • The same question gives different answers, because the dice roll differs each time.
  • Answers stream in word by word, because that is literally how they are produced.
  • Long answers take longer, because every token needs another full pass through the model.

Two settings shape the dice roll. Temperature makes the model more predictable (low) or more adventurous (high). Top-p throws away the unlikely long tail before rolling. Move the sliders, then sample a few hundred times:

The firewall blocked the ___

1.00
1.00

The right-hand column counts your samples. Greyed words were cut by top-p.

Why “thinking” models are slower

Newer reasoning models first write out a long hidden draft (working through the problem, checking themselves) and only then write the answer you see. Those thinking tokens are generated the same way, one at a time, which is why they take longer and cost more. In return, answers to hard problems get much better. This is called test-time compute: paying for quality when the question is asked rather than when the model was trained.

Engineer depthSoftmax, KV cache, pricing and decoding tricks

Softmax exponentiates the logits divided by temperature and normalises them. Below 1 the gaps between logits grow (sharper); above 1 they shrink (flatter). Top-p keeps the smallest set of tokens whose probabilities reach p, renormalises and samples.

KV cache. The second token doesn’t recompute the first. Every earlier token’s keys and values stay in a KV cache, so each new token runs one pass of attention against it. The cache grows linearly with context and is often the real memory bottleneck.

Prefill vs decode, and why output costs more. Reading your prompt (prefill) processes all tokens in parallel and keeps the GPU compute-bound. Writing (decode) produces one token per pass and streams the weights from memory each time, so it is bandwidth-bound. That’s why output tokens cost several times more than input tokens, and why “time to first token” and “tokens per second” are separate numbers.

Prompt caching. A coding agent resends the whole transcript every step. Providers keep the KV cache for a recently seen prefix (prompt caching), so the system prompt, tools and earlier turns aren’t recomputed, and cached input is billed at a fraction of the price. Change something early in the context and everything after it must be recomputed.

Speed and validity tricks. Speculative decoding: a small draft model guesses several tokens and the big model verifies them in one parallel pass; the output distribution is unchanged. Constrained decoding: tokens that would break a JSON schema or grammar are masked out before sampling, which is how tool calls are guaranteed to parse.

Non-determinism at temperature 0. Floating-point addition isn’t associative, and batched GPU kernels sum in varying orders depending on load.

Checkpoint. You turn the temperature down to 0.1. What happens?