Machinery
Reading path
Chapter 1 of 9

Text becomes tokens, tokens become numbers

Models never see letters or words. They see numbered pieces of text, and then points on a map of meaning.

About 8 minutesUpdated November 2026new since your last visit

Before a model can do anything with your text, the text gets chopped into pieces called tokens. A token is usually a word or a chunk of a word. Common words like “the” or “firewall” are one token each. Rare or long words get split into several pieces. The model works with a fixed vocabulary of roughly 100k–250k entriesNov 2026, and every piece has a number.

Try it. Type a sentence and watch it split. Then try “Strawberry”, a long German word, or some code.

Toy tokenizer with a tiny vocabulary, for illustration; the small numbers are made-up IDs. Try "Strawberry" or a German word.

Why this matters to you

  • Limits and prices are counted in tokens, not words. In English, 1,000 tokens is roughly 750 words. Other languages, German included, usually need more tokens for the same content.
  • Letter-level tasks can be surprisingly hard. “How many r’s are in strawberry?” used to trip models up, because the model sees two opaque pieces, Straw and berry, not ten letters.
  • Strange input still works. There’s never an “unknown word”: anything can be broken down into small enough pieces. It’s just more expensive.

From pieces to meaning

A number like 892 means nothing by itself. So each token number is translated into a long list of numbers, thousands of them, that act like coordinates on a map of meaning. Training arranges this map so that related words end up near each other, and relationships turn into consistent directions. The step from “king” to “queen” points the same way as from “man” to “woman”.

A map of meanings, squashed to 2D
same offsetcapital-of kingqueenmanwoman BerlinGermanyParisFrance malwareransomwaretrojanwormphishing malware family

Hand-drawn cartoon. Real embeddings have thousands of dimensions, and the directions are only roughly this clean.

Engineer depthTokenizers and embedding tables

A tokenizer is fixed preprocessing that maps text to integer IDs and back. Most use byte-level BPE: the vocabulary starts with all 256 byte values, then adds merged pieces learned from how often byte sequences co-occur in the training corpus. Any input can always be encoded; worst case it falls back to single bytes.

Each ID then selects an embedding: a row in the model’s first weight matrix, typically 4,000–16,000 floats wide. That width (often called d_model) is also the width of the residual stream: every later layer reads and writes vectors of exactly that size.

Code, German and numbers cost differently. Tokenizers trained on lots of code give common indentation runs and keywords their own tokens. German compounds like “Datenschutzgrundverordnung” split into many pieces. Numbers used to be chunked inconsistently (“12345” might become “123” + “45”), one reason arithmetic was shaky; many newer tokenizers split digits individually.

CS framing: the tokenizer is a deterministic compression codec; the embedding table is an array of vectors indexed by token ID.

Checkpoint. Why did LLMs famously struggle to count the r's in “strawberry”?