Machinery
Reading path
Chapter 3 of 9

Inside the stack

Look around, then think: one step repeated dozens of times. Plus the trick that lets giant models run cheaply.

About 8 minutesUpdated November 2026new since your last visit

A model is not one clever step. It is the same two-part step stacked dozens of times, 30 to over 120 layers in today’s large models:

  • Look around (attention, the previous chapter): each word gathers context from earlier words.
  • Think (the “MLP” part): each word, on its own, processes what it now knows. Much of the model’s stored knowledge seems to live here.

Each layer refines the understanding a little further. Roughly speaking, early layers sort out words and grammar, middle layers work with ideas and relationships (“this is a bug report”, “the user is frustrated”), and late layers turn all of that into a concrete guess about the next piece of text. At the very end, the model produces a score for every possible next token.

Token embeddingID → list of numbers (a lookup table)
repeated 30–120 times
Attentionwords share information with earlier words
+ added onto the running vector
MLP (or MoE experts)each word processed alone; holds much of the knowledge
+ added onto the running vector
Unembeddingvector → one score per possible next token
Softmax + samplingscores → probabilities → one chosen token

What “70 billion parameters” means

The parameters are the model’s dials: the numbers inside all those layers that training adjusted, one tiny nudge at a time. “A 70B model” has 70 billion of them. They encode everything the model learned, and they don’t change after training. That’s why models have a knowledge cutoff.

The trick that keeps giant models affordable

Bigger models know more, but running every dial for every word is expensive. So most frontier models now use a Mixture of Experts: the “think” part is split into many specialists, and a small router picks just a couple of them for each word. DeepSeek-V3, for example, has 671B parameters in total but uses only about 37B per tokenNov 2026.

Routing token

The router scores all 8 experts and only the top 2 run: about a quarter of the MLP weights per token. Experts don't map to neat topics like "code" or "geography"; their specialisation is statistical and mostly uninterpretable.

Engineer depthResidual stream, superposition and reading a model card

Each token’s vector travels down the residual stream: every layer reads it and adds its contribution (x = x + layer(x)), like processes writing to a shared bus. The last position’s final vector is multiplied by the unembedding matrix to give the logits, one score per vocabulary token. Most parameters sit in the MLPs (roughly two-thirds) and the attention projections.

Superposition. A model with an 8,192-wide residual stream represents far more than 8,192 concepts, by packing them into nearly orthogonal directions and accepting some interference. This superposition is why single neurons are usually polysemantic, and why researchers use sparse autoencoders to separate the vectors into millions of interpretable features.

Reading a model card. “70B parameters, 80 layers, hidden size 8192, 64 attention heads, 8 KV heads, 128k context, RoPE, SwiGLU, RMSNorm” decodes as: 80 blocks; a residual stream 8,192 floats wide; 64 query heads sharing 8 key/value heads (GQA, to shrink the KV cache); a 128k-token window; rotary positions; and standard choices for the MLP non-linearity and normalisation.

MoE in practice. The router is a small learned layer that scores experts per token; usually the top 2 to 8 run. All experts must sit in memory, so MoE saves compute, not RAM. Load-balancing losses keep the router from sending everything to a few favourites.

Checkpoint. A model has 671B parameters but only about 37B are “active”. What does that mean?