Machinery
Reading path
Chapter 8 of 9

Around the model: small models, decision models, harnesses

Real AI products are teams, not one big brain. Who does what, and where the new decision models fit.

About 11 minutesUpdated November 2026new since your last visit

When you use an AI product, you’re rarely talking to one model. Behind the scenes there is usually a small team, each member doing what it’s best and cheapest at:

  • Large frontier models do the open-ended thinking: understanding the task, planning, writing code.
  • Small language models are the same kind of model made much smaller. They’re fast, cheap, private, and can run on a laptop or phone, but they know less and are weaker at long reasoning. They’re good for summarising, sorting and extracting.
  • Decision models are new in autumn 2026. They don’t write text at all. You ask a narrow question with fixed answers (“allow, ask a human, or block?”) and get back a probability for each option, in about a tenth of a second. TypeSafe AI released Jev, the model that started the decision-model category, in mid-September 2026Nov 2026, and AWS followed with an open version.
  • The harness is ordinary software: the loop that passes messages, runs tools, enforces permissions and keeps the conversation. Agent frameworks such as the Strands Agents SDK live here. (The Strands project also released a decision model, Strands Decider, which is why the names blur.)

Click through the steps of a typical agent loop to see who handles what:

Frontier LLMDecision modelSmall modelHarness / your code

Try it: a safety gate for an AI’s actions

An AI agent proposes a command. A decision model scores how risky it is, and plain, auditable code applies the rule. The model supplies a judgement; the policy stays in code you can read.

0.90
 

Illustrative probabilities, not output from a real decision model.

Engineer depthDecision models as a category

A decision model keeps the entire pretrained transformer and swaps only the unembedding layer from the stack chapter. Instead of scoring ~200k vocabulary tokens, a small pointer head scores your options in a single forward pass. The torso keeps everything it learned about language; the model just loses the ability to ramble. Question types are typically predicate (yes/no), choice (one of N) and score.

Same torso, different head
Generative LLM
Transformer stack
↓
LM headscores all ~200k tokens
↓
sample a token, repeat hundreds of times
"I think billing should take this, because…"
Decision model
Same pretrained stack
↓
Pointer headscores only the options you passed
↓
one forward pass, done
state:  "Help! My payouts have been failing for 3 days!"
choice: Which team should handle this? = billing, sales, retail

Question types: predicate (yes/no), choice (one of N), score (a value on a scale). Example adapted from the Strands Decider README; probabilities are illustrative.

ModelFromAccessNotes
JevTypeSafe AI, Sept 2026Closed, APIStarted and named the category; TypeSafe calls it a "System One" model.
Strands Decider 2BAWS Strands Labs, Oct 2026Open weights, Apache 2.0AWS Strands Labs released Strands Decider 2B on 1 October 2026 under Apache 2.0: a Qwen3.5-2B torso whose text head is replaced by a pointer head, with Gemma-based variants from E2B to 26B-A4BNov 2026.
LayaNandakishor M., 2025OpenAn earlier non-autoregressive decision model resolving typed decisions in one pass.

Benchmarks in a two-month-old category: Strands' own table lists Decider 2B at 0.78 on JevBench, while an independent run on the 231 public tasks got 72.3%Nov 2026. Numbers depend on who ran them; measure on your own decisions.

Harnesses and MCP. a Strands agent boils down to a model, a system prompt and a list of tools, and the model itself decides when to call which toolNov 2026. Harnesses run tools, manage context and retries, and increasingly connect to tool servers over MCP. The common deployment pattern is a cascade: a cheap model or decision model handles everything, acts on confident answers, and escalates the uncertain tail.

SLMs in practice. Usually 0.5–15B parameters, made capable through distillation and runnable locally through quantisation. Typical roles: summarising a 50k-token log before it reaches the expensive model, extraction, classification, on-device features with data that must not leave the machine.

Checkpoint. What does a decision model such as Jev or Strands Decider return, compared with a normal LLM?