Machinery
Reading path
Chapter 9 of 9

Where research is heading next

What researchers are trying in order to fix today's limits. The most fast-moving chapter, revised every month.

About 10 minutesUpdated November 2026new since your last visit

Most research directions aim at one of the limits from the previous chapters. Grouped by the problem they try to solve:

“It can’t learn after training.” Test-time training lets a model write what it reads into a small, changeable part of itself while it works, so knowledge isn’t stuck between frozen weights and a temporary conversation. ByteDance's In-Place TTT (ICLR 2026 oral) reuses the MLP's final projection as fast weights, so it can be added without retraining from scratchNov 2026, and NVIDIA's TTT-E2E compresses the context a model reads into its weights through next-token prediction, aiming for full attention's quality at recurrent-model latencyNov 2026. Still research, but aimed at the biggest structural gap.

“Long texts are slow and expensive.” Hybrid models replace most attention layers with layers that keep a fixed-size running summary, keeping a few full-attention layers for exact recall. These are already in some open models, and researchers at TU Munich have critically compared sub-quadratic alternatives to attention on cost, benchmarks and fundamental limitsNov 2026.

“Writing one token at a time is slow.” Diffusion language models start with a blank draft and fill in many positions at once, refining over several passes, the way image generators work. a 2026 survey reports diffusion language models reaching several-fold speed-ups at performance comparable to autoregressive modelsNov 2026, though their hallucination behaviour is still largely unexploredNov 2026.

Writing one word at a time vs refining all at once

An autoregressive model writes one token per pass. A diffusion model starts with every position blank and fills in several per pass, most confident first.

Autoregressive
Diffusion

“Thinking in words is inefficient.” Latent reasoning lets a model think in its internal numbers instead of writing out each step, or loop its own layers to think longer. More efficient, but we lose the readable chain of thought, one of the few windows we have into what a model is doing.

“Language isn’t the world.” World models learn how situations unfold, especially physical ones, by predicting in an abstract space rather than word by word. Yann LeCun's AMI Labs raised a $1.03B seed round in March 2026 to build JEPA-based world modelsNov 2026; AMI Labs has no public product yetNov 2026.

“Research itself is slow.” Agents are starting to run AI experiments themselves. Karpathy's AutoResearch (March 2026) is a short script in which an agent changes training code, runs quick experiments and keeps what worksNov 2026.

See where each direction stands this month, and how it moved, on the frontier radar.

Engineer depthWhich limitation each direction targets
DirectionLong-context costSpeedFrozen knowledgeReasoning costPhysical grounding

● main target, ○ side benefit. SSMs (Mamba) and linear attention (Gated DeltaNet) keep a fixed-size state with lossy recall; hybrids interleave a few full-attention layers. Diffusion LMs come as masked (discrete) or continuous variants and trade per-pass cost for parallelism. TTT updates fast weights with a self-supervised objective during inference. Latent reasoning includes continuous chain-of-thought (Meta’s Coconut) and recurrent-depth models; tiny recursive models do surprisingly well on ARC-AGI. JEPA-style world models predict embeddings rather than pixels or tokens. the largest and most capable open-weight models are still autoregressive decoder-only transformersNov 2026.

Checkpoint. Which limitation does test-time training mainly target?