Two things the market map doesn't cover: the pipeline that turns raw next-token prediction into a reasoning model, and the security model of agents that share instructions and data in one token stream. Written for someone who ships with these, not just talks about them.
Two phases: pretraining (predict the next token on trillions of tokens) and post-training (turn that base model into something useful, aligned, and increasingly a reasoner). Where capability comes from is mostly a post-training story now.
A base model is a raw next-token predictor. Useful as a substrate, useless as an assistant (it completes text, it doesn't follow instructions). How you spend compute here is governed by scaling laws.
Chinchilla's arithmetic bites: a ~1T-param model wants ~20T tokens, but high-quality natural web text is estimated at only ~10–50T tokens total. That's the data wall — beyond trillions of tokens, high-information-density text gets scarce and returns collapse. Hence synthetic data. The nuance that matters:
The pipeline that makes a model helpful and aligned. Each stage solves a specific problem the previous one leaves open.
The classic recipe (InstructGPT): SFT on labeler demonstrations → train a reward model on ranked outputs → fine-tune with RL (PPO) against that reward. The headline result that proved alignment beats raw scale for usefulness: outputs from the 1.3B InstructGPT were preferred over the 175B GPT-3, a 100× smaller model, because it was aligned to what people actually wanted.
| Method | What it changes | Why it's used |
|---|---|---|
| RLHF (RM + PPO) | The original: explicit reward model + RL loop | Powerful but unstable, complex, compute-heavy |
| DPO (Direct Preference Optimization) | Closed-form link between reward and optimal policy → no reward model, no RL sampling loop. Just a classification-style loss. | Stable, lightweight, matches or exceeds PPO. The default for many open models. |
| Constitutional AI / RLAIF | Model critiques & revises its own answers against a written "constitution," then RL uses AI-generated preferences instead of human labels | Scales harmlessness without armies of human labelers (Anthropic's approach) |
The biggest capability shift since instruction tuning, and the one your market map (o-series, R-series, "thinking" models) rests on. The core idea is almost embarrassingly simple.
RLVR = Reinforcement Learning from Verifiable Rewards. Take problems with checkable answers (math, code, STEM). Let the model generate a long chain of thought, then reward it purely on whether the final answer is right — no human preference, no reward model, just a verifier. Do this at scale and reasoning emerges.
DeepSeek-R1-Zero is the striking proof ✓: trained by pure RL with no SFT stage at all. The paper's own words — "reasoning abilities of LLMs can be incentivized through pure reinforcement learning, obviating the need for human-labeled reasoning trajectories." And the behaviors weren't programmed: self-reflection, verification, and dynamic strategy adaptation emerged on their own (the widely-cited "aha moment"). R1 then added a little SFT for readability on top.
| Piece | What it is |
|---|---|
| GRPO (Group Relative Policy Optimization) | DeepSeek's RL algorithm. Drops PPO's separate value/critic model and instead scores each answer relative to a group of sampled answers → cheaper, less memory than PPO. A big reason R1 was so cheap to train. ✓ |
| Test-time compute | The reasoning shows up as a long chain of thought at inference. The model literally "thinks" longer to get harder problems right — a second scaling axis (spend inference compute, not just training compute). ✓ |
| Generalization debate | RLVR trains on verifiable domains (math/code). Whether that reasoning transfers to non-verifiable domains (writing, judgment) is genuinely contested — the open research question. ⚠ |
Once a big model can reason, you can distill it: train a small "student" on the big "teacher's" outputs (its reasoning traces). DeepSeek distilled R1 into small Qwen and Llama models ✓. The counterintuitive finding: a small model distilled from a strong reasoner often beats the same small model trained with RL directly — it's easier to copy a good reasoner's traces than to rediscover reasoning at small scale. This is the mechanism behind the cheap, fast, surprisingly-capable small models in the market.
| Model type | What it is | Reach for it when |
|---|---|---|
| Base | Raw pretrained predictor | Almost never directly; you're fine-tuning it yourself |
| Instruct / chat | Base + SFT + preference opt | Most tasks: extraction, chat, summarization, classification, tool use. Fast, cheap. |
| Reasoning / thinking | + RLVR, long chain-of-thought | Hard math, code, multi-step planning, agentic loops. Slower, pricier per task, more accurate. |
The security model of LLM apps, and especially agents. One architectural fact drives almost everything: instructions and data share the same token stream. There's no privilege separation between "what the developer told the model to do" and "what some web page it read told it to do."
People conflate these constantly. They're different problems with different fixes.
OWASP ranks prompt injection #1 (LLM01:2025) ✓. And the two flavors:
| Type | How | Threat level |
|---|---|---|
| Direct | The user types a malicious prompt themselves | Manageable — you can distrust the user |
| Indirect | Malicious instructions hidden in data the model retrieves: web pages, emails, docs, code comments, image alt-text, tool outputs | The scary one for agents & RAG — the payload rides in on "trusted" content, often invisible to humans |
Simon Willison's framing (he coined "prompt injection"). The single most useful mental model for deciding whether an agent is dangerous.
An agent becomes dangerous exactly when it combines all three of these. Any two is usually fine. The fix isn't a better filter — it's removing one leg. Willison: "the only way to stay safe there is to avoid that lethal trifecta combination entirely." ✓
Concrete exfiltration channels: a rendered markdown image ( leaks via the image request), a link the model is told to include, a tool call with attacker-chosen parameters, or a pull request to a public repo. The data leaves the moment the model is tricked into building one of these.
Tool use, computer use, and MCP each widen the surface. The root issue: tool metadata (names, descriptions, parameters) is injected into the model's prompt — so merely connecting a tool hands its server text-level influence over your model. ✓
| Incident | Date | What happened |
|---|---|---|
| GitHub MCP "Toxic Agent Flow" (Invariant Labs) | May 2025 | A malicious public GitHub issue prompt-injects an agent reviewing it, making the agent pull private repo data and leak it via a PR to the public repo. Explicitly architectural, not a bug in the MCP server code. ✓ |
| ChatGPT Operator PII exfiltration (embracethered) | Feb 2025 | OpenAI's browsing agent hijacked via indirect injection (payload on a visited page) to lift a user's email/phone/address from authenticated pages to an attacker site. The lethal trifecta in a shipping product. ✓ |
| WhatsApp MCP exploit (Invariant Labs) | 2025 | A benign-looking tool later swaps its definition to steal message history to a hard-coded recipient — rug pull + confused deputy. ✓ |
So you engineer the system, not the model. What genuinely reduces risk:
Your reference frame for the broader landscape. Published Nov 2024, current for 2026. ✓
| # | Risk | # | Risk |
|---|---|---|---|
| LLM01 | Prompt Injection | LLM06 | Excessive Agency |
| LLM02 | Sensitive Information Disclosure | LLM07 | System Prompt Leakage |
| LLM03 | Supply Chain | LLM08 | Vector & Embedding Weaknesses |
| LLM04 | Data & Model Poisoning | LLM09 | Misinformation |
| LLM05 | Improper Output Handling | LLM10 | Unbounded Consumption |
Two research passes (49 sources, 216 claims, 50 fact-checked), then every load-bearing claim re-verified by direct fetch of the primary source — because the automated verification round got throttled by transient API rate-limiting. All ✓ claims below trace to these primaries.