Reasoning Models Are Trained, Not Prompted

Here's an opinion I defended badly at a team lunch last year and have since come to believe even more strongly: if your mental model of a reasoning model is "a regular LLM that was told to think step by step," you don't understand reasoning models at all — and that misunderstanding is quietly costing you money and accuracy in production. Reasoning models are not a prompting technique. They're a training artifact. The chain-of-thought behavior that makes them good at math and code emerges from reinforcement learning with verifiable rewards, not from anything you write in a system prompt. I finally internalized this properly while working through the same ground Sebastian Raschka covers in his new 11-hour build-a-reasoning-model course: load a small Qwen3 base model, build a verifier, run GRPO, and watch the reasoning traces get longer and the accuracy climb. Once you've seen it happen in your own training loop, you can't unsee it.

Let me defend that claim properly, then walk it back where it deserves walking back — because there is one place where prompting genuinely does matter, and pretending otherwise would be hype of a different flavor.

Why the "just prompt it" mental model breaks

The popular story goes like this: in 2022, researchers discovered that appending "Let's think step by step" to a prompt dramatically improved arithmetic accuracy, and chain-of-thought prompting was born. True as far as it goes. What the story leaves out is what happened next — labs took that behavior and burned it into the weights with reinforcement learning. A modern reasoning model doesn't need your step-by-step incantation. It generates extended reasoning traces by default because it was trained to, rewarded for producing traces that terminate in correct answers.

You can verify this yourself with an experiment that takes an afternoon. Take a base model — Qwen3 0.6B works fine and runs on a single consumer GPU — and evaluate it on MATH-500 with and without chain-of-thought prompting. You get a modest bump. Now take the RLVR-trained variant of the same model family. The gap between those two conditions is not a bump; it's a cliff. No system prompt ever written closes it. That's the empirical core of my claim: the capability lives in the weights, and the weights got that way through training, not through clever token arrangements at inference time.

The confusion persists because from the outside, both look identical: the model emits a long stream of intermediate reasoning before answering. But surface similarity hides a causal difference. Prompting a base model to think step by step is like asking someone to narrate their thought process while doing mental arithmetic — the narration follows the ability. RL training actually changes the ability itself, because the model is optimized end-to-end for answer correctness, and extended reasoning happens to be an instrumentally useful strategy for getting correct answers on hard problems.

The verifier is the whole trick

When I was an ML engineer, the part of RLHF that always felt slightly cursed was the reward model. You train a separate neural network to imitate human preferences, then you optimize your policy against it, and then you spend the rest of the project fighting reward hacking — the policy finds some degenerate output that scores high with the reward model while being useless to actual humans. It's an adversarial relationship baked into the architecture.

Reinforcement learning with verifiable rewards — RLVR — sidesteps this entirely, and building a verifier from scratch makes you appreciate how elegant the move is. For math, the verifier is a boring, deterministic program: extract the final answer from the model's output, normalize it (handle fractions, decimals, equivalent forms like 1/2 vs 0.5), and check mathematical equivalence. Right answer, reward of one. Wrong answer, zero. There is no learned reward model to hack. The reward signal is ground truth.

This is why reasoning models got good at math and code first. Both domains come with cheap, reliable verifiers — symbolic equivalence checkers for math, unit tests for code. You don't need human annotators ranking outputs; you need a Python function that returns a boolean. The practical consequence for anyone building domain-specific reasoning is stark: if you can write a verifier for your domain, you can probably train a reasoner for it. If you can't — if correctness in your domain is a matter of taste, like marketing copy — you're stuck with RLHF and its reward-hacking headaches.

The bottleneck for reasoning in a new domain isn't compute or data. It's whether you can write down what "correct" means as a program.

One detail I love about building the verifier first, before any training: it does double duty. The same grader that scores your base model on MATH-500 for evaluation becomes the reward function during RL. When you implement it yourself you notice how much subtle work it does — answer extraction is genuinely fiddly, because models express final answers in dozens of formats, and a brittle extractor silently corrupts your reward signal. A model that writes "therefore x = 7" might get graded wrong if your regex only looks for boxed answers. Sloppy verifiers teach sloppy policies. This is the kind of lesson you simply do not learn from reading papers.

What GRPO actually does to the weights

The training loop itself, in its simplified from-scratch form, is surprisingly compact. Group Relative Policy Optimization — the algorithm DeepSeek popularized — drops the critic network that made PPO such a memory hog. Instead of learning a value function, you generate a group of responses to the same prompt, score them all with the verifier, and compute each response's advantage relative to the group average. Responses better than the group mean get their token probabilities pushed up; worse ones get pushed down.

def grpo_advantages(rewards: torch.Tensor) -> torch.Tensor:
# rewards: [group_size], one per sampled response to the same prompt
mean = rewards.mean()
std = rewards.std().clamp_min(1e-4)
return (rewards - mean) / std  # relative advantage per response
# Training step: maximize advantage-weighted log-probability
def grpo_loss(logprobs: torch.Tensor, advantages: torch.Tensor) -> torch.Tensor:
# logprobs: summed sequence log-prob under the current policy
return -(advantages.detach() * logprobs).mean()

Real implementations add a KL penalty to keep the policy from drifting too far from the reference model, plus clipping borrowed from PPO, but the skeleton above is honestly the core idea. And here's what makes it magical to watch: you don't tell the model anything about reasoning style. You only reward correct final answers. Yet over training, response lengths grow. The model starts producing self-corrections mid-trace — "wait, let me check" — because traces with verification behavior are more likely to end in correct answers, and correct answers are the only thing that earns reward. The chain-of-thought style is an emergent solution to the optimization problem, not a behavior anyone engineered. That's why I say it's trained, not prompted: the prompting-era version of this behavior was humans imitating what RL later discovered on its own.

There's a pragmatic engineering note buried here that the hype cycle skips: a 0.6B model with a decent GPU is enough to see this dynamic for real. You'll get measurable MATH-500 improvement from GRPO on hardware you can rent for a few dollars an hour. You will not get a frontier model. That's fine — that's not the point of the exercise.

Inference-time scaling: where compute replaces training (partially)

Before any RL happens, there's a family of techniques that improve answers with the weights frozen: sample multiple candidates with temperature and top-p, then combine them. Self-consistency is the workhorse — generate k answers, take a majority vote over final answers. It works embarrassingly well, and it works on base models too, which is the first crack in my opening claim that I'll concede: inference-time compute is a real, orthogonal axis of improvement.

But look at what these techniques actually require. Majority voting only helps if your model already has a non-trivial probability of producing correct answers — voting amplifies signal that exists, it doesn't create signal from nothing. Scoring-based selection (ranking candidates by sequence log-probability) has the same property. And self-refinement loops, where the model critiques and revises its own answer, turn out to be disappointingly weak on base models, because honest self-critique is itself a capability that RL installs. Every inference-time trick composes with training; none substitutes for it. The reason frontier systems combine RL-trained models with inference scaling is that the two multiply rather than add.

The cost side matters for practitioners. Self-consistency with k=16 costs sixteen times the inference. Whether that's a good trade depends entirely on your accuracy-per-dollar curve, and the only honest way to answer it is to plot MATH-500 accuracy against candidate count yourself. We covered a similar compute-placement question in our look at flash-tier models and inference efficiency — the recurring theme is that "spend more compute" has at least three different meanings, and they're not interchangeable.

Distillation, and the one place prompting survives

The final stage in the from-scratch pipeline is distillation: take the long reasoning traces of a strong (usually large) reasoner and fine-tune a smaller model on them directly, with plain supervised learning. No RL, no verifier in the loop. This is how many of the cheap, small reasoning models you'll actually deploy got made. And it reinforces the thesis rather than undermining it — the reasoning behavior is still coming from training data produced by an RL-trained model, just transferred through gradient descent instead of reward. The capability is never injected at prompt time; it's always baked in by optimization.

So where does my contrarian claim break down? Here, honestly: prompting still matters at the margins, and the margins aren't zero. Reasoning effort controls — the low/medium/high thinking-budget settings that current-generation models expose — are inference-time levers that meaningfully trade accuracy for latency and cost. Structured output constraints, verifier-in-the-loop pipelines, and constrained decoding are all prompt-and-harness-level techniques with real value, as we argued in our piece on constrained decoding. And if you're stuck with a base model and can't train anything, chain-of-thought prompting is still worth real points on evals. "Trained, not prompted" is a claim about where the capability originates, not a claim that inference-time choices don't matter.

Why building it beats reading about it

A fair objection surfaced in the usual places: isn't a build-your-own-reasoning-model tutorial like a build-your-own-CPU guide — charming but useless, given that real training runs need clusters and proprietary data? I think this misses what the exercise is for, and I'd say the same about the classic build-a-shell tutorial, which we praised in What Building a Shell Teaches You About Unix for exactly this reason. Nobody writes a shell to replace bash. You write one because fork, exec, pipes, and signal handling stop being abstractions you've memorized and become machinery you've held.

Same here. The things that permanently changed how I work with reasoning models came from implementation details, not papers:

• Writing the verifier taught me that reward quality is a software engineering problem — normalization edge cases silently distort training, and "the reward is ground truth" is only as true as your answer extractor.
• Implementing sequence log-probability scoring (with the numerical-stability gymnastics that entails) demystified half the loss functions in the post-training literature. They all reduce to comparing how likely responses are under different policies.
• Watching response length grow during GRPO, unprompted, killed my last residual belief that reasoning style is a surface behavior. It's what optimization finds when correctness is the only objective.
• Plotting accuracy against candidate count for self-consistency gave me an actual intuition for the accuracy-per-dollar trade that no benchmark table ever did.

The compute anxiety is also overblown for the learning use case. The early parts — loading a base model, streaming generation with KV caching, building the verifier, running MATH-500 — are CPU-and-laptop territory or close to it. GRPO training wants a GPU, but a single one, and rented by the hour it's cheaper than most textbooks. What you can't replicate at home is scale, and scale is genuinely a different discipline — that's one of the myths about DeepSeek's inference engine we tested: the serving stack of a frontier lab solves problems that simply don't exist at tutorial scale, and vice versa.

The pragmatic takeaway

If you take one thing from all this, take the verifier-centric worldview. The modern reasoning stack is: a base model, a program that decides what "correct" means, an RL algorithm that optimizes against it, and inference-time techniques that multiply the result. Prompting sits at the very edge of that system, steering a capability that training created. Teams that understand this spend their effort where it compounds — better verifiers, better reward design, better evaluation loops like the ones we've argued are missing from most AI benchmarks that miss real work. Teams that don't keep tweaking system prompts and wondering why accuracy plateaus.

My lunch-table claim survives with one asterisk: reasoning models are trained, not prompted — but the harness you build around them, prompts included, decides how much of that trained capability you actually harvest. Build the small version yourself sometime. Eleven hours of coding along is a cheap price for never being confused about this again.