Reasoning Models vs. Chat Models: Picking the Right Tool

A reasoning model answering "summarize this changelog in two sentences" will happily spend 1,500 tokens deliberating about how to write two sentences, then write two sentences that are indistinguishable from what a cheap chat model produced in 200 milliseconds. You paid roughly an order of magnitude more and waited fifteen seconds for a tie.

That is the entire trade in one example. Reasoning models spend tokens before emitting the answer, and those tokens are only worth buying when the problem is one where thinking longer actually changes the outcome. Most production tasks are not.

What the extra tokens buy

The mechanism is search. A reasoning trace lets the model propose an approach, notice it fails, and backtrack, all inside a single forward pass sequence. A standard chat model commits to its first token and then has to remain locally consistent with it; there is no undo. So the extra compute is productive exactly when the problem has a large solution space where a first guess is likely wrong and a wrong guess is recoverable if noticed.

There is a real result behind this rather than vibes. Snell et al., Scaling LLM Test-Time Compute Optimally, show that allocating additional inference compute to a smaller model can outperform a much larger model on the same task, and that the optimal allocation strategy depends on problem difficulty. The second half of that finding is the operationally useful part: the payoff from more thinking is not uniform. It is concentrated on hard instances, and it is near zero on easy ones. Spending a fixed large budget on every request is leaving money on the floor in both directions.

The decision rule

Two properties determine whether reasoning tokens pay for themselves.

Verifiability. Can you, or the model, check a candidate answer more cheaply than producing it? Compiling code, running a test, evaluating a constraint, substituting a value back into an equation. If yes, extra compute converts into accuracy, because search plus a cheap oracle converges. If the answer is a matter of taste (tone, phrasing, a marketing headline), there is nothing to converge to and the deliberation is decorative.

Search depth. How many interdependent decisions separate the input from the answer? One hop means the answer is essentially a lookup or a transformation. Ten hops with backtracking means a first guess is almost certainly wrong somewhere.

Cross those and you get a clean split.

• Reasoning earns its cost: competitive-programming-shaped problems, math with a checkable result, scheduling and constraint satisfaction, multi-step migration plans where step seven depends on step three, root-causing a bug from a stack trace plus source, refactors that must preserve semantics across files.
• Reasoning burns money: classification into known labels, extraction into a fixed schema, summarization, translation, reformatting, intent routing, and anything where the hard part was retrieval. If the bottleneck is that the necessary fact was not in the context, no amount of thinking synthesizes it. A reasoning model with bad retrieval produces a beautifully argued wrong answer.

That last case is worth naming because it is the most common misdiagnosis in production. Teams see low accuracy, escalate to a reasoning model, see a marginal improvement, and conclude they need the expensive model everywhere. What they usually needed was better context assembly.

Quantify before you commit

The cost is not the sticker price per token, it is the token count you cannot see. On a moderately hard problem, a reasoning trace commonly runs several times the length of the final answer, and on genuinely hard instances it can exceed it by more than an order of magnitude. Latency tracks that almost linearly, because decoding is sequential: a 4,000-token trace at 60 tokens per second is over a minute of wall clock before the user sees the first character of the answer.

Two consequences follow. First, streaming does not save you, because there is nothing to stream during the thinking phase. Second, reasoning models are structurally wrong for interactive request paths with a p95 latency budget. Put them behind a job queue, a background worker, or an explicit "think harder" affordance the user opts into.

Route, do not choose

The practical answer is not to pick one model. It is to try the cheap one and escalate on evidence.

def answer(task):
    draft = cheap_model(task)                   # ~200ms, ~1x cost

    ok, reason = validate(draft, task)          # compile, schema-check,
    if ok:                                      # unit test, or judge
        return draft

    log_escalation(task, reason)
    return reasoning_model(task, hint=reason)   # ~30s, ~15x cost

Everything depends on validate. Make it as close to a real oracle as the domain permits: run the generated SQL against a read replica with EXPLAIN, parse the JSON against the schema, execute the test suite, check that every cited claim maps to a retrieved chunk. A validator you trust turns model selection into an economics problem with a known answer.

Where no oracle exists, self-reported confidence is a weak substitute and token-level logprobs on the answer span are a better one, but be honest that both are noisy. It is often cheaper to accept a fixed escalation rate on a coarse heuristic (input length, presence of a numeric or code task, retrieval score below a threshold) than to build a confidence model you cannot calibrate.

The number to watch after you ship this is the escalation rate. If 3% of traffic escalates, you are paying near-cheap-model prices for near-reasoning-model accuracy, and the architecture is working. If 60% escalates, your cheap model is wrong for the workload and the router is just added latency. If 0% escalates, your validator is not actually checking anything.

One note relevant to memory systems specifically: escalation is most useful when both tiers see the same context, otherwise you cannot tell whether the reasoning model won because it thought harder or because it happened to get better inputs. Unimatrix keeps that context in one store shared across models, which also means a decision reached by the expensive model on Monday is available to the cheap one on Tuesday. The tool surface is in the MCP reference.