Few-Shot Prompting Still Beats Fine-Tuning for Most Teams

Fine-tuning is a compiler for behavior. You compile a specification into weights, and after that the behavior is cheap to invoke because you no longer ship the specification with every request. That is a genuine win. The problem is that recompiling costs you a dataset, a training run, an eval harness, and a week, and most teams change the specification more often than once a week.

If the build takes longer than the interval between source changes, you never ship a build that matches the source. That is the whole argument, and the rest of this post is the arithmetic and the cases where it flips.

What you are actually buying

Fine-tuning buys two things and people conflate them.

Token savings. Behavior encoded in weights does not occupy the prompt. Drop 8 examples averaging 250 tokens each and you have removed roughly 2,000 input tokens from every call. At a million calls a month that is 2 billion input tokens, which is real money.

Format compliance. This is the underrated one. If you need output that parses on the first try, every time, with no preamble and no markdown fence, a fine-tune on a few thousand correctly formatted examples is more reliable than any amount of prompt instruction. Constrained decoding gets you most of the way there for free, so check that first, but for idiosyncratic formats a fine-tune still wins.

What it does not buy is knowledge. Fine-tuning on your documentation does not make the model know your documentation; it makes the model sound like your documentation. Facts belong in retrieval. Teams that fine-tune to inject knowledge get a model that confidently invents things in the right house style, which is worse than a model that says it does not know.

The break-even, written out

Three conditions have to hold simultaneously for the compile to pay off.

• Stability.The spec's half-life exceeds your retraining cycle. If the output format, the label taxonomy, or the policy changes monthly and a full cycle (relabel, retrain, re-eval, redeploy) takes two weeks, you are permanently serving a stale artifact and spending engineering time to stay stale.
• Volume. Per-call token savings have to amortize the fixed cost. The fixed cost is not the GPU bill, it is the dataset and the eval infrastructure, which is engineer-weeks. Sketch it: if you save 2,000 input tokens per call, you need on the order of a million calls before the savings are comparable to two engineer-weeks of fully loaded cost. Below roughly 100k calls a month, the spreadsheet does not close.
• Narrowness. One task, one output shape. Fine-tuning a model to do six loosely related things produces a model that is mediocre at all six, and every future change to any one of them forces a retrain of all six.

The condition people skip is eval infrastructure, and skipping it is how fine-tuning goes wrong quietly. Without a held-out set and a regression suite, you cannot distinguish "the new checkpoint is better" from "the new checkpoint overfit and lost a capability nobody was measuring." You need that harness before your first training run, not after your first regression. And once you have built it, notice that you now have exactly the tooling that would have let you iterate on prompts with confidence, at which point the marginal case for training weakens further.

Dynamic examples, which a fine-tune cannot do

This is the argument I find most persuasive and it rarely comes up. A fine-tune is a fixed function. The examples are baked in at training time, averaged across the whole distribution, and every request gets the same learned prior. Few-shot examples are chosen at request time, which means they can be chosen for that request.

def build_prompt(query, labeled_pool):
    # retrieve the k nearest labeled examples to this specific input
    nn = labeled_pool.search(embed(query), k=20)
    shots = mmr(nn, lambda_=0.7, k=4)      # relevance, then de-duplicate
    return SYSTEM + render(shots) + user(query)

The effect is that a query about a rare edge case gets shown examples of that edge case, rather than four examples of the common case plus a fine-tuned prior that has smoothed the edge case away. On long tails this is a large difference, and it gets better every time someone labels a new example, with no retraining. A mislabeled example can be deleted from the pool in one query. A mislabeled example baked into weights requires a new training run. The mmr step matters: pure nearest-neighbor selection tends to return four paraphrases of the same example, which wastes three of your four slots.

Three to five examples, chosen for diversity

The scaling behavior of in-context learning has been known since Brown et al., Language Models are Few-Shot Learners, which measured accuracy against shot count across dozens of tasks. The curve rises steeply from zero to a handful of examples and then flattens hard. On current instruction-tuned models the flattening is even earlier, because the model has already seen the task format during post-training. It needs a pointer to which behavior you want, not a training set.

The practical rule: 3 to 5 examples, and select them to span the decision boundary rather than to cluster near the mode. Two examples of the two most confusable classes teach more than eight examples of the majority class. If you have a hard case that your system gets wrong, one example of that case is worth more than doubling the total count.

Two failure modes worth knowing. First, ordering matters more than it should; the label distribution and the position of the final example can bias predictions measurably, so if you are selecting dynamically, keep the order deterministic and check it in your golden set. Second, examples that share an incidental surface feature (all your positive examples happen to be long) teach the model that feature instead of the task.

The sequence I would actually follow

Prompt with 3 to 5 static examples. Add retrieval so the examples are dynamic. Add constrained decoding if format compliance is the failure. Measure. If, after all of that, you are still burning tokens on a stable high-volume narrow task and the numbers close, then fine-tune, and keep the prompt-based version as your fallback and your baseline.

Most teams never reach step five, and that is not a failure of ambition. It means the prompt was sufficient, which is the cheapest possible outcome.

Dynamic example selection needs a store of labeled examples with embeddings and a fast nearest-neighbor query, which is the same infrastructure a memory layer already is. We use Unimatrix's own recall path for exactly this, and the prompt patterns are on the prompts page.