Chain-of-Thought Alternatives: Tree Search and Self-Consistency

Self-consistency costs exactly k times as much as a single call and it is still usually the right first thing to try. The method is three lines of pseudo-code: sample k chains of thought at temperature above zero, extract the final answer from each, return the mode. No verifier, no search tree, no prompt engineering beyond what you already had. On grade-school math benchmarks the original paper reported roughly a 17-point absolute improvement over greedy chain-of-thought decoding, which is a larger jump than most people get from swapping to a bigger model.

The reason it works is worth stating precisely, because it tells you when it will not work.

Wrong answers scatter, correct answers converge

A model reasoning through a multi-step problem at temperature 0.7 makes a different sequence of choices on every sample. When it goes wrong, it goes wrong in a path-dependent way: it drops a factor of two here, misreads a constraint there, inverts a sign somewhere else. Each distinct error produces a distinct final value. When it goes right, every correct path lands on the same value, because there is only one correct answer.

So the distribution over final answers is one tall spike plus a long thin tail of one-off mistakes. The mode of that distribution is a much better estimator of the truth than any single draw from it, even when no individual sample is more likely than not to be correct. Concretely: if each chain is independently correct with probability 0.45 and the remaining 0.55 spreads across six different wrong values at roughly 0.09 each, then the plurality vote over five samples picks the right answer far more than 45% of the time, because 0.45 beats 0.09 by a wide margin and majority voting amplifies that gap.

The algorithm:

def self_consistent(prompt, k=5, temp=0.7):
    answers = []
    for _ in range(k):
        chain = model(prompt, temperature=temp)
        a = extract_answer(chain)      # must be normalizable
        if a is not None:
            answers.append(normalize(a))
    return Counter(answers).most_common(1)[0][0]

The load-bearing function is extract_answer composed with normalize. Two chains that both concluded 3/4 must produce the same key, whether one wrote 0.75 and the other wrote 3/4. If your normalizer treats those as different answers, you have silently split the correct-answer spike in half and the vote can lose to a wrong answer that happened to be written consistently.

The hard requirement

Self-consistency needs an answer that is extractable and comparable. Numbers, multiple-choice letters, booleans, dates, class labels, a JSON field with a closed vocabulary: all fine. Free-form prose: not fine. Five paragraph-length answers to "summarize this incident report" will be five distinct strings, the counter will show five keys with count 1, and you will have paid 5x for a random pick.

People try to patch this by clustering the outputs by embedding similarity and taking the largest cluster centroid. That is a defensible reranking heuristic but it is not self-consistency, and it loses the property that made the original method trustworthy: exact agreement on a discrete value is evidence, and cosine proximity between two paragraphs is not.

Where the gains curve flattens

The empirical shape is a steep rise then a plateau. Going from k=1 to k=5 captures most of the available improvement. k=10 adds a little. k=40 adds very little over k=10 and costs four times more. This follows directly from the mechanism: once you have enough samples to reliably distinguish a 0.45 spike from a 0.09 tail, more samples only sharpen a decision you were already making correctly. What additional samples cannot fix is a systematic error. If the model misreads the problem the same way every time, all k chains agree on the wrong answer and self-consistency returns it with high confidence. Majority voting reduces variance. It does nothing about bias, and it makes bias look like certainty.

The full ablation is in Wang et al., Self-Consistency Improves Chain of Thought Reasoning.

Tree search, and the scoring function problem

Self-consistency samples k complete solutions independently and never looks inside them. Tree of Thoughts spends the same budget differently: build partial solutions, score them, keep the promising ones, expand those.

frontier = [empty_partial]
for depth in range(max_depth):
    candidates = []
    for node in frontier:
        candidates += propose_steps(node, n=3)   # model generates next steps
    scored = [(score(c), c) for c in candidates]  # <-- everything depends on this
    frontier = top_b(scored, b=5)                 # beam width 5
return best(frontier)

This is beam search with an LLM as the successor function. It buys you something real that self-consistency cannot: the ability to abandon a bad prefix at step 2 instead of paying for the remaining eight steps of a doomed chain, and the ability to backtrack. On problems where a wrong early commitment is unrecoverable (the paper's Game of 24 and creative-writing tasks are good examples), that structure is the whole win. See Yao et al., Tree of Thoughts.

The catch is score(). Tree search converts your compute budget into accuracy only in proportion to how well the scoring function ranks partial solutions. With a perfect scorer, a beam of 5 is dramatically better than 5 independent samples. With a scorer that is uncorrelated with correctness, beam search is worse than sampling, because you have concentrated your budget on whatever the noise happened to favor and thrown away the diversity that made voting work.

In practice the scorer is usually the same model, prompted to rate its own partial work. That is the weak link, for a structural reason: if the model could reliably tell good reasoning from bad, it would not have generated the bad reasoning. Self-evaluation correlates with correctness but it also correlates with fluency and confidence, and the failure mode is a smooth, well-written wrong branch outscoring an awkward correct one.

So the decision rule is fairly blunt:

• Cheap external verifier available (unit tests, a type checker, a SAT solver, an interpreter, a known constraint to check against): use tree search. The scores are real, and search compounds.
• Discrete comparable answer but no verifier: use self-consistency at k around 5 to 10. Simple, no tuning, predictable cost.
• Free-form output and no verifier: neither method applies cleanly. Spend the budget on retrieval, better context, or a stronger model instead.

Both techniques are pure inference-time compute, which is the interesting part: they trade tokens for accuracy without touching weights, and they stack with anything else in the prompt. They also both get more expensive linearly in k, so the context you feed them matters more, not less. Unimatrix keeps the accumulated facts about a project available to every call, which means a k=10 vote is voting on the right problem rather than ten times re-deriving what the user already told you last week. There are worked prompt patterns in prompts.