Benchmarking LLM Memory Systems: Metrics That Actually Matter

Recall@10 is the metric everyone reports and it answers a question nobody asked. It tells you whether the gold chunk landed somewhere in a set of ten. It does not tell you whether the model answered correctly, whether the nine other chunks contradicted the right one, or whether those ten chunks fit in the budget you actually have at inference time. A system can go from 78% to 91% Recall@10 and get worse for users, because the extra recall arrived as four near-duplicates that pushed the decisive fact below the truncation line.

The fix is not a better retrieval metric. It is to stop measuring retrieval in isolation and start measuring the thing the user experiences, under the constraint the system actually runs with.

Fix the token budget first, then measure

Every retrieval evaluation should be parameterized by a retrieved-token budget, not by k. Reporting "top-10" is meaningless across systems because one system's chunk is 120 tokens and another's is 900. Pick a budget that reflects production. Two thousand tokens of retrieved context is a reasonable default for a chat assistant that also needs room for a system prompt, the user turn, and a few thousand tokens of recent history.

Then the headline number is answer correctness at 2k retrieved tokens. Graded by an LLM judge against a reference answer, or by exact match where the answer is a short span. This single change reorders leaderboards, because it prices verbosity. A system that returns one 80-token fact that answers the question beats a system that returns 2000 tokens containing the same fact plus noise, and Recall@10 scores them identically.

Run the same eval at 500, 2000, and 8000 tokens. The shape of that curve is diagnostic. If correctness is flat from 2k to 8k, your ranker is fine and your ceiling is elsewhere. If it climbs steeply, your ranking is bad and you are compensating with volume, which will cost you latency and money in production.

Four metrics that catch what correctness misses

Contradiction rate.Over a multi-session trace, count the fraction of answers that conflict with a fact established earlier in the same trace. This is the metric that catches the failure users describe as "it forgot what I told it." Compute it by holding a session-ordered ground truth state and checking each answer against the state as of that turn. A memory system with 85% correctness and a 2% contradiction rate is more usable than one with 88% correctness and a 15% contradiction rate, because contradictions destroy trust asymmetrically.

Staleness rate. Of the facts retrieved into context, what fraction have been superseded by a newer entry that was also available in the store? This isolates a specific bug: your index has the current answer and your ranker chose the old one. It is measurable without any model in the loop, which makes it cheap to run on every commit. Anything above a few percent means your scoring function has no time term, or the supersession link was never written.

Write precision. The fraction of stored memories that were ever retrieved into a context window that produced a correct answer. Most systems have never measured this and the number is usually brutal, well under 20% for anything that stores every conversation turn. Low write precision is not harmless bloat. It raises p95 latency, dilutes top-k, and makes the store more expensive to embed and re-embed. It is the metric that justifies a write policy.

p95 retrieval latency, measured at production store size. Not p50, and not on a 500-row fixture. HNSW recall degrades as you tune ef_search down to hit a latency target, so latency and quality are the same dial. Report them together or the numbers are not comparable.

Single-shot QA cannot measure memory

This is the part most evaluations get structurally wrong. If your eval is a set of independent (question, corpus, answer) triples, you are benchmarking retrieval over a static document set. Memory problems only exist across time: a fact stated in session 3 and revised in session 9, a preference expressed once and never repeated, a reference to "the thing we decided last week."

Use a benchmark built on sessions. LongMemEval is the most directly useful one: 500 questions over long interactive histories, split into abilities including information extraction, multi-session reasoning, temporal reasoning, knowledge updates, and abstention. That last category matters and is almost always omitted. A memory system that confidently answers questions it has no stored basis for is worse than one that says it does not know, and only an abstention split will show you that. LoCoMo covers similar ground with very long multi-session dialogues and is worth running as a second signal.

The harness shape is small enough to write in an afternoon:

for trace in traces:                       # a trace = ordered sessions for one user
    store.reset()
    truth = {}                             # entity -> (value, session_idx)

    for i, session in enumerate(trace.sessions):
        for turn in session.turns:
            store.write(turn)              # exercise the real write policy
            truth.update(turn.facts)       # ground-truth state as of session i

        for q in session.probes:
            ctx = store.retrieve(q, token_budget=2000)
            ans = model(q, ctx)

            log.correct(judge(ans, q.reference))
            log.contradiction(conflicts(ans, truth))
            log.stale(fraction_superseded(ctx, truth))
            log.latency(store.last_latency_ms)

    log.write_precision(store.retrieved_useful_ids / store.written_ids)

Two details make or break it. The store must be reset per trace, or facts leak between users and contradiction rate becomes noise. And the write path must be the production write path, including dedup and summarization, because write precision is meaningless if the harness bypasses the policy you ship.

If a change improves Recall@10 and leaves correctness-at-2k flat, you did not improve the system. You moved a number that was never load-bearing.

What to do with the results

Treat these five numbers as a vector, not a score. Different fixes move different components, and knowing which one moved tells you what you actually changed. Hybrid retrieval moves correctness at small budgets. A recency term moves staleness. Write-time dedup moves write precision and p95 together. Supersession links move contradiction rate and nothing else. If you only track one aggregate, you lose the ability to attribute.

We run this shape of harness against Unimatrix on multi-session traces because the cross-device case makes it unavoidable: a fact written from a phone in one session has to beat a stale one written from a laptop three weeks earlier. The retrieval and supersession behavior that gets measured is documented in the MCP reference.