Memory benchmarks
How Counterparts does on memory benchmarks.
We ran Counterparts on LongMemEval, LoCoMo and BEAM, measured cleanly: the benchmark’s own judge, every question, one run per row, and per-question results kept for anyone to check. Then we scored our answers the way our competitors score theirs, so the numbers can sit side by side. We answer with Claude Haiku 5.5, a small model; most systems above us on any chart answer with a larger one.
Highlights
LoCoMo, scored with Mem0's own judge
94.3%
+1.8 points over Mem0
Mem0 as published, answering with gpt-5; we answer with Claude Haiku 5.5.
BEAM 100K, as shipped
68.2%
+5.2 points over Honcho
Same judge as Honcho. Knowing when there's nothing to answer: 72.5% vs Honcho's 36.2%.
Memory read per answer, LongMemEval
~3,931
Leanest reported; 3.0× fewer than Honcho
Tokens, as each system reports it; fewer is cheaper. Reading the whole history is ~122,000.
LongMemEval-S, official judge
91.4%
+1.0 points over Honcho
Same judge, same 500 questions. The top score answered by a Claude Haiku model.
LoCoMo, same judge as Honcho
91.6%
+1.6 points over Backboard
Each as published; Backboard and Hindsight answer with Gemini models.
BEAM 100K, ability by ability
Twenty conversations of about 100,000 tokens each, 400 questions over ten abilities. The number beside each ability is our lead over Honcho, in points. Sorted by that lead.
- Counterparts
- Other systems
- No memory: the model reads everything
Knowing there's nothing to answer+36.3
Keeping up with updates+11.8
Reasoning about time+10.0
Summarizing+7.6
Following instructions+3.7
Following preferences+0.7
Resolving contradictions−1.5
Ordering events−2.3
Pulling out facts−6.8
Reasoning across sessions−7.3
The whole field
Every system’s own published number, as they state it, with the model that writes its answers. Hover a bar for its judge and how it was scored. The tags say how a number was made; they don’t adjust it.
- Counterparts
- Other systems
- No memory: the model reads everything
LoCoMo’s answer key has about 6.4% errors, so ~93.6% is the honest ceiling.
| Benchmark | System | Score | Answer model | Judge | What the answerer sees | How it was scored | Source |
|---|---|---|---|---|---|---|---|
| LongMemEval-S | Mastra Observational Memory | 94.9 | gpt-5-mini | gpt-4o, official prompts | Memory + transcript (observation log + recent raw messages, ~30k tokens) | averaged by question type | link |
| LongMemEval-S | Mem0 (platform) | 94.4 | gpt-5 (per result file) | gpt-5, own judge prompt | Memory only (top 200 memories) | own judge prompt; answer rules fitted to test questions | link |
| LongMemEval-S | Backboard | 93.4 | not stated | four unnamed judges; highest shown | not stated | best of four judges (91.2 to 93.4); no method published | link |
| LongMemEval-S | ByteRover | 92.8 | Gemini 3 Flash + Gemini 3.1 Pro (writes the answer) | Gemini 3 Flash | Memory only (agentic retrieval) | second model writes the final answer; own adapted prompt | link |
| LongMemEval-S | Honcho (Gemini 3 Pro) | 92.6 | Gemini 3 Pro | gpt-4o, official prompts | Memory + transcript search | question type picks which side is observed and asked | link |
| LongMemEval-S | Counterparts | 91.4 | Claude Haiku 5.5 | gpt-4o, official prompts | Memory + transcript search | official judge; every question; one run; per-question results published | ours |
| LongMemEval-S | Hindsight | 91.4 | Gemini 3 Pro | GPT-OSS-120b | Memory only | link | |
| LongMemEval-S | Honcho | 90.4 | Claude Haiku 4.5 | gpt-4o, official prompts | Memory + transcript search | question type picks which side is observed and asked | link |
| LongMemEval-S | Zep (2026) | 90.2 | GPT-5.4 | GPT-5.4 | Memory + source episodes | link | |
| LongMemEval-S | Counterparts, as shipped | 89.4 | Claude Haiku 5.5 | gpt-4o, official prompts | Memory only | official judge; every question; one run; per-question results published | ours |
| LongMemEval-S | EmergenceMem (Internal) | 86.0 | not stated for Internal | not stated | not stated | no code published | link |
| LongMemEval-S | Supermemory | 85.2 | Gemini 3 Pro | gpt-4o | Memory + source chunks | link | |
| LongMemEval-S | Zep (2025) | 71.2 | gpt-4o | gpt-4o, official prompts | Memory only | link | |
| LongMemEval-S | No memory: Haiku 4.5 reads everything | 62.6 | Claude Haiku 4.5 | gpt-4o | Whole history in the prompt | link | |
| LongMemEval-S | No memory: GPT-4o reads everything | 60.6 | gpt-4o | gpt-4o, official prompts | Whole history in the prompt | link | |
| LoCoMo | ByteRover | 96.1 | Gemini 3 Flash + Gemini 3.1 Pro | Gemini 3 Flash | Memory only (agentic retrieval) | 1,982 questions, not the standard 1,540; second model writes the answer; above the answer key's ~93.6% ceiling | link |
| LoCoMo | Zep (2026) | 94.7 | GPT-5.4 | GPT-5.4 | Memory + source episodes | link | |
| LoCoMo | Counterparts, under Mem0's judge | 94.3 | Claude Haiku 5.5 | gpt-5, Mem0's judge prompt | Memory + transcript | scored with Mem0's own judge | ours |
| LoCoMo | Counterparts, as shipped, under Mem0's judge | 93.1 | Claude Haiku 5.5 | gpt-5, Mem0's judge prompt | Memory only | scored with Mem0's own judge | ours |
| LoCoMo | Mem0 (platform) | 92.5 | gpt-5 (per result file) | gpt-5, own judge prompt | Memory only (top 200 memories) | own judge prompt; re-ran missed questions | link |
| LoCoMo | EverMemOS | 93.0 | gpt-4.1-mini (paper) | gpt-4o-mini + 2 unnamed, averaged | Memory only | three judges averaged, two unnamed | link |
| LoCoMo | Mnemon | 91.7 | gpt-4.1-mini | gpt-4.1-mini, lenient prompt | Memory only | lenient judge prompt | link |
| LoCoMo | Counterparts | 91.6 | Claude Haiku 5.5 | gpt-4o-mini, Honcho's prompt | Memory + transcript (whole conversation) | official judge; every question; one run; per-question results published | ours |
| LoCoMo | No memory: Haiku 5.5 reads everything | 90.3 | Claude Haiku 5.5 | gpt-4o-mini, Honcho's prompt | Whole conversation in the prompt | our run | ours |
| LoCoMo | Backboard | 90.0 | Gemini 2.5 Pro | GPT-4.1, lenient prompt | not stated | own judge prompt | link |
| LoCoMo | Honcho | 89.9 | Claude Haiku 4.5 | gpt-4o-mini | Memory + transcript (whole conversation) | per-conversation mean | link |
| LoCoMo | Hindsight | 89.6 | Gemini 3 Pro | GPT-OSS-120b | Memory only | link | |
| LoCoMo | Counterparts, as shipped | 87.8 | Claude Haiku 5.5 | gpt-4o-mini, Honcho's prompt | Memory only | official judge; every question; one run; per-question results published | ours |
| LoCoMo | No memory: Haiku 4.5 reads everything | 83.1 | Claude Haiku 4.5 | gpt-4o-mini | Whole conversation in the prompt | our recount on the 1,540 questions of Honcho's baseline file (their page: 75.6% over all 1,986) | link |
| LoCoMo | Zep (2025) | 75.1 | not stated | not stated | Memory only | link | |
| LoCoMo | Letta (file search agent) | 74.0 | gpt-4o-mini | not stated | Transcript as a searchable file | link | |
| LoCoMo | No memory: GPT-4o-mini reads everything | 72.9 | gpt-4o-mini | unnamed LLM judge | Whole conversation in the prompt | link |
Same judge, side by side
“Memory only (as shipped)” is what a Counterparts user gets. “Memory + transcript” adds search over the raw messages, and on LoCoMo the whole conversation in the prompt, the way Honcho runs its benchmark.
gpt-4o with the official prompts, all on the same 500 questions.
gpt-4o-mini with Honcho's prompt, scored per question (LoCoMo has no official LLM judge).
Our same answers, re-judged with Mem0's published judge prompt and its judge model, gpt-5.
Mem0 answers with gpt-5, and gpt-5 as judge rejects hedged answers, so here we trail.
Mean rubric score over 400 questions, Sonnet 4.5 judging as in Honcho's run.
Memory read per answer
Memory text the answering model read per question (recall results, footnotes, prefetch, and any pasted transcript), as a share of the whole history, counted as characters ÷ 4. Claude Code’s fixed system prompt is left out. Other systems’ counts are their own and may be measured differently.
| Benchmark | Row | Tokens read | History | Share | Score |
|---|---|---|---|---|---|
| LongMemEval-S | Counterparts as shipped | 3,931 | 122,324 | 3.2% | 89.4% |
| LongMemEval-S | Counterparts, memory + transcript | 5,907 | 122,324 | 4.8% | 91.4% |
| LongMemEval-S | Honcho (their count) | 11,622 | 101,571 | 11.4% | 90.4% |
| LoCoMo | Counterparts as shipped | 4,067 | 20,643 | 19.7% | 87.8% |
| LoCoMo | Haiku 5.5 reading everything | 20,643 | 20,643 | 100.0% | 90.3% |
| LoCoMo | Counterparts, memory + transcript | 24,444 | 20,643 | 118.4% | 91.6% |
Reading these honestly
- All three benchmarks at these sizes fit in a modern model’s context window. On LoCoMo, Haiku 5.5 reading each whole conversation scores 90.3%, so memory + transcript adds about a point there, and memory only (87.8%) is below reading everything. Honcho makes the same point about Gemini 3 Pro on LongMemEval.
- What memory buys at this size is cost: an answer reads a few thousand tokens instead of the whole history. BEAM’s 1M and 10M sizes are where memory has to do all the work; we haven’t run those yet.
- Systems above us answer with larger models (gpt-5, gpt-5-mini, Gemini 3 Pro, GPT-5.4), score with their own judge prompts, or both. On LongMemEval, Mem0 and Mastra are ahead of us however it’s scored.
- A re-run moves a row by about a point. Gaps under a point or two are a tie.
- Mem0’s judge leans toward yes by design, accepts one correct item out of a list, and accepts dates within 14 days. We show it so our numbers can sit beside Mem0’s, not as a better measure.
- Models. Memories are written by Claude Haiku 5.5. Answers use the model named on each row: Haiku 5.5 at effort high, or Haiku 4.5 (4,096 thinking tokens with memory + transcript, none with memory only). All rows use the free local embedder.
- LoCoMo. Answer prompts tell the answerer to commit to its best inference and show the date before each question. Memories are written with the same writer settings as LongMemEval and BEAM (October 20); that alone moved memory only from 84.0% to 86.8%, and the recall title tail took it to 87.8%.
- BEAM 100K. We ported Honcho’s judge prompts and scoring (per conversation, the mean of its ability means); on Honcho’s own answers for one conversation it gave their score on 19 of 20 questions. Honcho’s run had dreaming on and answered the 40 instruction-following questions with Sonnet 4.5. Their messages went in undated; ours carry the date each batch has in the dataset, and each question shows the date. On our newest build (recall title tail) BEAM scored 66.7%, about 6 questions lower, within re-run noise.
- Honcho. Numbers are their published per-question runs (December 2025). Their page’s LoCoMo baseline (75.6%) counts 446 adversarial questions; on the same 1,540 it’s 83.1%.
- Mem0. Numbers are as published (README); their public result files show 93.4% and 91.6%, with gpt-5 answering and judging and 200 memories retrieved.
- Builds. The Haiku 4.5 rows and the “mean of 3” rows ran on the October 18 build, before the October 19 recall fixes. Token counts are from the October 20 build.
- Competitors. Rows come from each system’s own pages, papers and repos, linked in the table; we didn’t re-run them.
Last updated: October 20, 2026