Memory benchmarks · the tests from a Memorilla review
See it at a glance: one table, details on request →
Counterparts gives about the same answers as pasting the whole history into the prompt, from about a tenth of the tokens or less.
On November 15, Andris Gauračs, Developer Advocate at Better Stack, tested Stanford’s Memorilla in a video, “Did Stanford Just Solve AI Memory? (Memorilla)”, with memory tests Andris wrote and put on GitHub. We ran the same tests on Counterparts. On the review’s two small tests, Counterparts scored the same as pasting the whole history into the prompt: 3/4 on the Buried Clue by the review’s strict rubric (by meaning, full context got all four) and 8/8 on the Hound of the Baskervilles. We also ran PersonalizationV4, the Memorilla paper’s own benchmark and the one test here with a real sample size. There Counterparts trails: 55.2% against 62.5%.
Tests and the first four columns are from Andris Gauračs’s review for Better Stack (code and results); we added Counterparts. Thanks for publishing the tests and the code.
A different model. The review ran all four of its columns on Qwen3-8B. Our main columns use Claude Haiku 5.5, which is much stronger, so its scores and ours can’t be compared as a contest. That’s why all three of our columns use the same Haiku 5.5 answerer: what this page measures is what Counterparts adds on top of the same model, not whether we beat Memorilla.
On Qwen3-8B too. We then ran Counterparts with Qwen3-8B answering, the model behind the review’s three text baselines (its Memorilla column is a tuned Qwen3-8B), as the 4-bit build on a laptop, recalling memory once before answering (several lookups in one step) with no search while answering; Haiku 5.5 still wrote the memories, and Haiku knows the Hound. It scored 7/8 on the Hound, above each of the review’s columns (its full context there is cut from the front to fit Qwen3-8B, so it missed the early chapters), and 1/4 on the Buried Clue (the control only), tied with the review’s RAG column. The at-a-glance page has both Counterparts columns beside the review’s.
Which tokens. “A tenth of the tokens” counts the memory or history handed to the model. With our harness’s own answer prompt, the recall tool and each turn re-reading its context, on the Hound a Counterparts answer read 27,773 to 66,304 tokens in all, every one below full context’s ~127K; on the Buried Clue, 25,642 to 50,485 against ~45K, about the same (two answers read more).
At a glance
The Buried Clue · 4 questions
3/4
Counterparts, on ~4.2K tokens of memory (9% of the history)
Full context 3/4 on ~44K; no memory 0/4
The Hound of the Baskervilles · 8 questions
8/8
Counterparts, on ~4.4K tokens of memory (3% of the book)
Full context 8/8 on ~126K; no memory 6/8: Haiku knows the book
PersonalizationV4 · 424 questions
55.2%
Counterparts trails here, on ~8.0K tokens of memory
Full context 62.5% on ~83K; no memory 31.4%
The Buried Clue
The review’s first test is 150 chats from a made-up person, Sam (we fed them in order, one a day). Three questions each depend on a detail mentioned once, in passing, in one chat; the fourth is a control, a fact stated plainly. A question passes when the answer contains a word from each part of its rubric (a plain text match, no AI judge). We used its questions and rubrics unchanged.
| Question | Ours · Claude Haiku 5.5 | The review · Qwen3-8B | |||||
|---|---|---|---|---|---|---|---|
| No memory | Full context | Counterparts | No memory | RAG top-5 | Full context | Memorilla | |
| Plan me a weekend getaway from Toronto.Clue: Sam’s dog Biscuit gets carsick on long drives. | FAIL | FAIL | PASS | FAIL | FAIL | FAIL | FAIL |
| What should I cook for my parents this Sunday?Clue: Sam has been vegan for years. | FAIL | PASS | FAIL | FAIL | FAIL | FAIL | FAIL |
| When's the best time for me to go to the gym?Clue: Sam works a 3 a.m. bakery shift. | FAIL | PASS | PASS | FAIL | FAIL | FAIL | FAIL |
| What's my sister's name, and when is her birthday?Control: Sam’s sister Priya, birthday March 14, stated plainly. | FAIL | PASS | PASS | FAIL | PASS | FAIL | FAIL |
| Correct | 0/4 | 3/4 | 3/4 | 0/4 | 1/4 | 0/4 | 0/4 |
| Avg tokens the LLM receivedOurs: the memory or history handed to the model, Claude tokenizer. The review’s: the whole prompt with the question, Qwen3 tokenizer; the two tokenizers count differently, so compare ratios. | 0 | 44,240 | 4,160 | 49 | 800 | 27,103 | 67 |
| Avg total input per answerOurs only: everything the model read, with our answer prompt, the recall tool and each turn of its loop. | 1,142 | 45,382 | 37,234 | — | — | — | — |
Ours: answers claude-haiku-5-5 (effort high), memories written by claude-haiku-5-5, a small local embedder (potion); 150 chats in order, 1 per lived day from 2026-01-01; questions asked as of 2026/05/31 (Sun) 12:00; 510 memories in the store; one run, 2026-11-17, on an Apple M3 Pro laptop; models served by Anthropic via claude -p (subscription). Harness bench/memorilla 7aa676e0, Counterparts 883b2887. The review’s: Qwen3-8B (Memorilla-Qwen3-8B checkpoint for Memorilla; plain Qwen3-8B for the others), on an NVIDIA H100 PCIe; its results file results/clue-20261113-221214.json, 2026-11-13.
- Two of the three buried clues, and the control. Counterparts connected the carsick dog to the weekend trip and the 3 a.m. bakery shift to gym time, and gave Priya and March 14. In the video, Memorilla, answering with Qwen3-8B, scored 0/4.
- The miss. “What should I cook for my parents this Sunday?” needed Sam’s veganism. Counterparts had written the memory “User has been vegan for years”, but recall didn’t reach it: it found the memories about making soup for a family gathering instead, and the answer never says vegan.
- Full context’s getaway FAIL is a strict text miss. Its answer says Biscuit should get a break because “longer car rides have been hard for him”; the rubric looks for “long car”, and “longer car” doesn’t match. By meaning it got it, so full context is really 4 of 4 here, one ahead of Counterparts.
- No invented facts in these four answers. We checked every statement in our four Buried Clue answers that draws on memory, by hand, and each one traces back to a real chat. That claim covers these four answers only; we didn’t check the other tests this way.
I don't have your gym hours or a stated preference, so this is an inference from your work schedule.
The most relevant thing I have is from May 11: you work a 3 a.m. shift every weekday, you feel like a zombie by Thursday, and I advised keeping an early-evening bedtime on weekdays and taking a 20-30 minute nap after the shift. That points away from evening sessions, since they'd cut into the bedtime that protects your sleep. A late-morning or early-afternoon session, after your nap, would probably fit best. I don't know when your shift ends, so that timing is a guess. Also, I have no information on how busy your gym is at different hours.
If you tell me when your shift ends and what time you go to bed, I can narrow this down.
Used as designed, with its nights
| Clue → question | With nights | No nights |
|---|---|---|
| Vegan → “What should I cook for my parents?” | 3/3 | 0/3 |
| 3 a.m. shift → gym time | 3/3 | 3/3 |
| Control: sister’s name and birthday | 3/3 | 3/3 |
| Carsick dog → weekend getaway | 0/3 | 1/3 |
Nights took vegan from 0/3 to 3/3. The dog clue went from 1/3 to 0/3: the last night dropped it from the summary. On Haiku 5.5, Sam’s chats lived as 15 days with a night after each, 3 samples per question.
How this was run
Counterparts runs a pass each night that rewrites its summary of the user. We lived Sam’s 150 chats as 15 days of 10, with a real night after each day, and asked each question 3 times (fixed before the run). The comparison is the same chats with no nights.
Nights took vegan from 0/3 to 3/3. None of the review’s columns found any of the three clues; vegan is the one the nights added for us. Sam mentioned it on day 9. That night Counterparts put “They've been vegan for years.” on its summary of the user, and every dinner answer brought it up, for example: “Tell me how many are coming and whether they eat meat, since you've been vegan for years.” (Haiku with the whole history pasted in also passed this question.)
Plainly: the answerer here is Claude Haiku 5.5, not the review’s Qwen3-8B; with the same summary, Qwen3-8B (4-bit) didn’t use the vegan line. Ten chats a day compresses real use: vegan was a week old when the question came, not two months. Three samples per question show behavior, not statistics. And a loss: the last night’s rewrite of the summary dropped the carsick dog (it had been there through night 14). The getaway failed 0/3, but it also failed earlier in the run with the dog still on the summary, so the summary line alone didn’t carry it. We’re fixing both.
The Hound of the Baskervilles
The review’s second test gives the system the novel as memory (we fed it in order, a chapter a day) and asks 8 questions about the plot. For Haiku 5.5 this mostly tests what it learned in training: with no memory at all it already scored 6/8, because it knows the book. So the column to watch is no memory: it shows how little room was left for memory to help. Counterparts and full context both got the two that no memory missed.
| Question | Ours · Claude Haiku 5.5 | The review · Qwen3-8B | |||||
|---|---|---|---|---|---|---|---|
| No memory | Full context | Counterparts | No memory | RAG top-5 | Full context | Memorilla | |
| In The Hound of the Baskervilles, who gave Dr. Mortimer the walking stick he left at Baker Street, according to its engraving? | PASS | PASS | PASS | FAIL | PASS | FAIL | FAIL |
| What did Dr. Mortimer see on the ground near Sir Charles Baskerville's body that he did not mention at the inquest? | PASS | PASS | PASS | FAIL | FAIL | FAIL | FAIL |
| How was the anonymous warning note that Sir Henry received at his London hotel made? | PASS | PASS | PASS | FAIL | FAIL | FAIL | FAIL |
| Who is the escaped convict hiding on the moor, and how is he connected to the Barrymores? | PASS | PASS | PASS | PASS | PASS | PASS | FAIL |
| What is the true relationship between Stapleton and the woman who lives with him as his sister? | FAIL | PASS | PASS | PASS | PASS | PASS | FAIL |
| What had been put on the hound to make it glow in the dark? | FAIL | PASS | PASS | FAIL | PASS | PASS | PASS |
| Who wrote the letter signed 'L. L.' that brought Sir Charles to the gate on the night he died, and who had persuaded her to write it? | PASS | PASS | PASS | FAIL | FAIL | PASS | FAIL |
| Who is Stapleton really, and why did he want Sir Henry dead? | PASS | PASS | PASS | FAIL | FAIL | PASS | FAIL |
| Correct | 6/8 | 8/8 | 8/8 | 2/8 | 4/8 | 5/8 | 1/8 |
| Avg tokens the LLM receivedOurs: the memory or history handed to the model, Claude tokenizer. The review’s: the whole prompt with the question, Qwen3 tokenizer; the two tokenizers count differently, so compare ratios. | 0 | 125,896 | 4,395 | 72 | 1,140 | 40,760 | 78 |
| Avg total input per answerOurs only: everything the model read, with our answer prompt, the recall tool and each turn of its loop. | 1,159 | 127,055 | 42,682 | — | — | — | — |
Ours: answers claude-haiku-5-5 (effort high), memories written by claude-haiku-5-5, a small local embedder (potion); 364 passages as sessions, one lived day per chapter; questions asked as of 2026/01/16 (Fri) 12:00; 172 memories in the store; one run, 2026-11-17, on an Apple M3 Pro laptop; models served by Anthropic via claude -p (subscription). Harness bench/memorilla 7aa676e0, Counterparts 883b2887. The review’s: Qwen3-8B (Memorilla-Qwen3-8B checkpoint for Memorilla; plain Qwen3-8B for the others), on an NVIDIA H100 PCIe; its results file results/book-20261113-222648.json, 2026-11-13.
Where we trail: PersonalizationV4
The Memorilla paper’s own personalization benchmark: a made-up person’s 200 short chats with an assistant, then questions that put them in a new situation and ask what they’d most likely do. We ran 424 questions from 3 of its 30 test users, and it’s the only test here with a real sample size. Counterparts trails pasting the whole history in: 55.2% against 62.5%, and it trails for each of the three users. On the hard subset (59 questions that a small model without memory gets wrong, as the dataset marks them) the two are roughly tied.
| Questions | No memory | Full context | Counterparts |
|---|---|---|---|
| All 424 | 31.4% | 62.5% | 55.2% |
| Hard subset, 59 | 22.0% | 40.7% | 42.4% |
| User 135, 142 | 31.7% | 73.9% | 66.9% |
| User 150, 141 | 31.9% | 52.5% | 46.8% |
| User 155, 141 | 30.5% | 61.0% | 51.8% |
| Avg tokens the LLM received | 0 | 83,415 | 8,006 |
| Avg total input per answer | 1,255 | 84,671 | 57,253 |
- Scoring. The dataset’s own metric: the answer and the five candidate answers are embedded with Qwen3-Embedding-4B (we ran it locally), and the answer is correct when the nearest candidate is the right one. The model never sees the candidates. One call in each column was refused by the API and counts as a miss. Our no-memory answers mostly say they don’t know rather than guess, which this metric scores as wrong.
- The paper’s numbers are not comparable. The Memorilla paper reports Memorilla at 66.65% and Mem0 at 65.07% on this benchmark (its best, MemoRAG, at 67.88%), with Qwen3-8B answering and in its own setup. They are indicative only; don’t read them against our columns.
- Data. PersonalizationV4 is by the Memorilla authors (Fallahpour, Idehpour, Kothapalli, Mounier, Wu, Pardis and Leskovec, “Memorilla: Latent Semantic Memory for LLMs”, arXiv:2610.21375), licensed CC BY 4.0, with seed personas from Google’s Synthetic-Persona-Chat (Jandaghi et al., 2023, CC BY 4.0); its text was generated with OpenAI models. We used the questions and chats of users 135, 150, 155 unchanged, picked by a fixed seed (23, the one Memorilla uses for its splits) before any answer ran, and added one framing line to every column’s prompt (the question is about the user, by the name it uses) and “answer in two or three sentences”.
The review’s ten PersonaMem-v2 questions
The review also ran ten questions from PersonaMem-v2 (Jiang et al., 2025, CC BY 4.0), scored the same way as PersonalizationV4. Ours, with Haiku 5.5: no memory 1/10, full context 3/10, Counterparts 4/10. His, with Qwen3-8B: no memory 1/10, RAG top-5 1/10, full context 1/10, Memorilla 2/10. Ten questions are too few to read anything into; we show the scores so nothing is left out.
Source
“Did Stanford Just Solve AI Memory? (Memorilla)”, Better Stack, November 15, 2026, by Andris Gauračs, Developer Advocate at Better Stack. Andris’s tests, code and results: github.com/andrisgauracs/memorilla-demos.
Memorilla is from Stanford: the project page, the paper (arXiv:2610.21375) and the code.
Reading these honestly
- Different model. The review’s columns are Qwen3-8B; the three columns on this page are all Claude Haiku 5.5, a much stronger model. Compare them with each other, not with the review’s. For a run with the same answering model as the review (Qwen3-8B, 4-bit), see the at-a-glance page.
- One run each. Every cell is one sampled answer, with no repeats. Tests of 4 and 8 questions show behavior, not statistics.
- The Counterparts column is our benchmark setup: the recall tool plus the recall that runs on each prompt, without the nightly pass the product runs. Nothing like a night had happened in these stores.
- Not faster. A Counterparts answer took about 8.2 s on the Buried Clue and 7.8 s on the Hound, against 9.9 s and 5.9 s for full context (whose first Buried Clue answer paid a one-off cache write); on PersonalizationV4, 9.9 s against 6.6 s. All through the Claude CLI.
- Tokens. The “Avg tokens the LLM received” row counts only the memory or history handed to the model, by Claude’s tokenizer. Total input per answer, with our answer prompt and the recall loop, was 27,773 to 66,304 tokens for Counterparts on the Hound and 25,642 to 50,485 on the Buried Clue, where it is about the same as full context. The review’s counts use Qwen3’s tokenizer, so compare them as ratios, not raw numbers.
- Source. Andris Gauračs’s video for Better Stack, “Did Stanford Just Solve AI Memory? (Memorilla)” (November 15, 2026), and Andris’s tests at andrisgauracs/memorilla-demos. The review’s columns on this page are its own results files, as published. Memorilla is from Stanford.
- Same answer rules in every column. Our three columns share one answer prompt; only the access line changes (no memory: none; full context: the whole history in the prompt; Counterparts: its recall). Memories were written by Claude Haiku 5.5, one store per test, the chats fed in order, one lived day at a time.
- Written down first. Before any answer ran we wrote down what we expected on the Buried Clue: the control would pass, the vegan question was the likeliest of the three clues, and the gym question the least likely. That was half wrong: gym passed and vegan failed.
Last updated: November 17, 2026