A Memorilla review’s memory tests, with Counterparts added

Tests and the first four columns are from Andris Gauračs’s review for Better Stack (code and results); we added Counterparts. Thanks for publishing the tests and the code.

Every column in the review runs on Qwen3‑8B (Memorilla’s is a tuned one). Our first Counterparts column does too (4‑bit); the second uses Claude Haiku 5.5, which can search its memory. Haiku wrote the memories for both. Method, misses, where we trail →

At a glance

  • 7/8

    Counterparts on Qwen3-8B, reading memories Haiku 5.5 wrote. The review’s best: 5/8.

    More

    Ours ran at 4-bit, reading memories Claude Haiku 5.5 wrote. The review’s best column is full context, cut short to fit the model. Our whole prompt: ~9.6K tokens a question, against ~40.8K for that full context (its RAG: ~1.1K). Two of our seven pass the rubric loosely (walking stick, warning note), and another adds a made-up detail (it calls Garcia a former husband; Garcia is Beryl’s maiden name).

  • ~43K vs ~127K

    Average input tokens per Hound answer on Haiku, both 8/8: Counterparts vs the whole book.

    More

    Fewer tokens in total on the Hound: every Counterparts answer (27,773 to 66,304) read less than pasting in the whole book, and both scored 8/8 (no memory: 6/8). The memory itself was ~4.4K tokens against ~126K. Not counting writing the memories once (~239K new input tokens and ~87K out). On the Buried Clue the totals are about the same. Not a cost claim.

  • Elsewhere

    We tie or trail on the other tests.

    More

    Buried Clue: 1/4 on Qwen3-8B (the control only, none of the clues), tied with the review’s RAG; on Haiku 3/4, the same as full context by the rubric, though by meaning full context got all four. PersonalizationV4: 55.2% against 62.5% for full context, on Haiku.

QuestionNo memoryAndris’s run · Qwen3‑8BRAG top-5Andris’s run · Qwen3‑8BFull contextAndris’s run · Qwen3‑8BMemorillaAndris’s run · Qwen3‑8B, tunedCounter­partsours · Qwen3‑8BQwen3‑8B · 4‑bit · recall before answering, no searchCounter­partsours · Haiku 5.5can search its memory
In The Hound of the Baskervilles, who gave Dr. Mortimer the walking stick he left at Baker Street, according to its engraving?FAIL0.3sPASS0.4sFAIL4.6sFAIL2.0sPASS—PASS6.3s
What did Dr. Mortimer see on the ground near Sir Charles Baskerville's body that he did not mention at the inquest?FAIL0.1sFAIL0.4sFAIL4.1sFAIL1.9sPASS—PASS6.7s
How was the anonymous warning note that Sir Henry received at his London hotel made?FAIL0.3sFAIL0.5sFAIL4.2sFAIL1.9sPASS—PASS7.4s
Who is the escaped convict hiding on the moor, and how is he connected to the Barrymores?PASS0.4sPASS0.5sPASS4.3sFAIL2.0sPASS—PASS7.0s
What is the true relationship between Stapleton and the woman who lives with him as his sister?PASS0.4sPASS0.3sPASS4.3sFAIL1.9sPASS—PASS7.4s
What had been put on the hound to make it glow in the dark?FAIL0.2sPASS0.3sPASS4.1sPASS1.9sPASS—PASS6.3s
Who wrote the letter signed 'L. L.' that brought Sir Charles to the gate on the night he died, and who had persuaded her to write it?FAIL0.3sFAIL0.4sPASS4.1sFAIL2.0sPASS—PASS9.0s
Who is Stapleton really, and why did he want Sir Henry dead?FAIL0.3sFAIL0.6sPASS4.3sFAIL2.1sFAIL—PASS12.4s
Correct2/84/85/81/87/88/8
Avg memory (the review’s: prompt) tokens the LLM received721,14040,760789,4984,395
Avg total input721,14040,760789,57042,682

On Qwen3-8B, Counterparts scored 7/8, above each of the review’s four columns (its best: 5/8). Haiku with no memory already gets 6/8: it knows the book.

A ring marks the best score among columns on the same model. PASS/FAIL is the review’s keyword rubric, for every column.

How these were run

Claude Haiku 5.5 already gets 6/8 here with no memory at all: it knows the book. Haiku also wrote the memories the Qwen column read, so some of that knowledge may have come in through them. The review’s full context is cut from the front to fit Qwen3-8B’s window (about 41K tokens here), so it missed the early chapters.

Andris Gauračs’s runs: Qwen3-8B on an NVIDIA H100 PCIe (Memorilla: its tuned Qwen3-8B checkpoint), 2026-11-13. Counterparts on Qwen3-8B: the 4-bit build, run locally on an Apple M3 Pro laptop; the review’s RAG column re-run on that build matched its marks 12/12. It recalled memory once before answering (several lookups in one step) and didn’t search while answering; none of the review’s columns search either. Given a recall tool to search while answering instead, the same setup scored 5/8 on the Hound and 1/4 again on the Buried Clue. The one-shot setup’s seconds aren’t in the table because they were measured differently: generation only, recall time not included, about 32.3 s per answer here. Counterparts on Haiku: Claude Haiku 5.5, one run, 2026-11-17, a laptop calling the API through the Claude CLI. Memories were written by Haiku 5.5 for both Counterparts columns. Seconds are wall clock per answer on different hardware: don’t compare them across columns. Memory tokens: the review’s and our Qwen column are Qwen3 tokens (the review’s include the whole prompt); our Haiku columns count the memory or history handed to the model with Claude’s tokenizer. Total input adds the prompt and, for Haiku, the search loop.

Source

“Did Stanford Just Solve AI Memory? (Memorilla)”, Better Stack, November 15, 2026, by Andris Gauračs, Developer Advocate at Better Stack. Andris’s tests, code and results: github.com/andrisgauracs/memorilla-demos.

Memorilla is from Stanford: the project page, the paper (arXiv:2610.21375) and the code.

Details
  • The review’s model. Our first Counterparts column answers with Qwen3-8B, like the review’s three text baselines (its Memorilla column is a tuned Qwen3-8B): 7/8 on the Hound, above each of the review’s columns, with a prompt of about 9,570 tokens against its RAG’s 1,140. Two of the seven pass the review’s rubric loosely (walking stick, warning note), and another adds a made-up detail (Garcia as a former husband; it’s Beryl’s maiden name); its full context there is cut from the front to fit the model, so it missed the early chapters. And 1/4 on the Buried Clue (the control only), tied with the review’s RAG. It ran the 4-bit build on a laptop and recalled memory once before answering (several lookups in one step) and didn’t search while answering; none of the review’s columns search either. The memories it read were written by Haiku 5.5, and on the Buried Clue two of the three clues never reached that one shot; the third, the bakery shift, did, but Qwen didn’t use it.
  • Our Haiku column has its own controls. They use the same model as it. On the Buried Clue, Counterparts and full context both score 3/4; full context’s getaway FAIL is a strict text miss (“longer car”, not “long car”), so by meaning it got all four. On the Hound both score 8/8.
  • What Counterparts adds is a much smaller memory handed to the model, not better answers than full context. About 4.2K tokens of memory on the Buried Clue against ~44K for the whole history, and ~4.4K against ~126K on the Hound. With our prompt and the recall loop, on the Hound a Counterparts answer read 27,773 to 66,304 tokens in all, every one below full context’s ~127K; on the Buried Clue, 25,642 to 50,485 against ~45K, about the same (two answers read more). That isn’t a cost claim.
  • The Haiku column’s miss. The parents’ dinner question needed “vegan”. Counterparts had written that memory, but recall didn’t reach it.
  • Where we trail. On PersonalizationV4, the Memorilla paper’s own benchmark (424 questions, the only test with a real sample), Counterparts scores 55.2% against 62.5% for full context.
  • Small tests, one run each. They show behavior, not statistics. Counterparts isn’t faster here, and its columns are our benchmark setup, without the nightly pass the product runs.
  • Source: the video (Better Stack, November 15, 2026) and Andris Gauračs’s tests on GitHub. Everything else, including dataset credits, is on the full page.

The full results page →

Questions

What is Memorilla?

Memorilla is a memory module for language models from researchers at Stanford and partner institutions (arXiv:2610.21375). Each document is encoded once into a stored embedding; when a question comes in, a trained module reads those embeddings and hands the model a few memory tokens (16 in the paper’s supervised experiments) in place of the text. The model’s own weights stay fixed: only the memory module is trained, and the released checkpoints are for Qwen3-8B.

Sources: Memorilla project page, the paper (arXiv:2610.21375), the code

Does Memorilla work?

On the paper’s benchmarks it does well, for example 84.33% on FactKG against 80.43% with full context. In Andris Gauračs’s review for Better Stack, which used Stanford’s released checkpoints without retraining them, it got 1/8 on plot questions about The Hound of the Baskervilles and 0/4 on the Buried Clue, facts hidden in 150 chats. Full context got 5/8 and 0/4, RAG 4/8 and 1/4; on a 10-question PersonaMem-v2 sample every method was at or below chance. Its speed held up: at 512 passages (about 128K tokens) its answering step took 0.12 s, or 3.25 s counting embedding, against 19.33 s for full context.

Sources: the review on YouTube, Andris Gauračs’s findings, the paper (arXiv:2610.21375)

How is Memorilla different from RAG?

RAG finds passages of text and pastes them into the prompt, so the model reads the original words. Memorilla gives the model learned memory tokens built from one embedding per document, and the review’s diagnostic found that “one vector per chat keeps the gist, not details”. In the review RAG scored higher: 4/8 against 1/8 on the Hound, and 1/4 against 0/4 on the Buried Clue (RAG’s one pass there was the control question). Memorilla’s prompts were much smaller: about 78 tokens against RAG’s 1,140 on the Hound.

Sources: Andris Gauračs’s findings, his tests on GitHub

How is Memorilla different from full context?

Full context puts the whole history in the prompt, so the time to answer grows with the history; Memorilla’s input stays at a fixed number of memory tokens. The paper reports 27× lower latency at 512 documents (0.74 s against 19.9 s, with document encoding counted separately), and the review measured about 6× counting embedding. On the review’s Hound test full context scored 5/8 and Memorilla 1/8; that full context was cut from the front to fit Qwen3-8B’s window, so it missed the early chapters.

Sources: Memorilla project page, Andris Gauračs’s findings

How did Counterparts do on the same tests?

On the Hound, Counterparts answered 7/8 on Qwen3-8B, the review’s model (ours a 4-bit build); the review’s best column got 5/8 and Memorilla 1/8. With Claude Haiku 5.5 answering, it got 8/8 from about 4.4K tokens of memory against about 126K for the whole book, which also got 8/8 (in total input, about 43K against 127K). Haiku wrote the memories both columns read, and it already knows the book: with no memory it gets 6/8. Elsewhere we tie or trail: on the Buried Clue 1/4 on Qwen3-8B (the control only, level with the review’s RAG) and 3/4 on Haiku, level with full context by the rubric; on PersonalizationV4, the Memorilla paper’s own benchmark, 55.2% against 62.5% for full context, both on Haiku.

Sources: the full results page, his tests on GitHub

How were these tests run?

We ran the review’s two tests, with its questions and rubrics, once with Qwen3-8B answering (4-bit, on a laptop) and once with Claude Haiku 5.5, one run each; Haiku 5.5 wrote the memories for both, and the review’s four columns are its published results. PersonalizationV4 (424 questions from 3 of its 30 test users) ran on Haiku only. Every one of our answers, the misses and the setup are on the full results page.

Sources: the full results page, his tests on GitHub

Last updated: November 18, 2026