What’s the difference between AI memory, RAG and the context window?

Updated 4 November 2026

The context window is everything the model reads for one reply. It lasts for that one request, and the app rebuilds and resends it every turn. RAG (retrieval-augmented generation) is a way of choosing what goes into the window: search a store, then paste the best matches in.

Memory is a store that outlives the conversation, plus something that decides what to write into it. It reaches the model only by being loaded into the window or retrieved into it. So the three work at different levels, usually together: memory is what’s kept, retrieval is one way of fetching it, and the window is where it has to land.

The research adds two warnings. A bigger window doesn’t make everything in it equally usable, and a memory that keeps only short notes loses detail.

An AI drafted this page from the sources below and another AI checked it. It’s a research summary: the figures are the cited researchers’ and vendors’ own, and no person has run these tests here. Counterparts, near the end, is ours: Mike LaPeter makes it.

	Context window	RAG	Memory
What it is	All the text the model reads for one reply	A search step that puts the best-matching passages into the window	A store kept between conversations, and something that writes to it
How long it lasts	One request; chat apps resend the conversation each turn	As long as the index	Until it’s edited, deleted or fades
Who decides what’s in it	The app, up to a size limit	A ranking of how well each passage matches the question	Whoever writes it: you, the app, or the AI
How it goes wrong	Information in the middle of long input gets missed	Finds what resembles the question, which may not be what the answer needs	Saves the wrong thing, or loses detail when it condenses
Fits	One task whose material fits	A large, mostly fixed body of documents	Continuity: who you are, what was decided, where you left off

Is a bigger context window the same as memory?

No. The window holds nothing between requests: “The Messages API is stateless, which means that you always send the full conversational history to the API,” in Anthropic’s words (why Claude forgets between chats covers the Claude side). Claude Code’s docs put it plainly: “Each Claude Code session begins with a fresh context window.”

Makers do describe the window in memory terms. Anthropic calls it the model’s “working memory”, and Google writes “An analogy for the context window is short term memory.”

Both also warn about long windows. Anthropic: “more context isn’t automatically better. As token count grows, accuracy and recall degrade, a phenomenon known as context rot.”

Google’s long-context guide reports about 99% on finding one fact in a long input, and warns that with several facts to find, “the model does not perform with the same accuracy.” Independent studies agree:

Lost in the Middle (2023, GPT-3.5 era): accuracy “significantly degrades when models must access relevant information in the middle of long contexts.” At worst, with the answer in the middle of 20 or 30 documents, GPT-3.5-Turbo did worse than with no documents at all (56.1%).
NoLiMa (Adobe Research, 2025): when the question and the fact share few words, “At 32K, for instance, 11 models drop below 50% of their strong short-length baselines.” GPT-4o fell from 99.3% to 69.7% at 32K.
Context Rot (Chroma, July 2025, 18 models including Claude 4, GPT-4.1 and Gemini 2.5): on questions about a chat history, models did better given about 300 relevant tokens than the full 113,000. Chroma makes a retrieval database, which is worth knowing when reading it.

The older studies used older models, and the size of the effect varies by model and task. None of the vendor docs read for this page says it’s gone.

Context window sizes and prices, as of November 2026

From each maker’s API docs, read 4 November 2026; input price per million tokens, standard tier.

Model	Context window	Input price	Long prompts
Claude Opus 5.5	1M tokens	$4	Same rate across the window
Claude Sonnet 5.5	1M tokens	$2	Same rate across the window
Claude Haiku 5.5	1M tokens	$0.10	$0.50 for the whole prompt once it’s over 100,000 tokens
GPT-6 Astra	1,050,000 (922,000 input)	$10	Whole request at double the input rate (1.5x output) once input passes 272,000 tokens
GPT-6 Luna	1,050,000 (922,000 input)	$0.10	Whole request at double the input rate (1.5x output) once input passes 272,000 tokens
Gemini 4 Argon	1M, shared by input, thinking and output	$2, rising to $4 on 1 December	None listed

Resending a long history costs money every turn. At Sonnet 5.5’s rate, 500,000 tokens of history is about $1 of input per reply, or about 5 cents once it’s cached (cache reads cost 5% of the input price on that model). In the Claude apps on paid plans, with code execution on, “When your conversation approaches the context window limit, Claude summarizes earlier messages to make room for new content.”

Is AI memory just RAG?

Often part of it is. Most memory systems store notes and use retrieval to find the relevant ones later. The difference is the writer: document RAG indexes text that already exists, while a memory system saves new text (a fact about you, a decision) and often updates it and brings it back without being asked.

Retrieval doesn’t have to mean vector search, either. Of the 21 systems in our directory, the five chat assistants don’t say whether they search by meaning. Of the other 16, five don’t out of the box: Claude Code’s auto memory and Letta load files into the window and open others by name, MCP Memory and Hyperconsciousness match text, and Screenpipe searches by keywords.

Letta’s docs: “MemFS does not include a semantic or vector index by default.” That count is from the directory’s own descriptions, read 4 November 2026; the comparison has the details.

At the other end, qmd is pure retrieval over Markdown notes you already have: it “combines BM25 full-text search, vector semantic search, and LLM re-ranking.”

Condensing has a cost. In the LongMemEval benchmark (ICLR 2025), storing extracted summaries or facts in place of the original conversation lowered accuracy “due to information loss,” except on questions that span several sessions, where facts helped. Their best setup searched the original turns, indexed together with facts drawn from them.

Full history or retrieval: what the research measured

LongMemEval hides 500 questions in long chat histories. These runs date from 2024, with GPT-4o and the paper’s Chain-of-Note reading prompt in every row:

GPT-4o on LongMemEval	Accuracy
Given only the sessions that hold the answer	92.4%
Given the whole history, about 115,000 tokens	64.0%
Retrieving the top 10 turns, each indexed with facts drawn from it, from a longer history of about 1.5M tokens	72.0%

Retrieval beat reading everything even with a history about 13 times longer, and both fell short of being handed the right sessions. Plain retrieval, without the added facts, scored 67.0%. Chroma’s 2025 rerun on part of the same benchmark, with Claude 4, GPT-4.1 and Gemini 2.5, found the same direction: every model did better given only the relevant part. On identical 115,000-token histories, a smaller model showed a wider gap: Mistral-Nemo scored 28.6% reading everything and 66.6% with retrieval (indexed with the added facts).

A shipped memory feature lost to plain reading when the history was short. On histories of three to six sessions, short enough to read whole, ChatGPT’s memory (GPT-4o, August 2024) scored 57.7% against 91.8% for GPT-4o reading everything; it “tended to overwrite crucial information as the chat continues.”

Memory makers publish their own scores on these benchmarks, and they don’t always reproduce. Redis, which sells a memory layer itself and tested several systems under one protocol: “Some vendors publish higher scores than we report. We were not able to reproduce those numbers under this protocol.”

How do the chat apps combine the three?

Claude’s apps use all three. Memory is saved “as a set of individual topics as you chat.” On paid plans, searching past chats is retrieval (“These searches use Retrieval-Augmented Generation (RAG)”), and a project’s files sit in the window until they near its limit, when “Claude will automatically enable RAG mode.”

Gemini personalizes from “memory of your past Gemini chats” and, for US users on a Google AI plan, keeps a memory summary you can edit. ChatGPT also has a memory; we couldn’t read OpenAI’s current help page for this guide, so it isn’t described here.

When does each one fit?

One task whose material fits: the context window. Keep it lean and put the question last; Google says performance “will be better if you put your query / question at the end of the prompt.”
A large, mostly fixed set of documents: RAG, or a tool that switches to it for you, as Claude projects do on paid plans. How that differs from chat memory.
The exact words of old conversations: keep the transcripts and search them, as Claude’s chat search does on paid plans.
Continuity across conversations (who you are, how you work, what you decided): memory, since something has to decide what’s worth keeping and write it down. Counterparts, below, is one for Claude.
An agent’s records that change: a memory layer built for that, such as Zep or Mem0; the directory compares them.

None of the three gives perfect recall. Windows have a size, retrieval misses things, and a memory keeps only what was written down.

Where does Counterparts fit?

Counterparts is a memory for Claude, and it’s ours. At the end of a session Claude writes down what it learned in its own words: “Notes, not transcripts.” What matters sticks and the rest fades unless it comes up again.

It uses the other two as well. After the first few sessions, each one opens with a short briefing Claude wrote itself (who it is, who you are, where you left off), loaded into the window.

As you talk, related memories surface through search by keyword and by meaning, on a small model on your machine. That search finds “my doctor” in a note about a physician, but “a pet that purrs” won’t find a note about your cat.

It runs in Claude Code on macOS and Linux, and in Claude Desktop on macOS (the Quickstart has the Desktop steps); Windows isn’t supported yet. In Claude Code:

/plugin marketplace add mlapeter/counterparts
/plugin install counterparts@counterparts

Then restart Claude Code. It’s open source (MIT), with no API key or account, and the memory lives on your machine in one SQLite file; the Claude Code page has the rest. A hosted version for claude.ai is coming soon, in beta; sign up on the home page to hear when it opens.

FAQ

Is context the same as memory?

No. Context is what the model reads for the current reply, rebuilt each turn; memory is kept between conversations and has to be loaded or retrieved into the context to be used.

If the context window holds past messages, why is the model called stateless?

Because the model keeps nothing between requests. The app stores the conversation and sends all of it with each new message; Anthropic’s API docs say “you always send the full conversational history.”

Do I still need RAG with a 1M-token context window?

For a document set that fits, often not, and Google’s long-context guide suggests putting it all in. RAG still earns its place when the material is bigger than the window, when resending it every turn costs too much, or when the answer depends on several scattered facts. Claude’s projects on paid plans make the switch on their own.

Can an AI have unlimited memory, or an unlimited context window?

Not today. The largest windows in the Anthropic, OpenAI and Google docs read for this page are about a million tokens, and long windows are used less reliably than short ones. A memory holds only what was written down, and retrieval can miss what’s there.

Do I even need a memory product, or are plain files enough?

Plain files are a real memory design: Claude Code’s CLAUDE.md and auto memory are Markdown files loaded into the window, and Letta keeps agent memory as files. A memory product adds the upkeep: deciding what to save, updating what changed, letting the rest fade, and finding the right note later. If you’ll write and prune the file yourself, a file can be enough.

Why does vector search keep pulling the wrong things from my notes?

Search by meaning ranks passages by closeness to the question, which is different from whether they answer it or are still current. LongMemEval found that indexing each passage with facts extracted from it raised recall by about 9% relative on average (roughly 6 percentage points, for example from 58.2% to 64.4% of the right turns in the top 5). The Counterparts README (ours) adds that exact words match best, and a short memory matches better than one padded with names and details.

How does context compression work?

Near the window’s limit, Claude’s apps (on paid plans with code execution on) summarize earlier messages and carry the summary forward in their place; Claude Code calls this compaction. Detail can be lost: Claude Code’s docs say an instruction given only in conversation may be gone afterwards, while CLAUDE.md is re-read from disk. In Claude Code, Counterparts puts your memory back after a compaction: its start-of-session hook hands the briefing back, and what was said before the compaction is still written up.

Questions? Ask in r/AIMemorySystems.