AI Adoption

Lecture 3. Memory, context and search

The question we will answer today

Where does an LLM get the information
for its answer?

Today’s lecture

The memory mapfive kinds of memory in an LLM system
Parameterswhat is baked into the model weights
The contextworking memory for a single request
Search and RAGfinding external knowledge and putting it into context
Memory across sessionscarrying notes and facts from one chat to the next
How it fits togethertracing a full agent request end to end
Practicechoosing the right memory architecture

Recap: how an LLM works

1

The memory map

Five kinds of memory and one core rule

The memory of an LLM system

Five kinds of memory

Kind	What it stores	Lifetime	Written by	How it is read
Parameters	Knowledge from training	Until next model	Training	Used on every token
Context & KV cache	This request’s tokens	One request	App and model	Read by attention
Prompt cache	Computed prompt prefix	5–60 minutes	Server	Reused on matching prefix
Document index	Document chunks & vectors	Until rebuilt	Indexing job	Searched, copied in
Memory store	Notes from past chats	Months	Model, app, user	Loaded or searched

Only parameters and context compute the answer—everything else must be copied in.

Computer memory as an analogy

In a computer	In an LLM system	Role
CPU cache	KV cache, prompt cache	Avoids recomputing work that was just done
RAM	Context window	Fast working memory, but limited and cleared after use
Disk	Indexes, memory store, files	Large persistent storage that must be loaded into RAM to use
Firmware / ROM	Model parameters	Built-in knowledge and skills baked in at the factory

MemGPT (Packer et al., 2023) treats an LLM agent like an operating system: old text is paged out of the context to disk, and the model calls a function to page it back in.

Question 1

You ask an agent: “Why does test_sum fail?”
Where does each piece of information come from?

(a) What the assert keyword does in Python
(b) Your note from last week that tests run with pytest -x
(c) The source code of the failing test
(d) Claude Code’s built-in system instructions
(e) A breaking change in a library released last month

Answer. (a) Model parameters. (b) Memory file, loaded at session start. (c) Repository, read via a tool call. (d) Context window, reused from the prompt cache. (e) Nowhere—unless searched.

2

Parameters

What is baked into the model weights

What the parameters hold

Weights store a lossy compression of the training data before the cutoff date
Common facts stick (~2 bits/parameter at 1,000 exposures), rare ones fade (~1 bit at 100): the model always knows the capital of France, but often invents the author of an obscure paper
Facts carry no sources—the model cannot check where it learned something
Stored ≠ usable in reverse: GPT-4 names a celebrity’s parent 79% of the time, but names the celebrity from the parent only 33% of the time
Weights also store skills: grammar, coding, step-by-step reasoning, and calling tools

Allen-Zhu and Li, 2024; the reversal curse, Berglund et al., 2023.

Can we teach the weights new facts?

Pretraining takes weeks of offline compute and can only be done by the model creator
Fine-tuning is good at teaching style and format, but poor at teaching new facts
Fine-tuning on unfamiliar facts makes the model hallucinate more on other topics

Questions on events after cutoff	Accuracy
Base model (no documents)	0.48
Fine-tuned on the articles	0.50
Fine-tuned on paraphrased articles	0.59
Articles put in the context (RAG)	0.88

Use fine-tuning for skills and tone. Put new or private facts in the context.

Mistral 7B: Ovadia et al., Microsoft, 2024. Hallucinations from unfamiliar facts: Gekhman et al., Google, 2024.

When the context disagrees with the weights

Given a coherent document in the prompt, the model usually trusts it over its own weights
That holds even when the document is wrong: GPT-4o followed a false claim in the prompt 61% of the time, despite knowing the right answer
When retrieved snippets lack the answer, models tend to guess instead of saying “I don’t know” (Gemma 27B guessed wrong 66% of the time)
Models also play along with false premises: asked to prove altered, false olympiad theorems, GPT-5 invented a fake proof 29% of the time

Whatever is in the context steers the answer—for better or worse.

ClashEval, Stanford, 2024; Sufficient Context, Google, 2025; BrokenMath, ETH Zurich, 2025.

3

The context

Working memory for a single request

The KV cache

To generate each new token, attention looks back at all earlier tokens
Earlier tokens’ keys and values never change—compute them once and store them in GPU memory
Prefill: processes the entire prompt in one parallel pass to fill the cache, setting the wait before the first token
Decode: generates one token per pass, reading all weights and the whole cache—so an output token costs ~5× an input token

Per token: 2 × layers × KV heads × head size × bytes

Llama 3 70B uses 320 KiB per token—about 40 GiB for 128K tokens.

The model does not remember the chat

Every API call starts from scratch: once the model finishes its reply, the active KV cache is cleared
To continue a chat, the app must re-send the entire conversation every turn: system prompt, all earlier messages, and the new message
Server-side helpers (like OpenAI’s previous_response_id) only save network traffic—every earlier token is still processed and billed
So total input tokens grow quadratically with the number of turns: a 4K system prompt plus 20 turns of 1K tokens processes 276K input tokens
Prompt caching keeps the KV cache of the unchanged beginning on the server, making those re-read tokens cheap

OpenAI conversation-state guide (previous_response_id); see Question 3 for the 20-turn calculation.

Question 2

GPT-3.5 answers 56% of questions with no documents.
What happens if we give it 20 passages and put the answer in the 10th?

Exactly one of the 20 Wikipedia passages contains the answer.

Answer. Accuracy drops to 54%—worse than giving it no documents at all! (With only the single right passage, accuracy jumps to 88%.)

Accuracy is U-shaped: the start and the end are used best, while the middle of a long context is used much worse. Always put the best passages first.

Lost in the Middle, Liu et al., Stanford, 2023 (GPT-3.5-Turbo, June 2023; Natural Questions).

Long contexts are expensive and uneven

Memory and API cost grow linearly with prompt length—and are billed on every turn
Time to the first token grows quadratically, since every token attends to all earlier ones
1M-token windows overstate effective capacity: when a question shares no words with the fact, GPT-4o’s accuracy drops past 8K of its 128K window
Extra noise hurts accuracy: a focused 300-token input beat a full 113K-token chat history

A large window is a ceiling, not a target—send only what the task needs.

NoLiMa, Adobe Research, 2025; Context Rot, Chroma, 2025.

Prompt caching

If two requests start with the exact same tokens, their KV cache for that shared beginning is identical
The server keeps that precomputed KV cache in memory for 5 to 60 minutes
The next request with the same prefix skips recomputing it and processes only the new tokens at the end
Cached tokens cost 5–10% of the normal input price to read, with a small surcharge when written

Put static content first: tools, system instructions, and reference documents.

Anthropic: read 0.05×; write 1.25× (5 min) or 2× (1 hour). OpenAI: read 0.1×. October 2026.

Question 3

How much does prompt caching save on a 20-turn chat?

System prompt: 4,000 tokens. Each turn adds 300 user tokens and 700 reply tokens.
Claude Sonnet 5.5: $2 per million input tokens (cache write ×1.25, cache read ×0.05).

Answer. Without caching: 276K input tokens billed at full price = $0.55.

With caching: 23K new tokens written + 253K tokens read from cache = $0.084 (6.6× cheaper).

Keeping an agent’s context small

Technique	How it keeps the context small
Clear old tool results	Drops bulky file reads or search outputs once used
Compaction	Replaces the conversation history with a short summary
Sub-agents	Runs a subtask in a clean window and returns a brief report
Notes in files	Saves plans and state to disk (NOTES.md) to read back later

Context management moves details out of the window—at the risk of losing nuances.

Anthropic, 2025–2026: clearing old tool outputs saved 84% of tokens on a 100-turn task; Claude Code’s context per request grew 2.6× in six months.

Demonstration

Inspecting what is inside the context

code.claude.com/docs/en/commands platform.claude.com/docs/en/about-claude/models/overview

Claude Code in a copy of this lecture’s folder; login required

Run /context in a fresh session to see what is loaded before the first question
Ask the agent to read three large files, then check /context again
Run /compact to summarize the session and check /context once more
Compare the model table: context window vs. maximum output tokens

4

Search and RAG

Finding external knowledge and loading it into the context

Retrieval-augmented generation (RAG)

Split documents into chunks of a few hundred tokens and index them once
For a question, search the index for the few dozen best-matching chunks
Rerank those candidates with a cross-encoder and keep the top few
Paste those chunks and their sources into the prompt alongside the question
The model answers from the context and cites the sources

RAG does not change the model—it searches external storage and copies text into the prompt.

Term introduced by Lewis et al., 2020, combining “parametric” (weights) and “non-parametric” (index) memory.

Two ways to search for relevant chunks

Method	How it scores a chunk	What it is best at
BM25 (keyword)	Counts shared words; rare words carry more weight	Exact names, error codes, identifiers
Embeddings (dense)	Maps text to vectors and compares the cosine of the angle between them	Synonyms, paraphrases, meaning
Hybrid search	Runs both and merges the two rankings (e.g., RRF)	Both exact terms and paraphrases

Approximate vector indexes (such as HNSW) find the closest matches among millions of vectors in under a millisecond.

What actually improves a RAG pipeline

Pipeline	What it adds	Missed in top 20
Embeddings only	Vector search over raw document chunks	5.7%
+ Chunk context	A short note placing each chunk in its document	3.7%
+ BM25 (hybrid)	Combines exact keyword matches with vectors	2.9%
+ Reranking	Re-scores the top 150 candidates and keeps 20	1.9%

Under ~200K tokens? Skip RAG and put all the documents into a cached prompt.

Contextual Retrieval, Anthropic, 2024.

Demonstration

Searching over the slides of this lecture

Terminal in materials/retrieval-demo; LM Studio running EmbeddingGemma-300m for step 3

Shared words: python3 search.py "why is the middle of a long context used worse"
Paraphrase: "do chatbots overlook facts buried halfway through many pages" (BM25 misses completely)
Add --embed with EmbeddingGemma: vector search ranks the right slide first
Exact identifier: "previous_response_id" (BM25 wins by a wide margin)
Add --show-prompt to see the exact RAG prompt sent to the model
Agentic search in Claude Code: “Where does search.py compute BM25?”

Question 4

1,000 questions a day: cache the whole base, or use RAG?

Knowledge base: 200K or 800K tokens. RAG retrieves 20 chunks × 800 tokens (16K per question).
Claude Sonnet 5.5: $2 per million input tokens (cache write ×1.25, cache read ×0.05).

Answer. Uncached base: $400/day (200K) or $1,600/day (800K).
Cached base: $20.50/day (200K) or $82/day (800K). RAG: $32/day either way.

Below ~320K tokens, caching the entire knowledge base in the prompt is cheaper than RAG.

Choosing how to bring documents into context

Approach	Best when	Breaks when
All in context, cached	Small base (<300K tokens), many requests	Base outgrows the window or adds noise
RAG pipeline	Large collection, similar questions, low latency	Search misses a chunk, or the index is stale
Agentic search (grep, reads)	Code, changing files, multi-step questions	Multiple calls use ~4× more tokens

For code, live search beats vector indexes: Claude Code uses grep, and Cursor dropped its embedding index in 2026.

Boris Cherny, Latent Space, 2025; Cursor forum, July 2026; Anthropic, multi-agent research, 2025.

5

Memory across sessions

Carrying notes and facts from one chat to the next

Long-term memory is saved text put back into the context

Always loaded at startup

User profiles, project instructions (CLAUDE.md), or a short memory index
Guaranteed to be seen, but costs tokens on every request

Fetched on demand

Past chat transcripts or detailed notes on specific topics
Saves context space, but can be missed if the model does not search

Coding agents combine both: Claude Code always loads MEMORY.md (a short index) and reads topic files on demand; Codex loads memory_summary.md and searches MEMORY.md when needed.

Claude Code memory docs: loads the first 200 lines or 25 KB of MEMORY.md. Codex source: summary up to 2,500 tokens; memories off by default. October 2026.

Write, read, update, forget

Operation	How it works in practice
Write	Explicit requests (“Remember…”), background extraction, or agent notes
Read	Injected at session start, or retrieved by search when needed
Update	Overwrite the old note, or keep both entries with dates
Forget	Delete the memory entry—deleting the chat often leaves extracted facts behind

Updating old memories is harder than saving new ones.Keep critical project rules in one file you edit by hand (CLAUDE.md, AGENTS.md).

How chat apps implement memory

App	What it stores	How it enters the chat	How to make it forget
ChatGPT	Saved facts plus background summaries of past chats	Automatically injected into every chat	Delete the memory item itself (deleting the chat is not enough)
Claude	Topic notes (scoped per project) and past chats	Loaded per project, or searched via visible tool calls	Edit or delete the topic note
Gemini	No separate memory store—only your past chats	Pulled from past chats when relevant	Delete the chats that contain it

Help pages of OpenAI, Anthropic and Google, October 2026. In the EEA and the UK, ChatGPT’s chat-history memory is opt-in.

What can go wrong with persistent memory

Prompt injection becomes permanent: a webpage or email can make the assistant save a hidden instruction that acts in every future chat
Ad injection: “Summarize with AI” links on websites ask the assistant to remember a company as a trusted source
Outdated facts linger: last year’s job or an old library version keeps getting applied to new questions
More sycophancy: a saved user profile makes the model even more likely to agree with your views
Selective forgetting barely works: in a 2026 benchmark, no system tested passed more than 1.5% of forget requests

Rehberger, 2024; Microsoft, 2026; Trojan Hippo, 2026; CHI 2026; forgetting benchmark, October 2026 (at most 1.5% of requests).

Demonstration

Writing and reading back a memory

code.claude.com/docs/en/memory claude.ai

Claude Code in a scratch git repository; logins required

Tell the agent: “Remember: tests run with pytest -x.”
Inspect MEMORY.md and the topic file it points to
Start a new session and ask: “How do I run the tests?”
Run /memory and /context to see what was loaded
In claude.ai: inspect Settings → Memory and an incognito chat

6

How it fits together

Tracing one request, resolving conflicts, and key takeaways

One agent request, step by step

#	What happens	Memory involved
1	Session starts: prompt, tools, and MEMORY.md load	Memory store → context
2	Question sent; the unchanged prefix is reused	Prompt cache → KV cache
3	The model decides to run grep test_sum	Parameters + context
4	The app runs the tool and appends the output	Repository → context
5	The model generates the answer token by token	Parameters + context
6	“Remember: run pytest -x” is saved to a file	Context → memory store
7	Window nears its limit: compaction summarizes the chat	Context → summary

Every arrow is a copy step—and every copy can drop a detail or let bad data in.

When memories disagree

Conflict	What happens	What helps
Document vs. weights	Model trusts a wrong document	Show citations; flag contradictions
Old memory vs. today	Outdated notes override current setup	Store dates; prefer the current chat
Retrieved text vs. prompt	Prompt injection hijacks the agent	Treat as data; limit tool permissions
Summary vs. original	Compaction drops a needed detail	Keep key facts in files; re-read source
Index vs. live files	Search index lags behind edits	Re-index, or search files directly

Models are trained to rank system and user instructions above tool outputs (the instruction hierarchy, OpenAI, 2024), but do not follow this reliably.

Summary: what to remember

Kind	What it gives you	When it fails
Parameters	Skills and common facts from training (no sources)	A fact is rare, private, or new
Context	Working memory for this request, paid per token	It gets too long or buries facts
Prompt cache	Reuses an unchanged prefix 10–20× cheaper	The start of the prompt changes
Document index	Finds chunks in large collections to copy in	A chunk is missed, or index is stale
Memory store	Carries notes and preferences across sessions	Notes go stale, bias, or get poisoned

Answer = parameters + context. The model remembers nothing between requests—everything else is copied in, and every copy can lose or add something.

7

Practice: choose the architecture

5–10 minutes to design the memory setup

Question P

Choose the memory architecture

1. Course assistant

Answers 300 student questions a day about a 600-page textbook (~400K tokens). Must cite page numbers.

2. Codebase helper

Helps developers navigate and debug a 2-million-line repo (~20M tokens) that changes every day.

3. Research assistant

Works with one mathematician across months of sessions. Tracks notation, open problems, and failed attempts.

For each system, decide:

Which kinds of memory it uses beyond parameters and context
What is always loaded into the prompt, and what is fetched by search
What gets written back after a session, and where

Reference answers

Task	Memory used	Always loaded	Retrieved or searched	Written back
1. Course	Prompt cache, document index	System instructions (cached)	20 chunks with page numbers, or the whole book cached	Nothing
2. Codebase	Prompt cache, memory file, repository	Prompt, tools, project memory file	Live grep and file reads (code changes daily)	Memory file; summary on compaction
3. Research	Memory store, past sessions	Short notes file (notation, open problems)	Search over past sessions and papers	Research notes with dates

AI Adoption

Where does an LLM get the information for its answer?

Today’s lecture

Recap: how an LLM works

The memory map

The memory of an LLM system

Five kinds of memory

Computer memory as an analogy

You ask an agent: “Why does test_sum fail?” Where does each piece of information come from?

Parameters

What the parameters hold

Can we teach the weights new facts?

When the context disagrees with the weights

The context

The KV cache

The model does not remember the chat

GPT-3.5 answers 56% of questions with no documents. What happens if we give it 20 passages and put the answer in the 10th?

Long contexts are expensive and uneven

Prompt caching

How much does prompt caching save on a 20-turn chat?

Keeping an agent’s context small

Inspecting what is inside the context

Search and RAG

Retrieval-augmented generation (RAG)

Two ways to search for relevant chunks

What actually improves a RAG pipeline

Searching over the slides of this lecture

1,000 questions a day: cache the whole base, or use RAG?

Choosing how to bring documents into context

Memory across sessions

Long-term memory is saved text put back into the context

Always loaded at startup

Fetched on demand

Write, read, update, forget

How chat apps implement memory

What can go wrong with persistent memory

Writing and reading back a memory

How it fits together

One agent request, step by step

When memories disagree

Summary: what to remember

Practice: choose the architecture

Choose the memory architecture

1. Course assistant

2. Codebase helper

3. Research assistant

Reference answers

Where does an LLM get the information
for its answer?

You ask an agent: “Why does test_sum fail?”
Where does each piece of information come from?

GPT-3.5 answers 56% of questions with no documents.
What happens if we give it 20 passages and put the answer in the 10th?