The most common complaint about AI assistants in the workplace is that they don't remember. You explain your role, your project and your preferences, and the next day you explain them again. The obvious fix, keeping everything, creates different problems: slower and more expensive responses, answers based on stale information, and a growing store of personal data nobody designed for.
Memory is an architecture decision, not a feature toggle.
Why "keep everything in context" fails
Stuffing history into the prompt runs into the limits covered in long context or RAG: cost and latency grow with every token, and models use long inputs unevenly. Chroma's research found performance grows increasingly unreliable as input length grows, even on simple tasks4.
LongMemEval, a benchmark presented at ICLR 2025, tested exactly this for assistants. Its 500 questions probe five abilities: information extraction, multi-session reasoning, temporal reasoning, knowledge updates and abstention, embedded in long user-assistant histories. The authors reported that commercial chat assistants and long-context models showed about a 30% accuracy drop when recalling information across sustained interactions3.
Two of those abilities deserve attention. Knowledge updates: when a user changes their address or a project changes scope, memory must reflect the latest truth, not the first. Abstention: when the assistant doesn't know, it should say so rather than inventing a memory.
Ideas from research
Memory as an operating system. MemGPT proposed virtual context management, borrowing from operating system memory hierarchies: the model pages information between a small, fast working context and larger external storage, using function calls to read, write and manage its own context1. The authors showed it maintaining consistency over long conversations and analysing documents larger than the context window1.
Retrieval by relevance, recency and importance. The Generative Agents paper gave simulated characters a memory stream of experiences, retrieved by combining how relevant, how recent and how important each memory is, plus periodic reflection that synthesises memories into higher-level conclusions2.
Indexing, retrieval and reading. LongMemEval's authors framed memory design as stages of indexing, retrieval and reading, and proposed improvements such as splitting sessions into smaller units, expanding memory keys with extracted facts, and making queries time-aware3.
Good memory isn't storing more. It's storing the right things in the right shape, and retrieving them with time and relevance in mind.
The memory architecture I use
| Layer | Contents | Lifetime |
|---|---|---|
| Working context | The current task, recent turns, retrieved items | One request or session |
| Session summaries | What was discussed, decided and left open | Weeks to months |
| Profile and facts | Stable facts and preferences, each with source and timestamp | Until changed or deleted |
| Episodic store | Past interactions, indexed for retrieval | Defined retention period |
| Organisational knowledge | Documents and data the user may access, via RAG | Governed by the source system |
Practical rules that make it work:
- Extract facts deliberately, as structured records with a source and timestamp, using structured outputs, rather than keeping raw transcripts as the only memory.
- Resolve conflicts by time. Newer facts supersede older ones; keep history but retrieve the current value.
- Retrieve with time awareness, so "last week's decision" finds last week's session.
- Let the model abstain. If retrieval finds nothing reliable, the answer is "I don't have that", not a guess.
- Evaluate memory specifically, with test cases for recall, updates and abstention, added to your agent evaluation suite.
Memory is personal data
A memory store is a profile of a person. Treat it accordingly:
- Visibility and control. Users can see what is remembered, correct it and delete it.
- Scope. Memory is partitioned per user and per tenant, with the same isolation tests as any multi-tenant store, as in multi-tenant SaaS patterns.
- Sensitive categories. Don't store health, financial or other sensitive details without a clear basis and purpose, especially under regimes such as the UAE PDPL.
- Retention. Define how long each layer is kept and enforce it automatically.
- Injection risk. Content written into memory from untrusted sources can carry instructions into future sessions. Treat remembered content as data, with the controls in prompt injection defence.
Start small
Most products don't need the full stack on day one. A useful first version is session summaries plus a small, user-visible set of facts. Add episodic retrieval when evaluation shows users need it, and measure whether each layer improves outcomes enough to justify its cost and risk.
How I can help
I design agent architectures, including memory, retrieval and the governance around them, so assistants become more useful without becoming a liability. See my AI architecture work, or analyse your challenge and bring the result to a call.
- 1MemGPT: Towards LLMs as Operating SystemsPacker et al. (arXiv) · 2023
- 2Generative Agents: Interactive Simulacra of Human BehaviorPark et al. (arXiv) · 2023
- 3LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive MemoryWu et al. (arXiv; ICLR 2025) · 2024
- 4Context Rot: How Increasing Input Tokens Impacts LLM PerformanceHong, Troynikov and Huber, Chroma · 14 Jul 2025