A Practical Guide to AI Agent Memory: Building It, Using It, Keeping It Honest

An agent with memory is an agent that re-reads its own notes before it answers. That is the entire mechanism. There is no persistent internal state inside the model, no accumulated understanding between calls, nothing that carries over on its own. Every request starts from zero, and whatever the agent “remembers” is text that your code selected, formatted, and pasted back into the prompt a few milliseconds earlier.

Once you hold that picture, most of the confusing parts stop being confusing. Memory becomes an engineering problem with familiar shapes: what to write down, where to keep it, how to find the right piece again, what to do when it goes stale, and who is liable when it holds something it should not. It stops being a question about intelligence and becomes a question about storage, retrieval, and cost — three things engineering teams already know how to reason about.

This guide covers the whole arc: what memory is at the API level, the four kinds worth distinguishing, why bigger context windows did not remove the need for any of it, what replay actually costs, where each kind of memory belongs, and the maintenance half that most guides skip entirely. The n8n guide that prompted this piece is a good map of the conceptual territory and lands the important practical point that production failures come from the implementation layer rather than the conceptual one. It stops short of the implementation layer itself, which is where the interesting problems live.

Memory is text you re-send

The API is stateless. You send a list of messages, you get one message back, and the server keeps nothing. When a chat interface appears to remember your name from twenty turns ago, what happened is that the client re-sent all twenty turns. The model read your name again, the way it read it the first time.

This is worth stating plainly because it makes the constraints obvious. If memory is text in the prompt, then memory competes for the same finite space as the system prompt, the tool definitions, the retrieved documents, and the actual question. It costs input tokens on every single call, not once. It can be wrong in exactly the way a text file can be wrong. And it can be written by anyone who can influence what gets stored — a point we come back to later, because it is the part with teeth.

Anthropic’s memory tool makes the mechanism explicit rather than hiding it: the model reads and writes files in a /memories directory, and you implement the storage backend. The model issues view, create, str_replace, insert, delete, and rename commands. Your code executes them against whatever you point it at. That is the honest shape of agent memory, and every framework that looks more magical is doing some version of the same thing with more abstraction on top.

So a working definition, useful for the rest of this piece:

Agent memory is the set of decisions your system makes about which prior text to re-insert into the next model call.

Two decisions, really. A write decision: what is worth keeping. A read decision: what is worth re-inserting right now. Almost every design question below is one of those two wearing a different hat.

What an agent without memory actually loses

Statelessness is easy to describe and easy to underestimate. It helps to be specific about the failure modes, because they show up in different places and get misattributed to model quality.

It re-asks. A support agent that has already collected an account ID asks for it again after a tool call, because the tool call ended one request and started another. Users read this as the agent not listening.

It contradicts itself. In turn 3 the agent recommends Postgres. In turn 30, having lost turn 3 to a truncation window, it recommends DynamoDB with equal confidence and no acknowledgement that anything changed. Nothing is malfunctioning; the earlier recommendation simply is not in the prompt anymore.

It re-derives. An agent working through a codebase reads the same three files on every task because it has no record that it read them yesterday, or what it concluded. You pay for that reading every time.

It cannot personalise. Preferences, house style, the fact that this customer is on the enterprise plan and that one churned last quarter — all of it has to arrive in the prompt or it does not exist.

It cannot improve. This is the one that matters most over a long horizon. An agent that gets corrected and cannot store the correction will make the same mistake in the next session, and the session after that. Correction without persistence is not learning; it is just a slower way of getting the same answer.

The last two are why memory is not a nice-to-have for anything running longer than a single conversation. An agent doing multi-day work without memory is not a junior engineer who needs supervision. It is a junior engineer with total amnesia every morning, which is a categorically different staffing problem.

The four kinds, and which ones you actually need

The taxonomy borrowed from cognitive science — working, semantic, episodic, procedural — gets repeated a lot, and it is genuinely useful, but only if you translate each one into a storage and lifecycle decision. Otherwise it is vocabulary.

Working memory is the current conversation, held in the context window. Lifecycle: dies when the session ends. Storage: none, it is already in the prompt. The only design decisions are how much of it to keep and what to do when it exceeds the budget.

Semantic memory is facts the agent should know regardless of when it learned them. Company policies, product documentation, the customer’s plan tier, the fact that this repository uses pnpm. Lifecycle: long, with occasional invalidation. Storage: usually a vector store for the unstructured parts, and — this gets missed — a plain database table for anything with a schema. plan_tier = enterprise is a column, not an embedding.

Episodic memory is what happened, and when. Past sessions, decisions taken, what the agent tried and what came of it. Lifecycle: append-only, growing forever unless you prune it. Storage: chronological, with a time index, because “what did we decide about auth last month” is a temporal query and vector similarity handles time badly.

Procedural memory is how to do things here. The steps for a deployment, the correct sequence for onboarding a customer, the escalation path. Lifecycle: slow-changing, edited deliberately rather than accumulated. Storage: honestly, files. Version-controlled markdown that a human reviews. Procedural memory is the one type where automatic capture is usually the wrong call, because a procedure the agent inferred from one lucky run is a procedure you will be debugging in three weeks.

The practical read: most teams need working memory (unavoidable) and semantic memory (high value, moderate effort) on day one. Episodic memory earns its place once sessions matter to each other. Procedural memory is often best served by a prompt file and a code review, not by a memory system at all.

The bigger-window detour

Around the time context windows crossed a million tokens, a reasonable-sounding argument appeared: memory is a workaround for small windows, so make the window large enough and the problem dissolves. Just put everything in.

It does not hold, for two independent reasons.

The first is positional. Liu et al. found that model performance “is often highest when relevant information occurs at the beginning or end of the input context, and significantly degrades when models must access relevant information in the middle of long contexts, even for explicitly long-context models” (Lost in the Middle). Retrieval accuracy is not uniform across the window. Bury a fact in the middle of a long transcript and the model is measurably worse at using it than if you had put the same fact at the top.

The second is more damaging to the “just put everything in” position, because it kicks in far below the limit. Chroma’s context rot report evaluated 18 models — including GPT-4.1, Claude 4, Gemini 2.5 and Qwen3 — across eight input lengths, and found that models do not use their context uniformly: performance degrades at every length increment, not just near the ceiling. A model with a million-token window still shows degradation at fifty thousand tokens. The report also finds that needle-question similarity, the presence of distractors, and the structure of the surrounding text all move the curve independently of raw length.

Read those together and the conclusion is not “long context is bad.” Long context is excellent, and it makes a whole class of workaround unnecessary. The conclusion is narrower: window size is capacity, not recall. A larger window raises the ceiling on how much you can include. It does not make everything you include equally usable, and it does not remove the selection problem. You still have to decide what goes in, and now the decision is about signal-to-noise rather than about fitting.

Which is what a memory system is for. Selection, ranked by relevance, small enough to sit near the front of the prompt.

What memory costs

Memory has a price that is easy to miss during a prototype and impossible to miss on a monthly invoice, because the naive implementation scales quadratically.

The arithmetic is simple. If every turn adds t tokens to the conversation, and you resend the entire history on each call, then cumulative input across n turns is t·n(n+1)/2. Twenty turns of a thousand tokens each is not twenty thousand input tokens. It is 210,000. Fifty turns is 1.27 million. The conversation looks linear to the user and bills quadratically to you.

Put current prices against that. Claude Opus 5 runs $5 per million input tokens, Sonnet 5 runs $3 (with a $2 introductory rate through the end of August 2026), and Haiku 4.5 runs $1. A fifty-turn support conversation at 1,000 tokens per turn is about $6.35 of input on Opus, $3.81 on Sonnet, against a naive $0.25 if you had somehow only paid for each token once. The multiplier is 25×, and it grows with conversation length.

Three levers change this, in ascending order of effort.

Prompt caching is the first and the largest. Cache reads cost roughly 0.1× the base input price — a 90% discount on the cached span. Cache writes cost 1.25× at the five-minute TTL and 2× at the one-hour TTL, so break-even is two requests on the short TTL and three on the long one. Anything past that is close to free. The mechanics matter more than the discount, though, because caching is a prefix match: any byte change anywhere in the prefix invalidates everything after it. A timestamp interpolated into the system prompt, a non-deterministic JSON key order, a tool list assembled per-user — each of those quietly turns a 90% discount into a 25% surcharge, with no error to tell you. The minimum cacheable prefix is model-dependent (512 tokens on Opus 5, 1,024 on Sonnet 5), and a shorter prefix silently fails to cache rather than warning you.

The design rule that follows: order your prompt by volatility. Frozen content first (system prompt, tool definitions), then session-stable content, then per-turn content. Never interpolate the current time into the system prompt. Check usage.cache_read_input_tokens in production and treat a persistent zero as an incident, not a curiosity.

Truncation is the second lever and the crudest. Keep the last N turns, drop the rest. It is one line of code and it works fine for short conversational agents. It fails silently for anything where turn 3 mattered.

Summarization is the third, and it is where memory design proper begins: replace the dropped turns with a compressed record instead of deleting them. Which raises the question the rest of this guide is about — compressed by what rule, stored where, and retrieved how.

Worth knowing before you build any of this: parts of it now ship inside the model APIs. Anthropic’s API includes context editing (clearing stale tool results and thinking blocks from the transcript), compaction (server-side summarization of earlier turns as you approach the window), and the memory tool mentioned above. These are three distinct mechanisms with different semantics — editing prunes, compaction summarizes, the memory tool persists across sessions — and they cover a meaningful share of what teams used to hand-roll. Check what your provider gives you before writing a summarizer.

Choosing where to put it

The default answer in most agent tutorials is a vector database. It is a reasonable default for one of the four memory types and a poor default for the other three. Match the store to the query you will actually run against it.

The failure worth naming: putting structured facts in a vector store. If the agent needs to know a customer’s plan tier and you embed the sentence “Acme is on the enterprise plan,” retrieval will usually find it and occasionally will not, and when it does not, the agent confidently answers with whatever adjacent chunk ranked highest. A SELECT returns the right answer or an error. Reserve similarity search for questions that are genuinely fuzzy.

On the specific tools, briefly and without ceremony. pgvector puts embeddings in the Postgres you already run, which is the right first move for most teams — one database, one backup story, transactional consistency between the structured and unstructured halves. Pinecone, Weaviate, Qdrant and Chroma are dedicated vector stores that become worth the extra operational surface at scale or when you need filtering and hybrid search that outgrows pgvector. Neo4j is the graph option when relationships between entities are the thing you query.

Above the storage layer sit three approaches worth understanding, because they represent genuinely different bets.

Paging. MemGPT (now shipped as Letta) treats the context window as RAM and external storage as disk, with the model moving data between tiers through function calls — “virtual context management,” modelled explicitly on operating system memory hierarchies. The appeal is that the agent manages its own memory rather than your retrieval heuristics doing it. The cost is extra model calls spent on memory management instead of the task.

Temporal graphs. Zep builds a temporally-aware knowledge graph (Graphiti) that tracks not just facts but when they were true, which is the right shape for “what did we believe about this account in March.” The published figures — 94.8% against MemGPT’s 93.4% on Deep Memory Retrieval, and up to 18.5% accuracy improvement with 90% lower latency on LongMemEval — are vendor-authored benchmarks from the team that built the system, and should be read as such.

Extraction. Mem0 runs a model over the conversation to pull out durable facts and store them as discrete memories rather than keeping raw transcripts. Reported results are a 26% relative improvement on an LLM-as-judge metric over OpenAI’s memory on the LOCOMO benchmark, 91% lower p95 latency and over 90% token savings against a full-context baseline. Same caveat: vendor-authored, and independent replications have reported lower numbers.

None of these are wrong. All three are also more machinery than a team needs in week one. The honest sequencing is: prompt caching, then a summarizer, then a semantic store, then consider whether one of the above solves a problem you have actually hit.

Deciding what is worth remembering

The write path is where memory systems succeed or quietly rot, and it gets far less attention than retrieval because retrieval is the part that looks like engineering.

Three strategies, from cheapest to best.

Store everything. Append every turn to a log, embed it, move on. Cheap to build, and it degrades predictably: the store fills with pleasantries, half-formed reasoning, and facts that were true for ninety seconds. Retrieval quality falls as volume rises, because you have increased the number of plausible-but-irrelevant neighbours around every query.

Store on a trigger. Write when something specific happens — a decision is reached, a tool returns an authoritative value, the user corrects the agent, a task completes. This is usually the right first implementation. It is deterministic, debuggable, and the trigger list doubles as documentation of what your system thinks matters.

Extract with a model. After each session, run a second model pass whose only job is to answer: what from this conversation will still be true and still be useful next week? Output a small set of atomic facts with timestamps. This produces the highest-quality store and adds a per-session cost plus a new failure mode, since the extractor can hallucinate a memory that was never stated.

Whichever you pick, four properties are worth enforcing on every stored memory:

• Atomic. One fact per record. “Acme is on enterprise and prefers email and churned in Q3” is three memories that will need updating on different days.
• Timestamped. Both when it was recorded and, where applicable, when it stops being true. Without this you cannot resolve contradictions later.
• Attributed. Which session, which user, which tool produced it. This is what makes deletion and audit possible, and you will need both.
• Scoped. Which user or tenant it belongs to, enforced at the query layer rather than by convention. A memory store without a tenant boundary is a cross-tenant leak waiting for its first ambiguous query.

The other half of the write path is deciding what not to store, and the strongest rule is simple: do not store secrets in memory. Anthropic’s own guidance on memory stores puts it bluntly — memories persist across sessions and are replayed verbatim into future contexts, so an API key written once is re-inserted into every later session that reads that store. Run a secret scanner on the write path, not just on your repository.

Getting the right three facts back

Retrieval is a ranking problem, and the mistake most teams make is to treat similarity score as the answer rather than as a candidate generator.

A retrieval path that works in production usually has four stages. Candidate generation pulls a generous set — twenty to fifty — using vector similarity, keyword search, or both. Hybrid retrieval, combining dense embeddings with BM25 keyword matching, consistently beats either alone, because embeddings handle paraphrase and keywords handle exact identifiers, and your queries contain both. Filtering applies the hard constraints that must never be approximate: tenant, user, time range, memory type. This runs as a WHERE clause, not as a similarity threshold. Re-ranking cuts the candidates to the three to five you will actually send, ideally with a cross-encoder or a small model that reads query and candidate together. Formatting decides how they appear in the prompt.

That last stage is undersold. Given the positional findings above, where you put retrieved memories in the prompt affects whether the model uses them. Near the top, clearly delimited, with their timestamps visible so the model can reason about recency. Not appended silently to the end of a long transcript.

Two failure modes to watch. Over-retrieval: sending fifteen memories because the budget allows it, which adds distractors and, per the context rot findings, measurably degrades performance. Three good memories beat fifteen mediocre ones. Retrieval without provenance: injecting a stored fact as though it were ground truth. Prefer a form the model can reason about — “recorded 2026-03-14: the customer said they were evaluating competitors” — over a bare assertion. It lets the model weigh a six-month-old claim differently from yesterday’s.

The half nobody budgets for

Everything up to here is build work, and it is the part that gets estimated. Maintenance is the part that gets discovered. Four problems, all of which arrive on a delay of weeks to months, which is precisely why they are missing from most guides.

Staleness. Facts expire. The customer’s plan tier changes, the deployment procedure gets rewritten, the person who owned that service left. Nothing in the system notices. The agent keeps citing the stored version with full confidence, because a memory record carries no signal that reality moved on. Mitigation: store a source and a recorded-at date on every memory, set explicit TTLs on categories that you know decay (pricing, personnel, config), and re-verify on read for the high-stakes ones rather than trusting the store.

Contradiction. The user said Postgres in March and DynamoDB in June. Both are in the store, both retrieve for the same query, and the model gets to pick. This is the failure that looks most like the model being unreliable when it is actually the memory layer being undecided. Mitigation: resolve on write, not on read. When you store a fact that conflicts with an existing one on the same subject, supersede rather than append — mark the old record inactive, keep it for audit, return only the current one. This is exactly the problem temporal knowledge graphs are built to model, and it is the strongest argument for that extra machinery.

Bloat. The store grows monotonically. Retrieval gets slower, more distractors compete with every query, and the useful signal thins out. There is no published threshold for where this becomes a real problem — treat it as a curve rather than a cliff. Mitigation: track whether each memory is ever retrieved, and prune the ones that never are. A memory that has not been read in ninety days is a candidate for archival, and archival is not deletion — move it somewhere cold and keep the audit trail.

Poisoning. One wrong fact enters the store and is repeated forever, with the agent’s full confidence behind it. This is the most damaging of the four because the failure is invisible: the agent is not malfunctioning, it is faithfully reporting what it was told. Mitigation: provenance on every record, a way for humans to see and correct what the agent believes, and a bias toward supersession over silent overwrite so that a bad correction can be traced.

None of these are hard problems individually. All four are ongoing work, and none of them appear during a demo. Budget for a person to own the memory store the way someone owns a database, because functionally that is what it is.

Memory is an attack surface

Poisoning has a deliberate cousin, and it is better documented than most teams realise.

Dong et al. demonstrated a practical memory injection attack — MINJA — in which an attacker with nothing more than ordinary query access plants records in an agent’s memory bank that later steer its reasoning on other users’ unrelated queries. No privileged access to the store, no compromised credential, no injected prompt at inference time. Just normal interactions crafted so that what gets written down is malicious.

The reported numbers are worth sitting with. Averaged across the tested configurations, the injection succeeded 98.2% of the time and the resulting attack succeeded 76.8% of the time. The systems tested were not toys: an agent over the MIMIC-III and eICU clinical datasets, a retrieval-augmented planning agent over Webshop, and a QA agent over MMLU. Per-configuration attack success ranged from 57.0% to 98.9%.

The structural point is that memory converts a transient prompt injection into a persistent one. A prompt injection ordinarily lives for one request; you handle it, the request ends, it is gone. Write it into memory and it becomes a stored instruction that gets re-inserted into future prompts — potentially other users’ prompts, if the store is shared. The write path is therefore a trust boundary, and most implementations treat it as a convenience layer.

Practical controls, none of which are exotic:

• Never write a memory scoped to one user into a store readable by another. Tenant isolation belongs in the query layer, enforced by the code, not by careful prompt writing.
• Treat model-extracted memories as untrusted input. If a model decides what to store, the thing deciding can be manipulated by the thing it is reading. Validate the shape, scan for instruction-like content, and keep the extractor’s output separate from human-authored procedural memory.
• Make memory writes visible. Log every write with its source, and give someone a way to read the store in plain language. An agent’s beliefs should be inspectable.
• Keep procedural memory human-reviewed. The memory type that most directly controls agent behaviour is the one you should least allow the agent to write unsupervised.

Deletion, retention, and the request you will get

At some point someone will ask you to delete a person’s data, and the memory store will be the awkward part of that conversation.

Deleting the source row is straightforward. Deleting every derived copy is not: the fact may also sit in an embedding in a vector index, in a summary generated three sessions ago, in an episodic record of a conversation that referenced it, and in a cached prompt prefix that is still warm. A deletion pipeline that only touches the primary store leaves the agent able to recall the deleted fact, which is worse than not deleting it, because now you have told someone it is gone.

What makes this tractable is design, not tooling. The attribution property from the write path — which user, which session, which tool — is what turns “delete everything about this person” from a search problem into a query. Without it, you are doing similarity search over your own store hoping to find all the copies, which is not a defensible answer to a regulator or a customer.

Three things to build in before you need them: attribution on every record so deletion is a WHERE clause; explicit retention policies per memory type, since episodic memory and procedural memory have genuinely different lifespans; and a documented purge path that covers derived artefacts — summaries, embeddings, and caches — not just source rows. Prompt caches expire on their own TTL, which helps, but the summary that quoted the deleted fact does not.

What to build first

If you are starting on Monday, the sequence that gets the most value per unit of work:

1. Fix your prompt ordering and turn on caching. Frozen content first, volatile content last, no timestamps in the system prompt. Then check cache_read_input_tokens in production. This is a few hours of work against the largest single cost lever you have, and it is worth doing before any memory design at all.
2. Add a summarizer for long conversations. Keep the last N turns verbatim, replace the rest with a rolling summary. Check whether your provider’s compaction already does this before you write it.
3. Add semantic memory with trigger-based writes. Start with pgvector in the Postgres you already run. Write on explicit triggers — decisions, corrections, tool results — not on every turn. Atomic records, timestamped, attributed, tenant-scoped from the first line of code, because retrofitting scope is painful and retrofitting attribution is worse.
4. Instrument retrieval before you tune it. Log what was retrieved, whether it was used, and what the outcome was. Almost every retrieval improvement people make by intuition can be made better by looking at which memories actually get read.
5. Only then consider the frameworks. Paging, temporal graphs, and model-based extraction all solve real problems. Adopt one when you have hit the problem it solves, and you will know, because it will show up in the retrieval logs from step 4.

The thing to hold onto through all of it: memory is not a component you add to make the agent smarter. It is a set of decisions about what your system re-reads before it speaks, and every one of those decisions is yours. The model does not remember. Your code does.

Built properly, that sequence is a quarter of work, and the maintenance half is the part most teams learn by discovery. Our implementation engagement exists for the ones who would rather not.

References

• n8n, AI agent memory: a practical guide
• Liu et al., Lost in the Middle: How Language Models Use Long Contexts
• Hong, Troynikov & Huber, Context Rot: How Increasing Input Tokens Impacts LLM Performance, Chroma, July 2025
• Packer et al., MemGPT: Towards LLMs as Operating Systems
• Rasmussen et al., Zep: A Temporal Knowledge Graph Architecture for Agent Memory
• Chhikara et al., Mem0: Building Production-Ready AI Agents with Scalable Long-Term Memory
• Dong et al., A Practical Memory Injection Attack against LLM Agents
• Anthropic, Prompt caching and the memory tool