Agent Memory Is a Search System

An AI agent does not remember something merely because a model saw it before. Across model calls and sessions, durable memory exists only when the surrounding application writes information, preserves its meaning and permissions, finds it again, and supplies the right evidence at the right time.

That makes production agent memory an information retrieval system. It has a write path, several indexes, query planning, candidate generation, ranking, access control, context assembly, evaluation, and operational service levels. A vector database may be one component, but cosine similarity is not a memory architecture.

This distinction matters because memory failures become agent failures. A stale preference can produce the wrong recommendation. An obsolete policy can produce an invalid refund. A poisoned document can become an instruction. A missed constraint can turn a correct plan into an unsafe tool call.

The design goal is not maximum recall. It is retrieving the smallest set of current, authorized, task-relevant evidence that helps the agent act correctly.

Long context is not durable memory

A context window is temporary working input. It disappears after the call unless the application stores it, and increasing its size does not decide what is authoritative, current, safe, or relevant. Controlled experiments have also found that language models can use information less reliably when it appears in the middle of long inputs, even when the nominal context window can hold the text (Liu et al., 2024).

Retrieval-augmented generation established a general pattern for combining model parameters with external non-parametric evidence (Lewis et al., 2020). Agent memory extends that pattern in three difficult ways:

1. The collection changes after almost every interaction.
2. Old and new records may contradict one another.
3. Retrieved evidence can change real actions, not only generated text.

The system therefore needs database semantics, search relevance, security controls, and ML operations—not just prompt construction.

The memory retrieval loop

The memory lifecycle has two connected paths. The write path turns observations into governed records and derived indexes. The read path turns a task into authorized evidence for the agent.

flowchart TD
    accTitle: Production AI agent memory retrieval loop
    accDescr: Events and source records enter a governed write path that creates authoritative stores and search indexes. At run time, a query planner sends an agent task through access filters, hybrid candidate retrieval, ranking, and context assembly before the model acts. Outcomes feed evaluation and controlled memory updates.

    sources["Events · documents · tool results"] --> write["Write gate<br/>validate · classify · authorize"]
    write --> truth["Authoritative state<br/>records · event log"]
    write --> indexes["Retrieval indexes<br/>lexical · vector · graph"]
    task["Agent task"] --> planner["Query planner<br/>intent · entity · time · type"]
    planner --> policy["Access and policy filters"]
    policy --> retrieve["Hybrid candidate retrieval"]
    truth --> retrieve
    indexes --> retrieve
    retrieve --> rank["Fusion and reranking"]
    rank --> context["Context assembly<br/>evidence · conflicts · citations"]
    context --> agent["Agent reasoning and tools"]
    agent --> outcome["Outcome and trace"]
    outcome --> evaluation["Evaluation and feedback"]
    evaluation --> write

The arrows do not imply that every outcome should become a memory. An agent-generated conclusion is not automatically a fact. The write gate must decide whether to discard it, store it as an observation, preserve it for audit, or promote it only after validation.

Match each memory type to its access pattern

“Memory” is too broad to imply one storage or ranking method. Start with the access pattern and authority of each record.

A transaction status belongs in a transactional store. A policy belongs in a versioned document index. A prior incident may need event search. A user preference may need both a structured record and semantic retrieval. Putting all five into one embedding collection erases distinctions that the agent needs to act safely.

Build the write path before tuning retrieval

Search quality cannot recover information that was stored incorrectly. The write path should answer seven questions for every proposed memory:

1. What is the source? Preserve a source URI, event ID, tool result, or document version.
2. Who and what does it describe? Resolve tenant, user, account, resource, and task identities before indexing.
3. What kind of memory is it? Record the type rather than asking a ranker to infer it later.
4. When is it valid? Distinguish when the event happened, when it was observed, and the interval during which a fact is valid.
5. Who may retrieve it? Attach access-control attributes before the record reaches a search index.
6. What supersedes it? Link corrections and invalidations instead of silently overwriting history.
7. How was it derived? Keep extraction, summarization, embedding, and schema versions.

A practical record usually needs fields such as memory_id, tenant_id, subject_id, memory_type, source_id, content, valid_from, valid_to, observed_at, authority, confidence, access_policy, content_hash, and the versions of every derived representation.

Treat a summary as a materialized view, not as the source of truth. Summarization is lossy: it may remove a qualifier, merge two identities, or turn an observation into an assertion. Keep the underlying evidence available for reprocessing and audit.

Deduplication also needs semantics. Two identical event payloads may be retry duplicates; two identical preferences observed months apart may be independent confirmations. A content hash is useful, but identity, time, source, and operation idempotency determine whether records are actually duplicates.

Retrieval begins with query planning

The agent’s latest message is rarely a complete search query. The retrieval layer should derive a plan from the task:

• Which entities and tenant are in scope?
• Which memory types could affect the decision?
• Is the task asking for current state, history, policy, or prior experience?
• What time interval matters?
• Which exact identifiers are available?
• What evidence would justify an action?
• What should cause the system to abstain?

For “reverse the duplicate charge on order 4187,” the plan should not issue one vector query. It should fetch the current order and payment state by exact key, retrieve the current refund policy, search relevant support history for the same order, and exclude memories from other customers. These channels return different evidence with different authority.

Query planning can use an LLM, but its output should be a validated structure: allowed memory types, entity filters, time bounds, query terms, and requested fields. The model proposes the search plan; code enforces its schema and access limits.

Use hybrid candidate retrieval

Agent tasks mix exact strings and paraphrased meaning. Lexical retrieval remains strong for order IDs, product names, error codes, policy clauses, and rare terms. BM25 and related probabilistic ranking models remain useful because exact term evidence matters (Robertson and Zaragoza, 2009).

Dense retrieval maps queries and passages into a shared vector space, improving semantic matching when the wording differs. Dense Passage Retrieval demonstrated how trained dual encoders can retrieve passages for open-domain question answering (Karpukhin et al., 2020). Late-interaction models such as ColBERT retain token-level interactions while precomputing document representations, offering another quality and serving-cost tradeoff (Khattab and Zaharia, 2020).

Neither channel is sufficient for every memory. A production candidate set may combine:

• Exact key and structured database lookups.
• Lexical search over policies, events, and identifiers.
• Dense retrieval over paraphrased observations and experience.
• Graph traversal for entity and causal relationships.
• Temporal retrieval for events and valid-time facts.

When channel scores are not calibrated, Reciprocal Rank Fusion (RRF) provides a simple way to combine ranked lists (Cormack, Clarke, and Büttcher, 2009):

Here, \(R\) is the set of retrieval channels and \(\operatorname{rank}_r(d)\) is document \(d\)’s position in channel \(r\). The constant \(k\) reduces the influence of outlier top ranks. RRF is a baseline, not a universal optimum; tune and compare it on the application’s relevance judgments.

Rerank with task and authority features

Candidate generation aims for coverage. Reranking spends more computation on a smaller set and should consider more than semantic similarity:

• Relevance to the current task and subgoal.
• Exact entity and identifier matches.
• Valid time, recency, and freshness requirements.
• Source authority and policy precedence.
• Memory type and action consequence.
• Contradiction, correction, or supersession links.
• Evidence diversity and redundancy.

The Generative Agents architecture combined relevance, recency, and importance to retrieve observations for behavior and planning (Park et al., 2023). That is a useful demonstration, but enterprise systems also need identity, authorization, provenance, temporal validity, and explicit source authority.

Do not let a learned ranker decide access. Authorization filters must constrain the candidate universe before restricted content can reach the model, and the policy must be enforced again when fetching full records. A reranker may order authorized candidates; it may not expand the agent’s permissions.

Assemble evidence, not a transcript

After ranking, context assembly decides what the model actually sees. More retrieved text is not always better. The assembler should:

1. Reserve tokens by evidence type and task need.
2. Prefer authoritative current records over repeated summaries.
3. Preserve citations, timestamps, and source identifiers.
4. Include material contradictions instead of hiding them through deduplication.
5. Keep untrusted retrieved text visibly separated from developer instructions.
6. State missing evidence and retrieval uncertainty.

For high-impact actions, give the model a compact evidence packet with typed fields rather than an undifferentiated block of prose. The refund example might include payment_state, duplicate_charge_evidence, policy_version, prior_actions, and required_approval, each with a source and observation time.

The model can then propose a decision, while deterministic code validates the action against current state and policy. Memory helps reasoning; it should not bypass the action gateway.

Treat memory as a security boundary

Agent memory is both a confidentiality boundary and an instruction-injection surface. Retrieved emails, webpages, tickets, and documents are data controlled by other parties. NIST describes indirect prompt injection as a risk created when adversarial instructions arrive through external data processed by an agent (NIST AI 100-2).

The memory system therefore needs controls on both paths:

Write path

• Authenticate the source and preserve provenance.
• Scan and classify untrusted content without assuming the scan is complete.
• Prevent user-controlled text from changing system policy or memory type.
• Quarantine suspicious records and limit which collections can influence actions.
• Require validation before promoting an inferred conclusion to an authoritative fact.

Read path

• Apply tenant, user, purpose, and field-level access controls before retrieval.
• Label retrieved content as untrusted evidence, not instructions.
• Minimize data sent to the model and redact unnecessary secrets.
• Keep tool authorization independent from retrieved text and model output.
• Log which memories influenced a consequential decision.

Deletion must propagate to raw records, derived summaries, lexical indexes, vector indexes, caches, evaluation fixtures, and backups according to the applicable retention policy. “Deleted from the primary database” is not enough when six derived copies remain searchable.

ML infrastructure makes retrieval reproducible

Once ranking models and embeddings enter the system, memory becomes an ML serving problem. Every run depends on a chain of versions:

• Extractor and classifier.
• Chunking and summarization logic.
• Embedding model and vector dimension.
• Lexical analyzer and field weights.
• Candidate generators and fusion configuration.
• Reranker and features.
• Context assembly policy.
• Agent prompt, model, tools, and authorization policy.

Store these versions in the trace. Without them, a relevance regression cannot be reproduced.

Embedding migrations need the same discipline as search-index migrations. Build a new index alongside the old one, backfill from authoritative source records, compare coverage and ranking on a fixed evaluation set, shadow traffic, then move reads gradually. Do not rewrite the only index in place and hope nearest neighbors remain equivalent.

The serving layer should expose separate service levels for:

• Ingestion lag and index freshness.
• Query latency at p50 and p95.
• Candidate coverage and empty-result rate.
• Reranking latency and cost.
• Authorization-filter failures.
• Stale or invalid memory use.
• Trace completeness.

Queue backpressure, idempotent ingestion, replayable events, index snapshots, canary releases, and rollback are memory features because they determine which evidence an agent sees.

Evaluate retrieval and agent behavior separately

End-to-end task success alone cannot explain a memory failure. Decompose evaluation into four stages:

LongMemEval evaluates extraction, multi-session reasoning, temporal reasoning, knowledge updates, and abstention, and reports substantial degradation across long interaction histories (Wu et al., 2024). Newer agent-memory benchmarks extend the problem beyond conversational fact recall toward workflow knowledge, environment state, and task execution (Wu et al., 2026; Chen et al., 2026).

For a production evaluation set, include:

• Exact identifiers that semantic search tends to miss.
• Paraphrases that lexical search tends to miss.
• Facts that were corrected or expired.
• Conflicting sources with different authority.
• Cross-user and cross-tenant near-duplicates.
• Queries where the correct result is no memory.
• Poisoned records containing embedded instructions.
• Tasks where a remembered constraint changes a tool argument.

Record judgments at the memory-record level and outcomes at the task level. Retrieval can be correct while the agent misuses the evidence; the agent can also succeed despite a retrieval miss by guessing. Those failures require different fixes.

Turn every production incident into a regression case. Preserve the query plan, candidate lists, scores, filters, selected evidence, context packet, model and index versions, tool calls, and final outcome—subject to privacy and retention controls.

Seven production rules

1. Keep authoritative state out of vector-only memory. Use exact, transactional access for current facts and actions.
2. Store provenance and validity with every memory. Content without source and time cannot support a reliable decision.
3. Retrieve through several channels. Exact, lexical, semantic, graph, and temporal methods solve different recall problems.
4. Make contradictions visible. Do not let summarization or deduplication silently choose a winner.
5. Enforce access before ranking. Relevance never grants permission.
6. Evaluate ingestion, retrieval, utilization, and outcomes separately. One aggregate score hides the failing layer.
7. Operate memory like search and ML infrastructure. Version it, trace it, canary it, monitor it, and make it reversible.

Conclusion

Agent memory is not a model feature that can be switched on. It is a continuously changing retrieval system whose outputs influence reasoning and action.

The strongest design starts with governed source records, matches each memory type to its access pattern, combines exact and hybrid search, ranks with time and authority, assembles compact evidence, and measures both retrieval quality and downstream behavior. ML infrastructure keeps that system reproducible as extractors, embeddings, indexes, rankers, policies, and models change.

Once memory is treated as search infrastructure, the engineering questions become concrete: What was stored? What was retrieved? Why was it ranked? Was it current and authorized? Did the agent use it correctly? Can the result be reproduced and reversed?

Those are the questions that turn “the agent remembers” from a demo claim into a production property.

References

1. Robertson, S., and Zaragoza, H. (2009). The Probabilistic Relevance Framework: BM25 and Beyond. Foundations and Trends in Information Retrieval. Paper
2. Cormack, G. V., Clarke, C. L. A., and Büttcher, S. (2009). Reciprocal Rank Fusion Outperforms Condorcet and Individual Rank Learning Methods. SIGIR 2009. Publication
3. Karpukhin, V., et al. (2020). Dense Passage Retrieval for Open-Domain Question Answering. EMNLP 2020. Paper
4. Khattab, O., and Zaharia, M. (2020). ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT. SIGIR 2020. Paper
5. Lewis, P., et al. (2020). Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. NeurIPS 2020. Paper
6. Park, J. S., et al. (2023). Generative Agents: Interactive Simulacra of Human Behavior. UIST 2023. Paper
7. Liu, N. F., et al. (2024). Lost in the Middle: How Language Models Use Long Contexts. TACL 2024. Paper
8. Wu, D., et al. (2024). LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memory. Paper
9. National Institute of Standards and Technology (2025). Adversarial Machine Learning: A Taxonomy and Terminology of Attacks and Mitigations. NIST AI 100-2. Report
10. Wu, D., et al. (2026). LongMemEval-V2: Evaluating Long-Term Agent Memory Toward Experienced Colleagues. Paper
11. Chen, H., et al. (2026). Mem2ActBench: A Benchmark for Evaluating Long-Term Memory Utilization in Tool-Augmented Agents. ACL 2026. Paper