An AI agent can complete every step successfully and still act on outdated information. Learn what memory and context are, why they matter for AI agents and how traces help pinpoint where information went wrong. See how memory tools help update or remove stale records before they affect another run.
When an AI agent gives a customer the wrong answer, where do you look first? Probably somewhere that says everything worked. Let’s say a customer asks your support agent for a replacement order. The agent retrieves a shipping address, creates the replacement and closes the ticket. Every operation reports success—but the replacement is headed to a house the customer moved out of two years ago.
As far as your dashboards can tell, nothing failed. The database answered, the model responded and the ticket closed. The problem was what the agent remembered about the customer and what it passed to the AI model. Somewhere along the way, outdated information became the basis for an action.
To understand why, you need to see which records the agent retrieved, which information reached the model and which address it passed to the shipping tool. That investigation starts with understanding the relationship between memory and context.
What Are Memory and Context in an AI Agent?
Memory and context are the two ways an AI agent supplies information to the model to enable it to make decisions, and your application controls both. An AI agent is an application that uses an AI model to choose its next step and call tools (outside functions like a database lookup or an API request). The model knows nothing about your customer’s history or the state of an order, so your application has to supply all of that.
Context is what your application supplies for one model call: system instructions, the current request, recent messages, records looked up for this request and descriptions of the tools the model may call. Those tool descriptions add up fast. Give an agent a few dozen tools, and their descriptions can take up more room than the customer’s question.
The context window limits how much information a model can process in one call. Bigger isn’t automatically better, because relevant information gets harder for the model to use when a long context surrounds it. When a conversation outgrows the context window, many frameworks quietly drop the oldest messages, summarize them or reject the request without indicating any failure. This can result in silent failure.
Memory is information the agent stores for later use, such as a user’s language preference. It persists beyond the current model call and often beyond the session. Plenty of memory is written by the model itself as conversation summaries. If one of those summaries is inaccurate, the error can influence future calls until the stored information is corrected or removed.
Stored memory has no effect on the model until your application retrieves a record and places it into context.
Pro Tip: Reserve “memory” for stored information and “context” for the input sent to a model call. Consistent labels make traces much easier to read and failures easier to investigate.
Why Do Memory and Context Change AI Agent Observability?
Memory and context change AI agent observability by adding a chain of decisions before the response that conventional monitoring never sees. To investigate a failure, you need visibility into these decisions. Observability is the ability to understand what a system did internally from the signals it produces.
In a conventional application, latency, error rates and logs can help narrow down a problem fast. An agent that returns 200 OK on time after making a bad decision produces none of those signals, and your dashboard will happily file it under success.
Before an agent answers, it may form a query against memory, retrieve stored records and select information to include in the model’s context. Each of these operations can complete successfully while passing along information that leads to the wrong outcome.
The request succeeded. The decision failed.
In the replacement-order example, when an agent mails a replacement to a customer’s old address, a database event that says “retrieval completed” leaves several questions unanswered.
Useful evidence for investigating whether it behaved as intended includes the customer identifier that filtered the query, the memory records that came back and the address that entered the model’s context. With such evidence, you can more readily separate a retrieval failure from a context-selection failure, and both from a model that made a bad call with good context.
A connected trace can help you follow information across the necessary boundaries, investigate where a failure may have begun and determine potential changes.
What Should an AI Agent Trace Capture?
A useful AI agent trace- should capture the relevant steps between the memory query and the answer the customer sees in a connected record. This includes retrieval, context assembly, model calls and tool execution within the same agent run.
Build the trace from spans. A span is a timed record of one operation, with a start, an end, a status and whatever attributes you attach to it. The emerging - OpenTelemetry conventions for generative AI agent spans- give developers a shared vocabulary for agent operations, so a retrieval span looks the same no matter which framework produced it.
You don’t have to build all of that instrumentation from scratch. The - Progress Agent Engineering- platform for observing and evaluating production agents builds on OpenTelemetry. Its software development kits (SDKs) can capture OpenTelemetry trace data for supported large language model (LLM) providers and frameworks, and its - trace view- can display available telemetry such as prompts, spans, model calls, tools and retrieval for a run.
For memory and context investigations, capture:
- Retrieval inputs: Record the memory query, the identifier that filtered it and the store that received it.
- Retrieved records: Record the IDs, timestamps and sources of the records returned.
- Context assembly: Record which retrieved information entered the context, plus the instruction version and available tools.
- Model or tool activity: Record the model output, any tool call arguments and the tool result.
Record IDs and versions matter because stored information can change after a run.
Capture only the content needed for diagnosis, redact sensitive fields before export and restrict access to traces containing customer information.
Warning: A trace that captures full prompts also captures every customer detail in them. Mask personal data before it lands in your tracing backend, or your observability tool could become a privacy leak.
What Memory and Context Failures Can Observability Expose?
Memory failures and context failures can produce different outcomes, leave different evidence in a trace and require different fixes:
Stale Memory. Retrieval may return a fact that was once correct but has since changed. Suppose a user changes an address, but retrieval returns the old record. That’s sneakier than it sounds, because semantic search ranks records by how similar they are to the query, not by how recent they are. An old address and a new one look nearly identical to an embedding model, so nothing stops the stale record from winning. The trace should show the record’s timestamp and source, so the stale record is obvious.
Incorrect Scope. A missing filter can create additional risk. If a user_id filter is left off the query, the support agent could retrieve another customer’s address and place it into context. The retrieval span would still be green because the database answered. Whether the database answered for the right user is a separate check.
Inaccurate or Poisoned Memory. Persistent storage also raises the risk of memory poisoning, where a malicious instruction saved to an agent’s long-term memory is retrieved later and steers its behavior. Bad context affects one run. Poisoned memory keeps supplying bad context until someone finds and removes the record.
Removing a record takes tooling of its own. The Agent Engineering memory and context features let you mark a memory active, outdated or removed, and semantic recall can return retrieved memories with source attribution and a relevance score.
Context Assembly Failures. Context has its own failures. Your application can retrieve the right record and then leave it out of the context. In other runs, the application attaches an outdated summary or bury the decisive instruction under so much history that the model underuses it.
Without both retrieval and context spans, omitted records, outdated summaries and buried instructions all collapse into the same useless diagnosis: The model gave a bad answer.
Pro Tip: Consider storing a timestamp and source with memory records and incorporating recency into retrieval for time-sensitive facts such as addresses.
How Can You Observe and Improve Memory and Context?
Observe and improve memory and context by turning one representative failed run into a repeatable test. Start by capturing enough relevant telemetry to help reconstruct the run for investigation. The National Institute of Standards and Technology (NIST) report on monitoring deployed AI systems separates operational monitoring from the harder question of whether an AI system keeps working as intended.
Pick a failed run and answer four questions from the trace.
- Which memory query ran?
- Which records came back?
- Which information reached the model?
- Which action followed?
If the trace can’t answer one of them without guessing from the final response, add a span at that boundary. Save the explainable run as a baseline and compare traces across releases. And keep failed-run telemetry, because trace sampling that keeps one run in 100 will, with depressing reliability, throw away the one you needed.
Next, turn the incident into a regression test. Preserve the request, the relevant memory state and the expected action. For the replacement-order example, test whether retrieval selects the expected current address for the intended customer and whether the shipping tool receives the expected value.
Use evaluations to assess the result against explicit criteria. An exact address or customer identifier suits a deterministic check. Response helpfulness may require human review or an evaluator model.
Build a small test set with successful cases and known failures. Datasets and experiments make model and prompt comparisons repeatable. When you test retrieval or tool behavior, reproduce the relevant memory and application state in the test workflow.
After deploying a change, inspect new traces to assess whether the corrected information reaches the model and whether the intended action follows.
For the replacement sent to an old address, the investigation should produce a specific correction and a regression test designed to detect similar failures before they affect another shipment.
The replacement that shipped to a two-year-outdated address still comes down to one question, and it’s the one to ask of every failed run: Did the failure begin in memory, in context or in the model’s decision?
If you don’t have tracing in place yet, the Progress Agent Engineering platform has a free plan and SDKs for .NET, Python and TypeScript.
Get Started
