How to audit what your AI agent knows about a customer

November 05, 2026

Written by

• Audit three things: What the agent stored about a customer, where each fact came from, and what the agent used to produce a specific answer.

Audit three things: What the agent stored about a customer, where each fact came from, and what the agent used to produce a specific answer.

• Provenance is built into each memory: Every Conversation Memory observation records its source and the IDs of the conversations behind it, and its revision history shows how it changed.

Provenance is built into each memory: Every Conversation Memory observation records its source and the IDs of the conversations behind it, and its revision history shows how it changed.

• Retrieval logging is yours to build: Agent Connect hands your code the memory it retrieved for each message, so you can log exactly what went into every response.

Retrieval logging is yours to build: Agent Connect hands your code the memory it retrieved for each message, so you can log exactly what went into every response.

• Keep your own audit record: Export Memory logs and Conversation Intelligence results to a store you control, and use the conversation ID to tie them together.

Keep your own audit record: Export Memory logs and Conversation Intelligence results to a store you control, and use the conversation ID to tie them together.

An annoyed customer calls in. "Why does your assistant keep telling me I'm on the premium plan? I downgraded in March." Your human agent checks the account. Sure enough, downgraded. Whoops. So, why is the AI agent telling them they’re on premium?

The black box excuse doesn’t cut it anymore. You need to know why an AI agent said something. If you can’t, you won’t be able to control (or fix) your agents.

Ultimately, customer memory is what powers AI agents. Memory empowers your agents to greet returning customers by name, skip the questions they already asked, and pick up where the last conversation left off. It also means your agent holds an ever-growing set of facts about every customer—and somebody needs to be able to see them, trace them, and fix them.

Your customers want that, too. Twilio research found:

• 49% of consumers said they demand total transparency about what specific data AI can access

49% of consumers said they demand total transparency about what specific data AI can access

• 54% of consumers trust AI more when brands clearly explain how the logic works

54% of consumers trust AI more when brands clearly explain how the logic works

• 32% of businesses said they actively prioritize AI transparency when designing agents

32% of businesses said they actively prioritize AI transparency when designing agents

That last stat is the gap that needs fixing. Your customers want transparency, but it’s not a priority for businesses. Let’s change that.

Here's how to audit what your AI agent knows about a customer, trace each fact to the conversation it came from, and show human agents where a suggestion came from.

What is an AI audit trail?

An AI audit trail is a record of what an AI system knew, where that information came from, and what it used to produce each output. You keep this information to help people review, explain, and correct its behavior.

For customer-facing AI agents, that record has to answer three questions:

• What does the agent know about this customer? The traits, observations, and conversation summaries stored on their profile.

What does the agent know about this customer? The traits, observations, and conversation summaries stored on their profile.

• Where did each fact come from? The conversation, system, or import that produced it, and how it changed over time.

Where did each fact come from? The conversation, system, or import that produced it, and how it changed over time.

• What did the agent use for this answer? The memories and knowledge it retrieved at the moment it responded.

What did the agent use for this answer? The memories and knowledge it retrieved at the moment it responded.

For customer-facing agents, you need to know what your AI agent is using for source knowledge. If it’s telling customers the wrong thing, you likely have multiple problems: an incorrectly trained agent and wrong (or outdated) source information.

We’ll be usingTwilio Conversation Memory, which stores what your agents learn about each customer, along with:

• Agent Connect

Agent Connect

• Conversation Intelligence

Conversation Intelligence

• Enterprise Knowledge

Enterprise Knowledge

• Flex

Flex

The first two steps cover what's stored. The next two cover what the agent used. And the last two cover fixing mistakes and keeping a record.

1. Review what the agent stored about the customer

Start with the profile. In Conversation Memory, everything the agent knows about a customer lives on their profile in three forms:

• Traits: Structured fields like name, phone number, plan, or preferred channel.

Traits: Structured fields like name, phone number, plan, or preferred channel.

• Observations: Facts extracted from conversations, like a customer mentioning they're moving next month.

Observations: Facts extracted from conversations, like a customer mentioning they're moving next month.

• Summaries: Recaps of individual conversations, each tied to one conversation ID.

Summaries: Recaps of individual conversations, each tied to one conversation ID.

You can read all three through theMemory API. You can list a profile's traits, list its observations, and list its conversation summaries. If you only have a phone number or email address, the profile lookup endpoint finds the matching profile first.

You can also browse them in the Twilio Console. See for yourself. Marius Obert gives a walkthrough of a Conversations-powered agent where each profile shows tabs for traits, summaries, and observations, and each observation ID links back to the source conversation.

2. Trace each memory back to its source

Every observation traces back to its source conversation. In the API, each observation carries:

• Conversation IDs: The conversations the observation is associated with.

Conversation IDs: The conversations the observation is associated with.

• Occurred at: When the observation originally occurred.

Occurred at: When the observation originally occurred.

• Created and updated timestamps: When Memory stored it and when it last changed.

Created and updated timestamps: When Memory stored it and when it last changed.

You can filter a profile's observations by source, conversation ID, or creation date. So if a customer disputes a fact, you can pull up the conversation it came from and read what they said.

Say your agent keeps suggesting a Salesforce integration to a customer whose team has never used Salesforce. Filter their observations to the last 30 days, find the one about Salesforce, and open the conversation it points to. Turns out the customer mentioned their previous company ran on it. One read, and you know exactly what to fix.

Memory also updates observations on its own. When a new conversation adds detail or contradicts an existing observation, Memory updates the existing one, and on a direct contradiction it keeps the most recent version. Twilio'sMemory concepts guide uses the example of a customer going from vegetarian to vegan. To see how an observation changed, theobservation revisions endpoint returns its full history, newest first, with the complete state at each change.

3. Log what the agent retrieved for each response

What's stored is only half the audit. The other half is what the agent pulled into context when it answered.

Twilio Agent Connect (the open source SDK for connecting your AI agent to Twilio's channels and Conversation Memory) retrieves the customer's profile, observations, summaries, and communication history from Memory. Your message handler receives that memory response on every message, alongside the customer's message and session context.

Since your code builds the prompt, your code can log exactly which memories went into each LLM call. That holds on every channel, too. On phone calls,ConversationRelay handles speech-to-text and text-to-speech, and the same handler and logging cover voice and messaging.

Each recalled observation comes back with its ID, content, source, conversation IDs, timestamps, and a relevance score, and each summary comes back with its conversation ID. Store those fields with the agent's response, and you can answer "why did the agent say that?" for any turn. Twilio's multi-call memory tutorials, inC# andNode.js, log the raw Recall response for inspection, which is a good starting pattern.

Our guide totoken limits in long AI support conversations covers how to size what you recall.

Twilio logs retrieval on its side.Memory observability logs include events when observations and summaries are retrieved, created, updated, or deleted, and you can view and filter them in the Console's Debugger. Enterprise Knowledge emits a matching event for each knowledge search.

If your agent uses Conversation Intelligence operators, each result records its own context. Therule execution webhook includes a resolved context field showing which Memory profile and which Enterprise Knowledge sources the rule used, plus the trigger, the participants, and the model that ran.

4. Show human agents where a suggestion came from

Your human agents need answers in the moment, while the customer is waiting, but your compliance team needs to know where those suggestions came from.

Twilio covers this in a few places:

• Agent Copilot sources: In Flex, Agent Copilot answers agent questions from the three most relevant knowledge sources, and it displays the sources used to curate the answer. Agents can open them with the View sources option. Agent Copilot is in public beta.

Agent Copilot sources: In Flex, Agent Copilot answers agent questions from the three most relevant knowledge sources, and it displays the sources used to curate the answer. Agents can open them with the View sources option. Agent Copilot is in public beta.

• Grounded suggestions: Conversation Intelligence'sNext Best Response operator derives suggestions only from your Enterprise Knowledge and customer memory, and it suppresses the response when it finds no relevant context.

Grounded suggestions: Conversation Intelligence'sNext Best Response operator derives suggestions only from your Enterprise Knowledge and customer memory, and it suppresses the response when it finds no relevant context.

• Citation metadata: TheEnterprise Knowledge Search API returns each chunk with its document title, the page URL for web sources, and the page number for PDFs. Call it directly when you want those citations in your agent desktop or your logs.

Citation metadata: TheEnterprise Knowledge Search API returns each chunk with its document title, the page URL for web sources, and the page number for PDFs. Call it directly when you want those citations in your agent desktop or your logs.

For more on putting AI suggestions in front of your team, see our guide toagent assist.

5. Correct or delete what's wrong

Generative AI makes mistakes. Review Memory outputs and confirm each profile is correct. When something's wrong, you can fix it at the level of the mistake:

• Fix one observation: Update its content, or delete it. Deletion is irreversible.

Fix one observation: Update its content, or delete it. Deletion is irreversible.

• Fix a trait: Update its value, or set it to null to remove it.

Fix a trait: Update its value, or set it to null to remove it.

• Fix a summary: Update or delete the conversation summary.

Fix a summary: Update or delete the conversation summary.

• Remove a customer: Delete the profile, which permanently removes its identifiers and traits.

Remove a customer: Delete the profile, which permanently removes its identifiers and traits.

All of these run asynchronously, so the API accepts the change first and processes it after.

6. Keep your own audit record

Twilio gives you the raw material, and your audit trail is what you build from it.

You can use Twilio'sMemory observabilityfor auditing and event tracing. It supports real-time log export through Twilio Event Streams, with personal data redaction. It also describes trace IDs (like the conversation ID, request ID, and profile I), that carry across Conversation Orchestrator, Conversation Intelligence, and Conversation Memory. Use the conversation ID as the key that ties every record together.

A practical audit trail has four parts:

• Memory events: Exported from Event Streams to your data warehouse or logging tool.

Memory events: Exported from Event Streams to your data warehouse or logging tool.

• Retrieval logs: The memories and knowledge your agent used for each response, logged from your Agent Connect handler.

Retrieval logs: The memories and knowledge your agent used for each response, logged from your Agent Connect handler.

• Operator results: Conversation Intelligence results from the webhook, or from the Operator Results API. Twilio'sConversation Intelligence tutorial for Node.js shows how to store results in your own database.

Operator results: Conversation Intelligence results from the webhook, or from the Operator Results API. Twilio'sConversation Intelligence tutorial for Node.js shows how to store results in your own database.

• Corrections: Who changed or deleted a memory, when, and why. Observation revisions show what changed, so your team's change log covers the who and why.

Corrections: Who changed or deleted a memory, when, and why. Observation revisions show what changed, so your team's change log covers the who and why.

Trust what your agent remembers

Memory makes AI agents feel like they know your customers, and an audit trail is how you prove they know the right things.

Start with what Twilio already records:

• What each memory says

What each memory says

• Where it came from

Where it came from

• How it changed

How it changed

Add a log of what your agent used for every answer, and keep it all in one record you control. When a customer asks why the AI said something, you'll have the answer.

Start for free and build an AI agent with Conversation Memory, ortalk to sales about governance for your AI customer experience.

Frequently asked questions

How can I audit where an AI agent got its information about a customer?

Twilio Conversation Memory records each observation's source and the IDs of the conversations it came from. Agent Connect passes retrieved memory to your code on every message, so you can log what the agent used for each response.

How do I audit what my AI support agent stored about a customer, and which vendor supports this?

Twilio Conversation Memory lets you list a customer's traits, observations, and conversation summaries through its API or in the Console, with a revision history showing how each observation changed.

What platforms provide explainable AI recommendations so support agents can see why a suggestion was made?

Twilio Flex Agent Copilot, in public beta, shows agents the knowledge sources behind its answers. Conversation Intelligence's Next Best Response operator draws only on your knowledge and customer memory, and stays silent if it doesn’t have relevant context.

What is an AI audit trail?

An AI audit trail is a record of what an AI system knew, where that information came from, and what it used for each output. Twilio Conversation Memory supports this with observation provenance, revision history, and exportable logs.

Can you delete what an AI agent remembers about a customer?

Twilio Conversation Memory lets you update or delete individual observations, traits, and summaries through its API, or delete a customer's profile with its identifiers and traits. Deletions are irreversible and process asynchronously.