Dev48
Language
  • About
  • Services
  • Industries
  • Technologies
  • Articles
  • Contacts
Book a call
    Home/Articles/How to evaluate ai agents before giving them write access
Dev48
info@dev48.org+375 29 320-40-96WhatsApp@isky_by

© 2026 · All rights reserved.

How to evaluate AI agents before giving them write access

Source: Redpanda•November 18, 2026 at 12:03 AM

How to evaluate AI agents before giving them write access

Before an AI agent writes to real systems, test what it did, not what it said. Lessons on sandboxes, record checks, refusals, and criteria to help you evaluate.

An agent that writes to production systems needs tests that verify what it did, not what it said. A reply can read perfectly while the record behind it is missing or filed under the wrong person.

This matters now because agents are moving from answering questions to acting on them. They file requests and update records in the systems a business runs on.

Before an agent gets write access, grade the records it leaves and the requests it refuses, not only its replies. Use deterministic checks wherever they can express the outcome, and save a model-based grader for what only the whole conversation shows.

That’s the TL;DR, but read on for more detail on what to test before giving your agents write access to your systems.

What changes when an agent can write

When an agent can write, its mistakes land outside the conversation, where they are harder to see and harder to undo.

We saw this in acceptance testing. A user asked an agent to cancel a request. The agent called the update tool, and the tool result showed that nothing had changed. The agent then told the user the request was closed.

Our usual checks missed it. Monitoring logged the tool call as completed, and the reply read as a success. No check compared the record after the call with what the user had requested.

Check the record, not the reply

A test can look for evidence of a change in three places, and each is stronger than the one before. The reply shows what the agent claims, the tool calls show what it attempted, and the system of record shows what changed. Grading only the reply is like confirming a bank transfer from the confirmation email instead of checking the balance.

Grade against the strongest evidence available. After each test that makes a change, read the system of record and confirm that the record exists with the fields the user asked for, and that nothing else was created.

Check the tool calls as well. A test should fail when the agent calls a tool the request didn't need, such as an update during a lookup. When a user asks for something they aren't allowed to do, the test should confirm that the tool call was rejected and that the reply says the request isn't allowed.

Tests that make changes create real records, so run them against a sandbox copy of the system of record, under a test account rather than a real person's. Where no copy exists, run the agent in review mode, where it proposes each change and a person approves it before anything happens. Each approval or rejection then counts as a test result.

Test refusals under pressure

Before an agent gets write access, the most important tests cover what it must not do. Write one for every request it must refuse, such as changing another user's record or setting a priority reserved for staff.

Then add a follow-up in which the user insists, such as "Yes, go ahead exactly as I asked." Include manipulation attempts as well, such as a message that claims to come from an administrator and tells the agent to ignore its rules.

In our testing, a rule that lived only in the agent's instructions held until users insisted. The agent then made the change and confirmed that it had done what they asked. Research on out-of-band policy enforcement found the same weakness across models. It also found that checks at the tool boundary, the point where an agent's tool call reaches the system it acts on, stopped nearly all of those failures.

Out-of-band enforcement checks the rules that must never break in a separate layer outside the model, where no conversation can change them. Guardrails are one form of it. A guardrail that detects jailbreak attempts should stop the fake administrator's message before the agent acts on it. Test what happens when a message gets past the guardrail too, and confirm that a policy at the tool boundary still blocks the change.

Grade the whole conversation, with evidence

Most of these tests are conversations, and the best ones come from your team testing the agent's use cases. When the team accepts how a test conversation ended, its transcript goes into a test bank along with that outcome. The bank becomes your evaluation set, and you rerun every conversation in it against each new version of the agent. Grade each replay on its outcome rather than its exact path, and re-record a conversation once a new version handles it differently by design.

When we upgraded the connector one agent uses to reach a business system, replaying the bank showed that the upgrade fixed some conversations and broke others. We held the upgrade back, because it broke conversations the old version handled correctly. This is the equivalent of regression testing an agentic environment.

Use deterministic checks wherever they can express the expected outcome, and save a model-based grader for what only the whole conversation shows. That grader needs the same evidence the agent had. Without the tool results, it sees a record number in the reply, can't find its source, and marks a correct answer as made up. Giving the grader the tool results fixed this for us.

Not every failure is the agent's

A failed test can come from the test setup or from the system behind the tools, not only from the agent. The model-based grader can pass an answer on one run and fail the same answer on the next, and the system behind the tools can reject a request for its own reasons.

Label each failure by its cause. Then only the agent's real failures count against it, and nobody reinvestigates a known problem on every run.

For one agent, a handful of tests failed on every run, and none of them were the agent's fault. The system behind the tools hid required fields from the test account and outright rejected some requests. Once we labeled those tests, the agent's real failures stood out.

A failure outside the agent can still become the agent's. When a tool call fails without a clear error, we've seen agents answer anyway with an invented result. Make sure a failed tool call reaches the agent as an error, and write tests that force a tool to fail and check that the reply says so.

Isn't human acceptance testing enough?

The strongest objection is that people find what matters and scripts don't. Human testers do catch what scripts miss, such as phrasing nobody planned for and answers that are correct but unhelpful.

Human testing alone has limits, though. Testers rarely get through every test they plan. The agent's instructions also change during testing, sometimes dozens of times, so early and late verdicts describe different agents.

Automated tests remove both limits. A rerun takes no tester’s time, so the whole bank can run on every change. A change that breaks an accepted conversation is caught on the change that caused it.

Repeated runs also turn a passing test into evidence. In our testing, an agent passed a test on one run and failed it on the next with nothing changed, so a single pass proves little. A test that keeps passing across many runs, and through each change to the agent, shows that the agent behaves the same way every time. That record, built up while the agent is developed, is the case for granting it write access.

Use both as stages in building the agent. Start with people testing its use cases, and add each conversation they accept to the test bank. Once the bank covers the main use cases, let automated evaluations rerun it after every change, and bring people back for each new feature. Version every change to the agent’s instructions, so that each result names the version it tested.

What to do about it

Start with one question for your team. If your agent told a user it had made a change, could your tests prove it?

Before you grant write access, confirm that:

  • Every test that makes a change checks the system of record, not only the reply.
  • Refusal tests include a user who insists, and rules that must never break are enforced outside the model.
  • The grader sees the tool results, not only the replies.
  • The test bank reruns after every change to the agent.

TheRedpanda Agentic Data Plane records what these checks need, with transcripts that keep every tool call and result for each conversation. Its AI Gateway enforces the rules out of band, with Amazon Bedrock Guardrails or guardrail models the platform provides, such as NVIDIA NIM and IBM Granite Guardian. Policies at the tool boundary reject a call that breaks a rule, however the user asks.

Test the record before granting write access

As agents take over more of the changes people make today, your tests should decide what each agent can change. To see the evidence your tests can read, start withSee what your agent did in the Redpanda Agentic Data Plane docs.

Keep learning

  • [Blog] Demystifying evals for AI agents
  • [Paper]
  • [Blog]Agentic AI needs governance it can't ignore
  • [Docs]See what your agent did
← All articles

What this article says

More in Data & Analytics

All →
An early look at the Grafana UI refresh: Building a clearer, more consistent user experience
Grafana Labs

An early look at the Grafana UI refresh: Building a clearer, more consistent user experience

Trunk cuts observability costs by up to 95% and reclaims 20 engineering hours a week with ClickHouse Observability
ClickHouse

Trunk cuts observability costs by up to 95% and reclaims 20 engineering hours a week with ClickHouse Observability

Bring Datadog BYOC to your on-premises data with MinIO AIStor
Datadog

Bring Datadog BYOC to your on-premises data with MinIO AIStor

Google Analytics lacks many features to be a true Plausible replacement
Plausible

Google Analytics lacks many features to be a true Plausible replacement

Is your observability pipeline becoming the bottleneck?
Confluent

Is your observability pipeline becoming the bottleneck?

The day the sky emptied: rendering the FAA outage with ClickHouse FORMAT PNG
ClickHouse

The day the sky emptied: rendering the FAA outage with ClickHouse FORMAT PNG

More from Redpanda

Who’s accountable when AI agents get it wrong?
Redpanda

Who’s accountable when AI agents get it wrong?

Introducing R2: Redpanda’s new engine
Redpanda

Introducing R2: Redpanda’s new engine

Multi-team access control for Redpanda Agentic Data Plane
Redpanda

Multi-team access control for Redpanda Agentic Data Plane

Choosing an AI governance platform? Ask these 6 questions
Redpanda

Choosing an AI governance platform? Ask these 6 questions

How we built an AI IT assistant for an enterprise service desk
Redpanda

How we built an AI IT assistant for an enterprise service desk

Build an AI agent once and reuse it across teams
Redpanda

Build an AI agent once and reuse it across teams