AI is moving quickly from a tool people use to generate information and recommendations to technology that can increasingly take action on their behalf. That shift is driving a public debate about how confidence in systems that increasingly act with autonomy could be shared—and who should be responsible for providing it. In PwC’s latest, The AI trust dividend: Three governance shifts to build trust in agentic AI autonomy, we explore what that shift means for organizations as they begin deploying AI agents with greater autonomy across their businesses.
The opportunity is significant. So is the change in how companies think about trust. PwC's AI performance study found that the 20% of organizations capturing the large majority of AI's economic value are roughly twice as likely as their peers to run AI autonomously.
As AI becomes embedded in business processes, trust shouldn’t exist only in policies, principles, or reviews conducted before a system is deployed. It should increasingly be built into how AI operates every day—who or what is authorized to act, what decisions an agent can make, how those actions are monitored, and when a person needs to step in.
In other words, trust should become operational. It also has to become demonstrable. Increasingly, the question isn't whether leadership is confident in a system, but whether someone outside the company would reach the same conclusion.
Human oversight evolves with AI
Keeping humans appropriately involved has long been an important principle of responsible AI, and that isn’t changing. What is changing is the role people need to play as AI systems become more capable of executing work.
When an organization is experimenting with a relatively small number of AI applications, people can be directly involved in reviewing outputs and approving decisions. As organizations begin deploying portfolios of agents capable of completing multi-step processes and interacting across systems, reviewing each individual action becomes increasingly difficult.
That doesn’t make human oversight less important. It makes clear human accountability even more important.
The human role increasingly shifts toward designing the environment in which agents operate: establishing objectives and guardrails, defining decision rights, determining where human judgment is non-negotiable, and creating clear escalation points when an agent encounters something outside its authority or expectations.
Every agent ultimately needs an accountable business owner. Accountability that cannot be evidenced is an intention rather than a control. The practical test is whether the owner could show, after the fact, what the agent was authorized to do, what it actually did, and where a person intervened. Leaders should know what an agent is responsible for, what it is permitted to do, and when a person needs to approve, review, or override its actions.
Treat agents like part of the workforce
One practical way to think about this is to consider AI agents as part of an organization’s broader workforce.
Companies know who their employees are. They define roles and responsibilities, establish access rights and decision authority, evaluate performance, and determine who is accountable for outcomes. As agents begin performing more meaningful work, organizations will need many of those same disciplines for their non-human workforce—with controls appropriate to the risks and autonomy involved.
That means knowing which agents are operating across the organization and giving each a defined role, appropriate access, and clear boundaries. It also means retaining sufficient evidence of what agents are doing so organizations can understand whether they continue to operate as intended.
This is harder than it sounds. Many organizations have a reasonable view of the agents they built and a much weaker view of the ones arriving inside software they purchased. Capability is being embedded into enterprise platforms and service providers faster than the documentation describing it, which means the inventory problem may begin before anyone has made a deliberate decision to deploy an agent at all.
This is where observability becomes particularly important, and where a common gap appears. Agent monitoring being built today often focuses on engineering purposes: identifying failures, improving performance, and managing cost. Evidence has different requirements. It needs to survive long enough to be useful, resist alteration, and allow someone to reconstruct what conditions were in place at the time an action was taken. Those are design decisions, and they are considerably cheaper to make now than to retrofit later.
Organizations should be able to understand the material actions an agent took, what information it relied upon, where exceptions occurred, and when an issue was escalated. The goal isn’t to capture everything simply because the technology makes it possible. The level of oversight and evidence should reflect the significance of the process and the autonomy an agent has been given.
Build oversight for how AI operates
AI governance has traditionally placed significant emphasis on what happens before deployment: how a model was designed, what data it uses, how it was tested, and whether it was approved.
Those fundamentals remain important. But more autonomous systems increase the importance of what happens after deployment as well.
AI systems and their environments can change. Some of that change will originate outside the organization. Models, tools, and underlying services are updated by providers on their own schedules, which means a system can behave differently without anyone inside the company having changed anything. Their use can change. And as agents begin interacting with other agents, tools, and business processes, organizations need ways to identify when performance or behavior moves outside expectations.
For higher-impact uses, that means moving toward more continuous monitoring and creating the ability to intervene when needed. The question is no longer only whether an AI system was appropriately designed and tested at launch. Leaders also need confidence that it continues to operate within its intended purpose and boundaries over time.
That shift can also create an opportunity. Agentic systems can be designed to generate richer evidence about their actions, decisions, approvals, and exceptions as they operate. Done well, organizations can build the record they need along the way rather than trying to reconstruct it after something happens. It also matters who that record is eventually serving—internal and external. Boards, customers, counterparties, and regulators are beginning to ask versions of the same question, but they may not all accept the same answer.
Trust should enable progress
None of this is about creating more process for the sake of process or slowing AI innovation. The objective is the opposite.
As organizations give AI greater autonomy, they need confidence that the systems performing that work are operating within the boundaries the business has established. Building accountability, observability, and oversight into AI from the beginning can give leaders greater confidence to expand its use—and focus their attention on where AI can create meaningful business outcomes.
Much of the current public debate about AI is really a debate about verification and independent assurance: who assesses these systems, against what criteria, and what the assessment means to someone relying on it. Those questions are being worked out at the frontier, slowly and in public. They are arriving inside long-established companies faster, and with more immediate consequence. The organizations that make trust in AI operational alongside their AI ambitions will likely be better positioned to answer them, to move from experimentation toward confidence at scale, and to capture more of the value AI can create.