Salesforce’s Paula Goldman on why AI needs a human at the helm | IBM

The more work AI agents do on their own, the harder it becomes to know when people should step in.

Paula Goldman, Salesforce’s Chief Ethical and Humane Use Officer, told IBM Think in an interview that the familiar idea of keeping a “human in the loop” is no longer enough for AI systems carrying out complex tasks. In her new book, Manage the Machine, she argues for putting people “at the helm” instead, giving them the authority to set boundaries, intervene when necessary and shape how AI changes their jobs.

The following interview has been lightly edited and condensed for clarity.

What led you to write Manage the Machine, and why do you think the human side of AI adoption gets overlooked?

All too often, we treat AI as solely a technical challenge. But the human side, including organizational questions, is equally and sometimes more important to get right.

I’ve been working in AI ethics for the better part of a decade. When AI grabbed the public imagination in 2022, I started getting calls from friends, customers and policymakers. There was a lot of excitement and a lot of fear. Some people saw AI as an inexorable force descending upon us.

That was the opposite of my experience. Working with customers and seeing our own transformation at Salesforce, I’ve seen that putting people at the center and considering how people and AI work together is how you get the right results.

You trained as an anthropologist. What have you noticed about AI adoption that technologists might overlook?

People closest to the work should help shape how AI gets used. They often know where it can help and where it can get in the way.

I tell a story in the book about visiting a local town council in the UK. They were piloting AI to help social workers take notes and document next steps after field visits. Interestingly, the social workers rejected it.

The reason was something small that would have been easy to fix. They wanted to return to the office after their visits to brainstorm with colleagues and talk about difficult cases rather than burden their spouses with them.

The pilot had been introduced in a way that would replace a workplace ritual they valued. Asking how AI should fit into their work would have made a big difference.

You argue that companies need to move from having a “human in the loop” to a “human at the helm.” What’s the difference?

Today, “human in the loop” is widely understood to mean AI drafts something and you approve it or hit send.

That doesn’t really make sense for a world of AI agents. It almost defeats the point.

If an agent is doing complex work on your behalf, you’re delegating tasks with many intermediate steps that touch lots of data. It’s practically impossible for a human to look through all of it. And if you’re looking through everything, what’s the point of having an agent?

What we need is something better, and that’s what we’re calling “human at the helm.”

First, you should be able to set parameters in advance: “Here’s what I want you to do, and I don’t want you to go beyond this.”

Second, the system should escalate something when it requires judgment or goes outside those parameters.

Third, you want a system that increases what you’re capable of at work, extending your agency, creativity and judgment.

How do you make sure people actually pay attention when an AI agent needs their judgment?

We know from research that when AI produces something plausible once or twice, people tend to trust it and stop scrutinizing it. And if they get alerts too often, they may stop paying attention.

I experienced this myself when I was vibe coding a website. I had the wrong setting, so every time my AI agent wanted to visit an external website, it asked permission.

There was a good reason. It was trying to avoid prompt injection attacks and malicious websites. But after the second or third alert, I was just clicking yes, yes, yes.

A better setting would have been to tell the AI to handle those checks itself and alert me if it encountered a suspicious website.

There’s an art to deciding what to escalate to a person versus having AI run its own tests and flag anomalies. If an alert is appropriately calibrated, I’m much more likely to pay attention.

Are there decisions AI agents simply shouldn’t make?

Not every decision AI can make is necessarily one it should make.

Let’s say you’re setting up a customer service agent to check order status and issue returns that comply with company policy.

If anything else gets thrown at that agent, you want it to escalate the request or follow the appropriate protocol. And if someone tries to get the agent to go off policy, you want that to be effectively impossible.

Businesses should be able to have confidence and control over their systems. When systems don’t work that way, it’s a big problem.

Can human oversight create a false sense of security? Who should be accountable for AI decisions?

I think there’s a new model of accountability that’s necessary.

Instead of just saying, “Here’s AI, go use it,” organizations need to ask who’s responsible for the tool.

I’m seeing what I’d call matrixed accountability evolve.

It used to be that IT was responsible for all the tools within an enterprise. But if you’re using AI agents in almost every line of work, some accountability needs to move closer to where the work happens.

IT may still be responsible for governance and strategy. But if you’re using a marketing agent, the marketing team should define the brand voice, establish the parameters and determine how campaigns are orchestrated.

At the individual level, you’re still responsible for your work, whether or not you’re using AI.

And yes, simply having a human in the loop can create false confidence. If accountability means checking every single data point, you may be focusing on the wrong things.

You should be looking at how testing is going, whether the system is meeting a quality bar and whether there are outliers or anomalies.

You want an empowered person who can set parameters and tune the system based on results. That’s very different from approving one data point after another.

What determines whether AI strengthens human judgment rather than undermines it?

So much depends on how intentionally it’s deployed.

We’ve seen how AI in hiring can amplify historical patterns and exclude qualified people. But used thoughtfully, AI can do the opposite.

Our CEO, Mark Benioff, uses AI in sales executive meetings. After executives discuss their territories, they’ll ask AI what questions haven’t been addressed or what perspectives haven’t been represented. They’re using it to challenge their thinking.

A venture capitalist told me about using AI in investment committee decisions. They establish criteria in advance, and AI can flag potential inconsistencies or biases during discussions.

Sometimes it’s difficult for people to raise those questions in the moment. AI can surface them so the team can have a different discussion.

Instead of treating AI as a silver bullet, organizations should ask how they want decisions to work, what AI is good at and what its shortcomings are.

It sounds like common sense because it is. But people aren’t necessarily thinking about AI that way yet.

As AI becomes more capable, has your thinking changed about which human qualities are irreplaceable?

My humility on this topic has grown over the years.

Because AI is modeled on human knowledge and behavior, it’s difficult to draw bright lines between what’s human and what’s AI.

Even judgment is a little blurry. When you hand a task to an agent, it’s making intermediate choices, such as which files to pull or which websites to research. It’s exercising something along the lines of judgment.

Still, as we hand increasingly sophisticated tasks to AI, human responsibility, wisdom and judgment become more important because the work becomes more consequential.

I stand by the notion that human judgment, creativity, connection and relationships become more important. I’ve just learned to resist rigid distinctions between what’s exclusively human and what’s exclusively AI.

Where have you seen AI make work more satisfying, rather than just more productive?

One example that struck me came from a visit to the California Department of Motor Vehicles.

I had gone to get my REAL ID. Employees discovered I worked for Salesforce and started telling me about their use of AI and automation to modernize services.

I’d expected to spend hours there, with everyone grumpy. Instead, everything was running like a tight machine. I was in and out in minutes.

But what really struck me was that everyone was happy. The employees were downright tearful.

It made me think about the virtuous cycle you can create with technology. When customers aren’t getting answers, they become frustrated and yell at employees. Those employees become stressed and are less likely to go above and beyond.

But when customers have good experiences, they treat employees better, who in turn provide better service.

Another example comes from my team at Salesforce, where we’re responsible for product accessibility.

Accessibility teams can accumulate large backlogs of bugs. AI makes it possible to identify and automatically remediate some issues before products ship.

That lets us focus on more complex challenges. Instead of concentrating on compliance, we can think about making the experience genuinely wonderful for people with disabilities.

You’re elevating the role.

You don’t get these outcomes automatically. Rather than just asking what AI is capable of, we should ask how it enables us to do things we might not even have dreamed of before.

We should be asking our teams that question, because that’s what makes AI feel like something they’re part of, rather than something that’s happening to them.

The latest AI trends, brought to you by experts

Get curated insights on the most important—and intriguing—AI news. Subscribe to our weekly Think newsletter. See the IBM Privacy Statement.