Your AI Agent Doesn’t Know What “Active Customer” Means

How many active customers does your company have? Opening DataArt’s webinar “Demystifying semantic context for agentic data products,” moderator Oleg Royz, who leads Data and AI Solution Consulting at DataArt, warned that it is probably the one your AI agent will get wrong.

Put five people in a room and you get five answers. Finance counts customers it has invoiced. Sales counts accounts it is selling to. Marketing counts anyone who opened an email. Product counts recent logins. Customer Success adds that Dave at Acme hasn’t logged in for six months, but they spoke to him on Tuesday. The failure is that nobody wrote down which answer belongs to which question, for which audience.

The session, the first in DataArt’s series on becoming an agentic enterprise, paired Paul McDonald, Principal Data Solutions Consultant, with Alexey Utkin, who leads DataArt’s Data and Analytics Lab. Royz’s premise: in the team’s experience, agentic programmes stall in the same place. The agent doesn’t know what the business means by its own words.

When agents remove the argument, they remove the control

Companies have survived wrong numbers for decades, so why worry now? McDonald answered with a story. Someone puts 742 active customers in a board pack. Finance says that doesn’t look right. Marketing says it’s over a thousand. Twenty minutes later, nobody is discussing the slide.

That argument was the control. An agent removes the argument. It reports 742 with confidence, and the next agent in the chain starts acting on it. The wrongness is the same; the scale and throughput are far greater.

Utkin added a second loss. There used to be a person behind every number, a data specialist a business user could call. Agents take that person out of the path, so failures turn silent. And people forgive people more readily than machines: a business that gets wrong answers from an agent disengages, and winning it back is hard.

A better model won’t close the gap. McDonald noted that text-to-SQL scores doubled in three years to 64.5% on dbt’s benchmark, and the wrong answers still look like the right ones. Pointed at raw tables, an agent gets about half its answers right. With definitions written down and owned, accuracy reported by early adopters rises to 85–98% on the same model.

Semantic context is knowledge about your business, and no vendor ships it

Most enterprises already have a Power BI semantic model, a catalogue and bronze, silver and gold layers. A live poll during the webinar showed how loosely the term is used, with answers ranging from BI models to the gold layer to the business glossary. All of it was built for people, each group working in its own corner, Utkin noted. An agent has no corner. Asked about active customers, it searches everywhere, finds three definitions of “active,” and picks one without saying which.

The harder problem is meaning that no system ever captured. The fiscal year starts in February. That table isn’t compared during a migration. Knowledge like this lives in chats, Jira tickets, wikis and people’s heads. Booking.com told Snowflake Summit it has fourteen versions of one reservations table, all with the same name.

Agents need more than shared metric definitions: relationships, business rules, instructions, domain boundaries and validation criteria, because a machine can’t answer “it depends.” McDonald drew the distinction sharply:

“A semantic layer tells you how to compute something. Semantic context tells an agent what things mean, what the rules are, and what it might do with the answers. The first is a technical artefact, and the second is an organisational asset.”

— Paul McDonald, Principal Data Solutions Consultant, DataArt

Asked in the Q&A about ontologies, Utkin called semantic context the umbrella term. In industries with public ontologies, such as pharma and finance, it may take the form of a knowledge graph; simpler domains can rely on instructions and metric views.

The business owns the meaning, and the data team encodes it

Who owns the definition of “active customer”? McDonald ruled out the data team, which usually gets handed it because it happens to be in the room. Data teams are custodians; the meaning belongs to the business, and someone needs the mandate to settle disputes. Southern Glazer’s told Databricks’ Data + AI Summit it distilled more than 20,000 reports into about 600 metrics, each with a named owner. Writing definitions down is the easy part, McDonald said; mandating them is the hard part.

Ownership doesn’t mean forcing one number. A subscriber who cancelled yesterday with 29 paid days left is active to Finance, a maybe to Marketing, and emphatically active to Retention. The job is recording which definition applies in which domain, so the agent never has to choose.

A first version for one domain takes about a quarter with a small team from both business and tech. A missing definition makes an agent stop, McDonald warned, while a stale one makes it lie convincingly. Mastercard’s context, built with Atlan, has to carry the line “Europe is on holiday in Q3.”

On lock-in, McDonald pushed back: isn’t moving definitions from BI into Snowflake the same lock-in, one rung up? Utkin’s answer: keep definitions in open formats, follow standards such as the Open Semantic Interchange, and generate platform-specific versions from there.

Ninety-five percent accuracy is a nightly habit

Royz followed one question up the ladder: “How many active customers did we have in Europe last quarter?” On raw tables, the agent guesses whether Europe means a sales territory or a billing address. Agreed metric definitions stop it inventing the maths. A business vocabulary and a set of business-approved answers take it further.

The best teams approaching 95% accuracy add an operational layer and re-test agents with real questions overnight. If the Europe answer comes back wrong on Monday because a customer moved its head office from London to Singapore and was counted in both regions, the team sees it and fixes it. When Marketing redefines “active” from 30 days to 60, the change is made in one place and every agent picks it up.

Once the questions that matter in a domain come back right and, more importantly, you’ve taught your agent to refuse the questions it cannot answer, that’s what gets you to 95% accuracy. From there, you need to keep monitoring, because models, data and semantic context all change.

Alexey Utkin

Head of Data and Analytics Lab, DataArt

Guardrails follow new-hire logic. Decide what an agent can see and do before it has access, McDonald advised, rather than during the incident review when someone asks who approved 14,000 laptops.

Start with the number your organisation already argues about

Reading a second poll on what holds teams back, Royz saw the same pattern: the hardest barriers are people and culture, not technology. That shapes the starting point:

• Find the pain first. Ask Finance and Sales for last month’s revenue, then “customer,” then “active customer.” The first number that makes someone ask “where did that come from?” is your starting point.
• Start in one domain. Teams that reach production begin with core tables and the questions that move the needle, giving agents the minimal context those questions need.
• Run the readiness test. Ask your own people what counts as an active customer. When they agree, and the agent answers the same way, you are close to production.
• Budget for a programme. Royz named three capabilities: people who know the business well enough to write it down, engineers who can build on fast-changing tooling, and a way to run it for good.

McDonald’s closing point:

Somebody has to write the onboarding manual. And the people who know what goes into it already work within your business. No vendor can supply them out of the box. That’s why this gets uncomfortable, and it’s also why it’s defensible once you’ve done it.

Paul McDonald

Principal Data Solutions Consultant, DataArt