Why Most AI Vendor Conversations Never Get Past the Demo
An AI engagement is a budget line a CFO defends, a risk profile a board expects managed, and a competitive bet a CEO or business unit leader will answer for. A 2025 MIT study of enterprise generative AI deployments found that 95 percent of pilots produced no measurable return, and only a small fraction ever reached production at scale. Every executive sponsoring AI faces the same pressure behind that number: show a return without quietly taking on new technical debt, regulatory exposure, or operational risk. DataArt built its AI Engagement Diligence Guide around that pressure, and it shapes every question a services partner should expect, and volunteer, before anyone signs.
You're Not Buying AI. You're Buying the Ability to Run It Without the Vendor in the Room
Take that seriously, and it changes what a credible partner must demonstrate before you sign anything: a realistic total cost of ownership, evidence of the value of similar work delivered, a clear governance and risk model, and a plan for who maintains what gets built once the engagement ends. Call it AI operational sovereignty: the ability to run a system, monitor it, improve it, and explain it without the vendor in the room.
Hold a partner to that standard and a few specific demands follow. Pace and cost come first. AI changes the old speed-quality-cost tradeoff without erasing it: speed comes first, quality follows with iteration, and cost efficiency compounds only once the system matures, so a partner promising all three from day one sets up an awkward conversation at your first quarterly review.
What gets left behind matters just as much: look for an agentic framework your own team can run and extend, with the value shifting from hours delivered to intellectual property transferred, plus a way to show when the system's outputs can and can't be trusted, not just whether the servers are up. They should also keep both managed services and upskilling on the table from the start, since forcing that choice early, before your team has a basis for deciding, usually means the partner is optimizing for their own retainer.
“This Is Just Ordinary Vendor Due Diligence”
That's a fair objection, and worth answering directly, because plenty of vendor scorecards already ask about cost, timeline, and references. What changes with agentic AI is what you're evaluating. A model that recommends produces a suggestion someone can ignore; an agent that acts, adjusting a price, approving a transaction, updating a customer record, produces a consequence. Once AI moves from advising to acting, the question of who is accountable when it acts stops being a procurement formality and becomes an operational control.
That is also where most generic AI-vendor content runs out of road. Search results for “questions to ask an AI vendor” are dominated by horizontal checklists written for any enterprise software purchase, or by compliance checklists written for legal counsel. Almost none of it is built for evaluating an agentic AI engineering partner specifically. That's a harder question than what the tool does; it's who runs it, who is accountable when it acts, and what happens after the team that built it moves on.
The Questions That Separate a Partner From a Vendor
A handful of questions do most of the work of separating a genuine partner from a project vendor selling on demo-level promises. Start with the one that matters most: who maintains this in year two? Most engagements end at go-live, and the real difference shows up afterward, because most service firms' commercial models reward building, not sustaining.
- How is oversight structured between an agent's decisions and human approval? The answer should be a specific, evolving map, not a general assurance that “a human is always in the loop.”
- Is security designed in from the start, or checked at go-live? Agentic systems create a partially emergent attack surface that you can't fully map at design time, so security by design becomes a load-bearing requirement, not a footnote.
Then ask how they manage model change over time. Major model generations now arrive roughly every six months, so a partner who hardcodes a single model into your production system is building against today's ceiling instead of tomorrow's floor. Push further and ask whether a real, mature AI delivery lifecycle sits behind that answer, or an ad hoc process running on your budget. Two more questions belong on the list, and proposals tend to skip both: what it will cost to own the system, and what value similar work delivered for other clients. Both are covered below.
What Governance Has to Produce Before Launch
Regulatory frameworks are catching up quickly. The EU AI Act, SOC 2, and ISO 27001 are expanding to cover AI systems, and auditors are starting to ask questions most firms aren't ready to answer yet. A human-in-the-loop boundary map has to withstand that kind of scrutiny, so it can't be pulled together for the kickoff deck. It needs to exist as a project deliverable: an auditable record of which decisions an agent can make alone today, which ones still require a person, and how that boundary evolves as confidence in the system grows, rather than loosening by accident under delivery pressure.
Total Cost of Ownership and Proof of Value Belong in the Diligence
Total cost of ownership. Cost should come up in the first conversation. A token rate, a license fee, a first-month price, or an initial commitment says little about what a system costs to own, so ask for total cost of ownership and the expected run rate at months three to six and six to twelve, with the growth assumptions behind them.
Hidden costs. Most AI business cases account for labor but undercount or ignore the non-labor costs: tokens, compute, embedding pipelines that scale non-linearly once an agentic workflow reaches production volume. A partner who cannot walk you through that full picture before you sign is asking you to discover it after you have already committed the budget. DataArt's AI Engagement Diligence Guide takes the cost layer further, from token and compute modeling to the FinOps practice that has to sit behind it.
Pricing and exposure. How a partner answers is as telling as the numbers. A flat price or flat structure with no model behind it is a yellow flag at best, and often a red one: either the partner doesn’t know what else comes with deploying the system, or you will find out and absorb it. Nobody can see far into the long run, since enterprise generative AI has been in production for only a few years, so ask for a clear indication of financial exposure at the start.
Proof of value. Value deserves the same scrutiny. IBM's 2025 survey of 2,000 CEOs found that only 25 percent of AI initiatives had delivered the ROI they were expected to. Security and compliance carry real weight, but a partner should also show what its work produced: efficiency gains, faster delivery, better data quality, or capabilities a client could not ship before. The useful question is what financial and strategic value a similar engagement created, which says more than whether it added one more AI or data platform. Ask for case studies and comparable engagements on both counts: a partner who works with you on cost and value, rather than acting as a transactional supplier, will have a track record to show, and you will not be the first client to ask these questions.
Run the Diligence Before You Sign
This isn't an argument for slowing down. Answer the right questions first, and you can move just as fast; you won't find gaps at the first board review, the first security incident, or the first quarter the promised cost curve doesn't show up. Come back to the maintenance question above all: a vague answer there usually means the governance, the cost model, and the security posture upstream of it are being run informally too.
DataArt's AI/ML Lab built that framework to be run against a specific engagement, not read once and filed away. Schedule a Diligence Review to benchmark a prospective partner's architecture, cost model, and governance readiness before you sign, or to pressure-test the AI initiative you already have underway.
