Gartner expects more than 40 percent of agentic AI projects to be canceled by the end of 2027. It blames escalating costs, unclear business value, and inadequate risk controls, not model quality. That tracks with what shows up in vendor conversations today: a partner who can demo an agent convincingly, and can't answer what happens to it in month six. The six-layer framework below is what a technical evaluator should check before any of that risk becomes yours.
Why “Ship Fast” Isn't a Delivery Methodology
Gartner's own estimate puts the number of agentic AI vendors with genuine agentic capability at around 130, out of the thousands currently claiming it. The rest is what the firm calls agent washing: existing chatbots, RPA scripts, and AI assistants relabeled for a hotter category. None of that shows up in a sales demo. A demo is built to clear exactly one bar: convincing a room the agent can complete the task in front of it. It says nothing about what happens when the model version underneath changes, when the agent meets a document format nobody tested against, or when a compliance officer asks for the audit trail behind a specific decision six months after launch.
The six layers below are the checklist for that gap: context, economics, governance, security, model strategy, and skills, moving from strategic fit through to what actually keeps running in production. Three of them are where most vendor conversations quietly fall apart, and they're the ones worth walking through here, along with the one question that cuts across all six and tends to expose which of them a partner has actually thought through. The other three get the same diagnostic treatment, question by question, in the full Guide. Underneath all six sits the same question the campaign keeps coming back to: who maintains this in year two? A partner who can't answer that for all six is the one a demo made look ready.
The Six-Layer Advisory Framework, Layer by Layer
Context: Most enterprise AI work is brownfield, not greenfield. That distinction changes almost everything downstream: dirty, siloed data; undocumented institutional knowledge in people's heads; systems never built with an API surface an agent could call. A partner who skips discovery and remediation and goes straight to agent development builds on whatever mess is already there, and the agent will expose it fast.
The Vetting Checkpoint: What is the state of the data that would feed this system? A partner who can't answer it specifically for your data hasn't done the discovery work yet.
Security and governance are worth their own sections below, because they're where a partner's real maturity shows up fastest. The other three layers raise questions just as pointed: what are all the costs, labor and non-labor, over time; what did it cost your last client to move to a new model generation; which roles will be on this at each phase, and what happens when one of them leaves. Those get the same layer-by-layer breakdown, question by question, in DataArt's AI Engagement Diligence Guide.
Security by Design: The Attack Surface a Demo Never Shows
The OWASP Top 10 for LLM Applications has ranked prompt injection as the number one risk for two consecutive editions. For an agentic system, indirect injection matters most: an agent reading a CRM record, an email, or a vendor API response that contains embedded malicious instructions, with no reliable way to tell legitimate content from an attack unless a sanitization layer already sits in front of it.
Privilege Escalation Is Why Least-Privilege Design Isn't Optional
Layered on top of prompt injection is privilege escalation. An agent granted broad system access becomes a high-value target in its own right. If manipulated, it can exercise permissions no single human user would hold, which is why scoping access tightly matters as much as catching the injection in the first place.
Relying on the orchestration framework alone to police inputs and outputs is a single point of failure. A production-grade architecture adds an independent guardrail layer outside the orchestration stack, the same defense-in-depth principle security teams already apply everywhere else, so a failure in one layer doesn't take down the whole system.
The Vetting Checkpoint: What does the threat model for this system look like, and who owns it? If nobody can name an owner, nobody is actually watching the attack surface.
The AI Governance Framework: Moving HITL Boundaries to HOTL
Most AI governance framework claims don't survive contact with a real audit. The one that does is a defined process for where an agent acts and where a human approves, not a slide from the kickoff deck. Human-in-the-loop, or HITL, is the synchronous version of that process: the agent is blocked at a defined checkpoint and can't proceed until a person reviews the proposed action. It fits high-stakes, low-frequency decisions: a pricing change above a threshold, a contract clause modification, a refund above policy limits. Human-on-the-loop, or HOTL, is asynchronous auditing instead: the agent acts on its own and logs every action to a queue, and a person can intervene, override, or roll back within a defined window. It fits high-volume, lower-stakes work, where the cost of blocking every action exceeds the risk the decision actually carries.
A credible partner delivers the HITL map as a project deliverable at the start of the engagement, not an afterthought, and designs it to evolve: which checkpoints have a clear path from HITL to HOTL as confidence in the agent's behavior grows, and which stay HITL permanently because of regulatory, contractual, or risk-management requirements. Manual QA doesn't scale against any of this, because it assumes the same input produces the same output every time, and a non-deterministic system doesn't work that way. Automated evaluation, LLM-as-a-judge, a separate model scoring outputs against defined quality criteria at production volume, has moved from optional enhancement to foundational requirement for exactly that reason.
The Vetting Checkpoint: What can the AI decide alone, and what needs a human? If you don't write that line down, it isn't a governance model yet; it's an assumption.
The AI SDLC Question That Exposes an Ad Hoc Partner
Every layer above only holds together inside a defined lifecycle a partner actually follows, not one that exists on a slide. Ask what their AI SDLC looks like: discovery, build, testing, deployment, operations, and where checkpoints and model change reviews sit inside each stage. A vague answer there is diagnostic on its own; it usually means the rest is being run informally too. The full lifecycle breakdown, stage by stage, is in the Guide.
Conclusion: Read the Framework Before You Read the Proposal
A proposal describes what a partner says they'll build. This framework describes what has to be true after they've built it, starting with the three layers above: a system your own team can run, a governance model that survives an audit, a security posture that doesn't treat the attack surface as someone else's problem. Economics, model strategy, and skills determine whether that system stays affordable and current, and they're where most vendor comparisons are won or lost.
Score a prospective partner against all six before comparing their pricing to anyone else's. A lower number attached to a thinner answer on governance or security isn't actually the cheaper option, and the layers you haven't pressure-tested yet are usually the ones that cost you later.
Three layers down, three to go, and the same six questions apply whether you're asking them yourself or handing them to someone who already has the answers. This framework is drawn from DataArt's AI Engagement Diligence Guide, which walks through all six in the same depth. DataArt's AI/ML Lab runs it as a Diligence Review against a specific engagement: your architecture, your cost model, your governance readiness, checked before you sign rather than discovered after.
