The hidden bill lives between the prompt and the outcome. The question for the enterprise is who owns the cost of getting the work done.
A developer asks an AI coding assistant to fix a bug. The request sounds simple: find the problem, write a patch, run a test. Behind the scenes, however, that single task can turn into many. The assistant scans the repository, checks documentation, writes code, and runs a test; when the test fails, it retrieves more context and tries again. To the developer, it is still one task. To the infrastructure beneath it, it is a growing pile of billable operations.
That is why a cheaper model does not automatically make the work cheaper. A smaller, specialist model may handle routine tasks perfectly well, but the savings disappear if it needs extra corrections. A token cap creates a similar risk if it stops the assistant just before it finds a usable fix. In either case, the AI bill may shrink even as the enterprise spends more time and money getting the work done.
Organizations focus on tokens because they are easy to count. Yet counting tokens is like running a factory by tracking electricity use: the meter is essential, but it cannot tell you whether the factory made anything worth buying. Inference, the work a model performs when used, turns each request into a recurring expense. Falling prices help, and growing usage can be good news when it produces valuable work. The problem begins when consumption grows faster than anyone can establish what that work is worth.
Choosing worthwhile pilots and funding them carefully is disciplined investment. Restricting those systems once they create value is a different decision, because a spending cap can cut productive capacity along with waste. For CIOs, CFOs, and business leaders, the economics therefore come down to three questions: What drives the cost? What does useful work actually cost? And who has the authority and evidence to improve those economics?
Return to the coding assistant. Within minutes, it may send the same repository context through several requests, repeatedly processing information that has barely changed. Prompt caching can avoid some of that repetition by reusing computation when the beginning of a prompt remains unchanged.
The catch is that the reuse window can be short. Anthropic’s default prompt cache lasts five minutes and refreshes on use, with a paid one-hour option. Reuse across days requires a more durable approach, such as a maintained retrieval index or stored summaries. In other words, caching can eliminate repetition, but it does not eliminate the architecture needed to support it.
Who controls that architecture matters. With a supplier’s API, the supplier owns the cache and the machines serving it, while the enterprise receives a discount when its prompts qualify. When an enterprise runs inference itself, it can also route requests to machines that already hold the relevant cache, freeing capacity for new work. The model has not become cheaper; the surrounding system has become more efficient.
There are limits to that reuse. Prompt caching reuses computation, not an earlier answer, and changed prompt content breaks the match from that point onward. If outdated material enters the prompt, the retrieval system needs attention; reusing previous answers through semantic caching creates a different freshness problem. Either way, current information and permissions still matter.
The same principle applies to model choice. A narrowly scoped fix may suit a small, specialized model, while a bug spanning several services may require something more capable. Model choice and reasoning effort are separate decisions, and more effort cannot reliably supply capabilities the model lacks. Retrieving only relevant code reduces unnecessary context; overnight jobs such as triaging a bug backlog or generating tests may qualify for discounted batch processing. Engineering determines how much repetition, idle capacity, and rework the workflow builds in.
Efficiency at the system level still does not tell you what successful work costs. In this workflow, producing a patch is only an intermediate step. The team needs an agreed definition of an accepted fix, including appropriate tests and developer review, before it can judge what the assistant actually delivered.
To illustrate the accounting, suppose 100 bug-fix tasks consume USD 30 in tokens across all calls and retries, including tool responses fed back into the model, but only 60 produce accepted fixes. That is 30 cents per task submitted and 50 cents per accepted fix. These hypothetical figures are not benchmarks. The first obscures unsuccessful tasks; the second excludes gateway, observability, and developer review costs. The remaining 40 bugs still need to be resolved.
To judge whether AI saves money, the team needs to compare the full cost of completing those 100 tasks with what it would spend resolving them through its existing process. Suppose that process costs an illustrative USD 100 per accepted fix for comparable work: USD 10,000 to resolve all 100 bugs. The AI-assisted process must include review, corrections, overhead, and resolution of the remaining bugs in its comparison with that baseline. At the same quality, it must clearly beat the existing cost. Higher cost requires demonstrated benefits such as faster delivery, lower risk, or capacity the team couldn’t otherwise provide.
Cost per useful outcome provides a lens for that comparison, rather than a universal KPI. Even with accepted fixes defined, the accounting takes work. A trace can record prompts, tool calls, and outputs, but it cannot automatically assign this coding workflow its fair share of the monthly bill for an AI gateway serving several use cases. Shared costs need an explicit allocation method and agreement about its limits. False precision is still false.
A lower token bill alone proves neither a saving nor a return.
Receiving a supplier discount is different from controlling the economics. The rate determines what the enterprise pays today; its ability to adapt determines its options when those terms change. Google’s Gemini Developer API pricing page offers a concrete example: for Gemini 3.8 Flash, standard input pricing rises from USD 0.75 to USD 1.50 per million tokens on January 1, 2027, while output pricing rises from USD 3.75 to USD 7.50. Listed batch and context-caching rates also double, including cached storage from USD 0.50 to USD 1 per million tokens per hour. The lower rates have an end date, so a budget built around them needs one too.
For the team operating the coding assistant, a pricing change raises a practical question: how hard would it be to switch to another model that can perform the same work? If the assistant is built in-house, tools may be written for one provider’s interface and prompts tuned to one model’s behavior. If it is a purchased product, model choice may sit largely with the vendor. A favorable rate can still create an expensive dependency, which is why procurement negotiates today’s price while architecture determines tomorrow’s options.
Modular design can preserve those integrations, permissions, and audit controls when the model changes. Even then, prompts may need retuning, provider-specific features may need replacing, and the team must rerun evaluations and regression tests on representative bug fixes. Portability reduces rebuilding, but it does not remove the need to prove that the replacement works.
The repository itself can narrow the options. Proprietary code and sensitive configuration data cannot leave an approved environment when security, contractual, residency, or sovereignty requirements prohibit it. A cheaper model does not override those restrictions. Within those boundaries, transfer charges, latency, and extra copies shape the economics. Intelligence must come to the data when those constraints demand it, through a suitable managed service, enterprise-operated infrastructure, or both.
Neither arrangement wins automatically. Self-hosting brings evaluation, security, and maintenance responsibilities, and underused GPUs can become expensive quickly. Preserving portability costs money too. Control is valuable not because it dictates one deployment model, but because it lets the enterprise make the better trade-off, including choosing a managed service when that is the economical answer.
The coding assistant’s apparent simplicity conceals decisions across the workflow: which model receives a bug report, which repository files it can access, where inference runs, and which tools and guardrails apply.
The complication is that those decisions rarely live in one place. A gateway may handle routing, an orchestration framework may manage tools, a retrieval system may supply context, security may set guardrails, and infrastructure teams may determine placement. Suppliers may control parts of each. Together, they create costs no single component can fully explain.
An AI operating model therefore has to connect those responsibilities: who defines success, who can change the workflow, and who judges the combined result. That requires decision rights and evidence across existing systems; buying another orchestration tool does not settle the ownership question. Nor does visibility alone. Seeing a routing policy doesn’t mean you can change it, and having the authority to change it doesn’t mean the change is worth making.
A practical first step is to examine a representative set of successful and failed bug-fix tasks. Identify repeated context, retries, and developer intervention, then combine that evidence with allocated infrastructure and support costs to establish the baseline. Test one change, such as routing narrowly scoped fixes to a smaller model, and compare accepted fixes and total expense at the same quality threshold.
Suppose the token bill falls but developers spend longer reviewing patches. Across the 60 accepted fixes in our example, just 10 additional minutes of review per fix would add 10 hours of developer work. The platform team records a saving while the engineering manager absorbs the extra work. The process owner, accountable for the outcome and budget, must be able to bring those findings together. IT supplies the implementation levers and technical evidence, finance accounts for cross-budget costs, and security checks whether necessary controls hold. Accepted fixes, total expense, and quality determine whether to keep or reverse the change.
In most enterprises, every individual AI cost question already has an owner: tokens, cloud bills, contracts, use-case results. What often lacks an owner is the total cost of getting the work done. Until someone owns that complete result, AI spending can keep growing while every team reports that its piece is under control.
[1] Anthropic, Prompt caching. Cache lifetime and reuse. Checked October 19, 2026. https://platform.claude.com/docs/en/build-with-claude/prompt-caching
[2] Google, Gemini Developer API pricing. Gemini 3.8 Flash Standard and Batch tables, including context caching and storage. Checked October 20, 2026. Rates are in USD. https://ai.google.dev/gemini-api/docs/pricing
Background materials: Arvind Krishna, Bending the AI Cost Curve, supplied internal draft; IBM Consulting, A tokenomics playbook for the enterprise, confidential client presentation. The coding workflow and all workflow figures are illustrative, not a reported client case or performance benchmark.
