MuleSoft’s Cost Management features can help you understand token consumption and identify opportunities to reduce the cost of your agentic workflows. For a capability overview, be sure to check out the documentation here.
Prior to the release of our cost management policies, we ran internal benchmarks to examine their cost and functional impacts. These benchmarks also inform the potential savings projections in our Cost Management workspace, helping you identify which policies to consider for your traffic.
Today, we are sharing how we configured those experiments, what the results show, and where we modified our approach.
Benchmark – cost reduction
We selected public, popular MCP servers with use cases relevant to customer workloads and clear opportunities for the policies to operate. The 18 workloads include collaboration tools, search services, databases, and documentation retrieval.
For the cost comparison, we use a golden set of tool requests/responses: a reference dataset that pairs test inputs with expected outcomes. Golden sets give repeated evaluations a consistent basis for checking results. We built ours by generating MCP responses from the selected servers’ tool output schemas and freezing those responses as test inputs.
Tasks supply the intent for those comparisons. Each task specifies a question, an expected tool, and a rubric describing required facts and claims the answer must avoid. We can define multiple test cases against the same golden set, asking different questions of the same fixed responses without rebuilding the dataset.
Tasks contain fuzzy and non-fuzzy prompts. Fuzzy prompts describe the objective in the terms a person might naturally use, leaving the agent to work out how to complete it. Non-fuzzy prompts add specifics, such as the expected tool or call sequence, that a typical user would not provide. Those details make the intended action explicit and help us judge the agent against a consistent task.
The controlled agent loop receives a fuzzy task prompt and the tool definitions exposed to the model. At each turn, the agent determines whether the task is complete or to probe for more information. If the agent chooses to take an action, it will make a tool call through the gateway, receive the policy-processed MCP response, and use that information to synthesize an answer.
The harness routes all MCP traffic through provisioned Omni Gateways configured with cost optimization policies and compares results against a no-policy baseline. The gateway captures the original and transformed payloads, enabling us to measure and compare their token counts.
For this cost comparison, we measure only the MCP tokens tracked by the gateway. Tokens consumed by the agent runtime, including its prompts, reasoning, and final answer, fall outside this measurement.
For functional correctness, we leverage a hybrid of deterministic checks and LLM-as-a-judge using each task’s rubric and non-fuzzy prompts. The judge scores the final answer against the task’s answer key and returns a pass or fail, a score, and an explanation. To improve judging consistency, we use multiple LLM judges and shuffle prompts across judging runs.
It is important to note that task outcomes and whole-conversation token usage remain separate from the gateway token comparison. The savings percentages below do not establish equivalent reductions in a complete agent workflow or prove unchanged answer quality.
Benchmark results
The table includes all 18 tested workloads and their reported projected token savings. Each policy column describes a separate treatment so the percentages cannot be directly added together.
From the results, it is clear that policy effectiveness depends heavily on the server’s tool catalog, response structure, and the information a task needs. A policy that removes substantial context from one workload can offer little benefit on another. The table below summarizes where each policy offers the clearest opportunity and when to avoid it.
Learnings
Our first attempt at a benchmark ran an agent through tasks against live MCP servers managed through Omni Gateway and used agent traces to compare token usage across each complete conversation. We wanted to capture how applying a policy affected the agent’s subsequent calls, including its tool choices and the number of turns needed to finish.
The hardest part was separating the policy’s effect from normal variation in agent behavior. We cached tool responses so repeated calls received the same data from the upstream server, normalized equivalent requests so minor formatting differences could reuse the same cached response, and fixed changing fields such as timestamps to keep the inputs consistent.
We also used a low model temperature and a fixed random seed to reduce variation in the model’s responses. More explicit prompts narrowed the choices the agent needed to make. Even with those controls, agents could choose different tools, produce different intermediate outputs, or take different numbers of turns. This variation affected both the baseline and policy-enabled runs, making statistically significant cost differences difficult to establish without running excessive amounts of trials.
That variation had a large impact on cost. Every additional model request carries prior conversation context, so a run with one extra turn can consume substantially more tokens even when the policy reduces the individual payloads.
We therefore moved the cost comparison to fixed MCP responses at the gateway boundary. The golden set holds the evidence constant while the policy changes its representation. Agent evaluation addresses the separate question of whether that representation supports the task.
What’s next
The next step is to make potential savings projections more specific to each customer’s servers and traffic. The results show why: the same policy can remove substantial context from one workload and almost none from another.
We want those projections to reflect the workloads customers actually run, helping them choose where an optimization deserves closer evaluation. Better cost estimates and functional validation together can support more informed policy decisions.
Cost is only one part of running agents in production. To see how Omni Gateway performs as the governance layer for LLM, RAG, and MCP traffic, download the MuleSoft Omni Gateway benchmarking guide. For regular updates on integration, APIs, and AI, subscribe to Technically Speaking, MuleSoft’s newsletter on LinkedIn.
