Home
Blog
AI Governance & Security
Brad Griffith
Nov 11, 2026
You route LLM calls by cost, latency, or task without losing governance by running the router inside the gateway that already enforces your policy. Routing then chooses among approved models, counts against the same budget, and logs to the same audit trail. Workato Model Gateway takes this approach: routes resolve to a provider and model through rules, and rate limits, token limits, and key policy apply to every route.
The failure mode to avoid is a router bolted on beside the gateway. It saves tokens, but it also creates a second path that no policy governs.
Why does routing usually weaken governance?
Routing weakens governance when it runs outside the policy layer. A standalone router picks a model, but the approved-model list, the team’s budget, and the audit log all belong to a different system. So the router can send a request to a model security never reviewed, and finance only sees the spend on the provider invoice. The fix is architectural: put the routing decision and the policy decision in the same place, and a request can’t be routed to something it isn’t allowed to use.
What should a governed routing design include?
A governed routing design includes five elements: an approved-model list, identity on every call, per-route limits, usage metering, and an audit record of the routing decision. Workato Model Gateway covers each of them. Administrators set the approved models and assign virtual keys at the user or group level, and rate and token limits attach per team, project, or automation. Token usage is reported by type, model, provider, and route, and each call is captured for audit and observability.
How do you route by cost?
Match task difficulty to model price, and cap spend per team. Start with the workloads that dominate your token bill and ask whether each one really needs the most capable model. Workato’s rule-based routes can evaluate conditions per request, and in one Workato demonstration a rule selected a model using user balance, quality, and cost.
Token usage reporting by input, output, cache read, and cache write shows where spend concentrates, since cached reads and fresh input tokens price differently. Workato’s AI Control Plane adds spend by team, model, and cost center, with budgets that alert early.
How do you route by latency?
Send time-sensitive calls to faster models and let batch work use slower, cheaper ones. A customer-facing chat turn and an overnight summarization job shouldn’t share a route. Define separate routes for interactive and batch workloads, tag keys by priority, and set rate limits so batch traffic can’t crowd out interactive traffic. Workato supports key tags by department, sensitivity, or priority, so a rule can match on priority. Then measure real latency on your own traffic, because provider latency varies with load.
How do you route by task?
Classify the request and map each class to a model. Extraction, classification, and summarization often run well on smaller models, while multi-step reasoning and tool use often need larger ones. Workato runs AIRO on its own model gateway and switches models by task complexity, with budgets and quotas per period. That is a first-party example of task-based routing under budget control.
How do you keep one budget across every route?
Make the limit a property of the key or team, not of the route. When limits attach to the caller, a request that falls over to a cheaper route still counts against the same allowance. Workato attaches rate limit and token limit policies per team, project, or automation. Because the gateway meters every call, a change in routing shows up in the usage report right away, and the team that moved traffic can see what the change did.
How do you handle failover without breaking policy?
Define the fallback model in advance and confirm it’s approved. A fallback to an unreviewed model during an outage is a policy violation nobody chose. With Workato, failover as a built-in Model Gateway capability. Pair every route with a named fallback from the approved list, and test the failover path before an incident forces it.
How do you test a routing change?
Test a routing change against a fixed prompt set before you move traffic. Compare quality, token usage, and error rate between the current route and the proposed one. Workato Model Gateway reports success and error views by route, which makes the comparison measurable.
Move traffic in steps and keep the old route available for rollback. Because applications reference a route ID, rollback is a configuration edit.
Summary: Routing LLM Calls With Governance
Run routing inside the gateway so every routed call stays approved, metered, and logged. Route by cost, latency, or task using rules on tags, keys, and parameters, and keep the approved-model list, identity, and limits on the gateway. Attach budgets to teams and keys so failover cannot bypass them, name a fallback from the approved list for every route, and test changes against a fixed prompt set.
FAQ
How do I route LLM calls by cost?
Match task difficulty to model price and cap spend per team. Review token usage by model and route to find the biggest workloads.
Can routing and governance live in the same product?
Yes. Workato Model Gateway performs routing and policy together, so routed calls stay under one approved-model list and one audit trail.
What is the difference between routing and failover?
Routing chooses a model by rules. Failover switches to another model when the first fails.
Does routing affect output quality?
It can. Smaller models handle some tasks well and others poorly. Test with a fixed prompt set.
Who owns routing rules?
Platform or AI engineering writes the rules. Security approves the model list. Finance sets budgets.
About the Author
More on AI Governance & Security
Discover the latest product updates and features
What Is an AI Control Plane?
Nov 10, 2026
Which Agentic Workflow Steps Should Be Deterministic?
Nov 5, 2026
What Is an AI Gateway?
Nov 5, 2026
Table of Contents
