What Is an AI Gateway? Routing, Cost Control, and Governance

An AI gateway is a centralized proxy layer between your applications or AI agents and large language models (LLMs) and tools they call. It routes requests, controls AI costs, enforces security, and observes usage, so you don't have to build those controls separately into every application. Which would quickly become impractical as your teams connect more models and deploy more agents.

Here's what an AI gateway does, how it differs from a traditional API gateway (and from LLM and MCP gateways), its capabilities and limitations, and how to tell whether you need one.

An AI gateway is a middle layer that routes every call from your application or AI agent to a large language model (LLM).

Thanks to it, you can access many different models as if they were one.

Each provider (OpenAI, Anthropic, Google, an open-source model, or your own self-hosted endpoint) has its own SDK, API keys, and request format. The gateway accepts one standard request, translates it for whichever provider you’re calling, and returns the response.

You add, swap, or retire models by changing gateway configuration, not application code.

And because every request flows through the AI gateway, you can apply cost, security, and logging controls for every model in one place.

How an AI gateway works

When you point your application or AI agent at the AI gateway’s base URL, every request moves through the same sequence:

• Authenticate and authorize: The gateway checks the caller’s key and role.
• Apply policy: It enforces token limits and budgets, runs guardrails such as PII redaction.
• Route: It picks the model or provider by cost, latency, or availability, and fails over to a backup if the first one is down. It also translates the request into the target provider’s format.
• Log: It records the model, token counts, latency, and outcome for AI spend tracking and debugging.

So a request that would have gone directly to an AI provider now passes through the gateway, which can cap its tokens, swap in a cheaper model, or return a cached answer before it reaches an LLM.

AI gateway capabilities

AI gateway routes model traffic, controls cost, protects and logs it, and governs who can use what.

Routing and failover

The AI gateway chooses which AI model handles each request, based on cost, latency, or capability.

If the chosen provider is down or rate-limited, the gateway automatically sends the request to another model.

So, your application keeps working during an outage, and you can spread loads across several providers at once.

Token-based rate limits and budgets

The AI gateway lets you set team, project, or user caps, and the gateway rejects requests once they reach the allocated token limit.

So, spending stays within budget, and no single team or project can use it up on its own.

Semantic caching

AI gateways' semantic caches store past answers and match new prompts by meaning. When a new prompt is close enough to one the gateway has already answered, it returns the saved response without calling the model.

On AI workloads with many similar requests, this lowers cost and response time.

Guardrails and data protection

The AI gateway inspects each prompt and response that passes through.

It removes personal or sensitive data before the request reaches the model provider, reduces the risk of prompt injections, and filters content you don’t want to send or receive.

The checks apply to every request, so protection doesn’t depend on each AI application getting it right.

Observability

The AI gateway records every request: which model ran, how many tokens it used, how long it took, and whether it failed.

That record of every AI interaction lets you attribute cost to a team, debug a slow or failing model, and see usage across all providers.

AI governance and access control

The AI gateway controls access to AI across your organization. It grants model access per your role settings, authenticates all callers, and records every call in an audit log, so you can review AI usage.

The AI gateway limits: latency and what it doesn't do

An AI gateway adds a small amount of latency. And it doesn't assess the output quality, which remains your responsibility.

How much latency a gateway adds

An AI gateway adds latency, which is the delay between sending a request and getting a response back, but it's relatively small for LLM traffic.

In an independent benchmark by Broadcom’s VMware Cloud Foundation team from June 2026, a tuned Envoy AI Gateway added about 2 ms per request at up to 224 concurrent users with routing and rate limiting but no content inspection (guardrails and semantic caching could add tens to hundreds of milliseconds).

AI gateway doesn’t control the prompt or output quality

An AI gateway governs the traffic, not the model’s output or the application logic around it.

AI gateways don’t decide what to ask a model, remember your conversation or where a task left off, or run an agent’s step-by-step logic.

And they don't assess the answer quality. So they can't guarantee that an answer is correct or safe.

In fact, implementing AI gateways can reduce quality under some circumstances. For example, if the gateway routes a request to a cheaper model to save cost, that model can return a worse answer.

So checking the output quality is still your job.

AI gateway vs. API gateway

An AI gateway and an API gateway work in a similar way but govern different kinds of traffic. An API gateway handles general API calls (REST or GraphQL between your clients and backend services), where each request is predictable. An AI gateway handles calls to a language model, measured in tokens and carrying free-form prompts.

Here are the main differences between AI gateways and API gateways:

• An API gateway routes requests to a backend service, while an AI gateway routes them to LLMs.
• An API gateway limits how many requests you make and measures bandwidth, whereas an AI gateway meters tokens. Tool request count is irrelevant for AI model usage tracking because one call can use a few tokens or hundreds of thousands.
• An API gateway serves a cached response only on an exact match, while an AI gateway can also cache by meaning and return a stored answer when a new prompt is close enough to an earlier one. Prompts are rarely identical, so without it, you would pay for model calls to answer the same questions again and again.
• An API gateway only authenticates callers and checks keys, while an AI gateway also inspects the prompts for sensitive data and hidden instructions. It stops sensitive data from leaving your network and blocks prompt injections.

Dimension

API gateway

AI gateway

Traffic

General HTTP, REST, GraphQL

Prompts, tokens, model-specific payloads

Rate limiting

By request count

By token consumption

Caching

Exact URL or header match

Exact match or semantic (matches similar prompts)

Security focus

Auth, OAuth, API keys

PII redaction, DLP, prompt-injection defense

Routing

To backend services

To models, by cost, latency, or capability

Cost tracking

Request counts and bandwidth

Token usage and model spend

"LLM gateway" and "MCP gateway" get used alongside "AI gateway," but they name narrower layers:

• An LLM gateway handles the model call itself (token metering, routing, and prompt filtering).
• An MCP gateway governs which tools and data sources an agent can reach once the model decides to act, using the Model Context Protocol (MCP).

In practice, these overlap. Several AI gateways, including LiteLLM, Portkey, Azure API Management, and Envoy AI Gateway, now add MCP governance, so the terms describe different layers of control.

How AI gateways manage AI agents and MCP traffic

The AI gateway controls the actions an AI agent can take with the tools and data it reaches through the Model Context Protocol (MCP).

AI agents work in loops: call a model, use a tool, feed the result back, and call again until the task is complete. To access external tools and resources, they use MCP, a standard protocol that lets AI services connect to third-party tools.

The AI gateway monitors every step in the process. It controls and monitors all traffic initiated by the agent, authorizes which tools and data sources the agent can access, and records all model and tool calls in an audit log.

As AI agents call multiple tools and models to complete tasks, the gateway has more tokens to budget, more latency to track, and a greater security surface to oversee.

Why AI gateways matter now

In a survey of 8,039 senior business leaders at organizations with 500+ employees, 83% said they plan to develop or deploy AI agents (Cisco AI Readiness Index 2025), and Gartner expects task-specific agents in 40% of enterprise applications by the end of 2026, up from under 5% in 2025.

The growing agentic AI adoption exacerbates old problems and adds new ones, some of which AI gateways help contain:

• AI costs run over. 79% of finance leaders who participated in the DoiT/Sapio 2026 survey overran their AI budget in the past year. What's worse, this often goes unnoticed because many businesses don't have reliable ways to track their AI spend.
• Companies use more than one AI model. 93% of enterprise CIOs surveyed by Dataiku and Harris Poll (2026) say different models perform better for different use cases. 81% of enterprises expect to run two or more model providers, and 55% have already switched LLM providers at least once.
• Provider outages are costly. Independent monitoring shows that the leading LLM providers are reliable: uptime of around 99.85-99.9% (May 2026) translates to roughly 9-13 hours of downtime a year. But when outages happen, they are expensive: two-thirds of 1,125 senior infrastructure leaders estimated that an hour of AI downtime would cost their organization over $100,000 (Cockroach Labs 2026).
• Prompts leak data and open organizations to attacks. Prompt injection is the top LLM security risk (OWASP, 2025), and the sensitive data flowing to AI tools is climbing fast: enterprise data sent to AI apps rose 93% in a year, and ChatGPT alone triggered 410 million data-loss-prevention violations (Zscaler ThreatLabz).

Do you need an AI gateway, and how do you choose one?

An AI gateway becomes essential when your organization is scaling AI product applications, deploying AI agents, or enabling different teams to use multiple AI models across diverse departments.

Several signals tell you the point has arrived:

• Provider keys, endpoints, and retry logic are copied or hardcoded across several application codebases.
• Monthly model bills spike unpredictably, or finance can’t attribute spend to a team or project.
• Customer-facing features hit rate-limit or downtime errors whenever one provider struggles.
• Applications send uninspected prompts that may carry personal data, source code, or other sensitive information to external models.
• Different teams adopt competing providers (OpenAI, Anthropic, Google, or self-hosted models), each with its own integration code.
• You’re deploying autonomous agents or MCP tool integrations that make multi-step calls against internal systems.
• Security, IT, or compliance teams have no overview of which models are running, who calls them, or whether outputs meet regulatory requirements.

On the flip side, an AI gateway isn't necessary when building a single prototype, running a low-volume side project, or calling only one model endpoint with minimal security requirements.

Self-host or buy a managed AI gateway

When you deploy an AI gateway, you can choose between a self-hosted or managed solution.

With self-hosted, you run an open-source gateway like LiteLLM, Portkey, or Apache APISIX on your own infrastructure, whether that's a private cloud, a Kubernetes cluster, or on-premises.

Self-hosted AI gateways are a good choice when:

• Data residency or compliance is strict. The gateway, its logs, and its keys stay inside infrastructure you own.
• You need deep customization. Your team wants custom plugins, routing logic, or key and budget configuration beyond what a managed service offers.
• You have the DevOps capacity. Your team can patch, scale, and run the gateway and its state stores, such as PostgreSQL, Redis, or etcd.

In managed solutions, like Cloudflare AI Gateway, Azure API Management, or Amazon Bedrock AgentCore Gateway, the provider runs the gateway as a service.

Fully managed AI gateways are a better fit when:

• You want it running fast with little maintenance. The provider handles scaling, patching, and uptime.
• Your stack lives in one cloud. You get native identity and access management, logging, and security from AWS or Azure out of the box.
• You want your engineers to focus on the product. Their time goes to developing AI features, not operating proxy infrastructure.

Consideration

Self-Hosted AI Gateway

Managed (SaaS / Cloud) AI Gateway

Data Control & Privacy

Complete ownership; traffic never leaves your network perimeter.

Data transits provider-managed infrastructure.

Operational Burden

High — DevOps handles scaling, security patches, and state databases (PostgreSQL/Redis).

Low — Provider handles infrastructure, uptime, and scaling.

Deployment Speed

Requires infrastructure provisioning, testing, and configuration.

Fast — Turnkey integration via simple endpoint or cloud console.

Integrations & Access

Ideal for developer-centric teams requiring custom open-source code.

Ideal for organization-wide adoption and native cloud IAM stacks

Frequently asked questions

What is a generative AI gateway?

A generative AI gateway is the same thing as an AI gateway; the name just emphasizes traffic to generative AI models. Some vendors, like AWS, use “generative AI gateway” for a control layer that governs how applications consume large language models.

You can treat the two terms as interchangeable.

What is AI model routing?

AI model routing is how an AI gateway sends each request to the most suitable model based on cost, latency, or capability. It lets one application reach many providers through a single endpoint, without any routing logic built into the application itself.

What’s the difference between an AI gateway and an MCP gateway?

An AI gateway governs traffic between your applications and AI models: prompts, tokens, routing, and cost. An MCP gateway governs which tools and data sources the agent or client can reach through the Model Context Protocol.

Many AI gateways include MCP governance, so the two sit in one product.

What is the best open-source AI gateway?

There is no single best open-source AI gateway. To pick the right one, compare them on the providers, plugins, and controls you need.

LiteLLM is a widely used proxy with broad provider coverage; Kong AI Gateway suits teams already running Kong, while Apache APISIX fits teams that want one open-source gateway for both API and AI traffic.