How Does an LLM Gateway Work? A Practical Guide

An LLM gateway sits between your applications and the model providers you connect through it, giving you one place to manage credentials and track usage.

“At the moment, it’s a s**t show,” one engineer wrote on Reddit in r/LLMDevs in September 2026.

“Everyone has their own set of keys for Claude, OpenAI, AWS Bedrock, and all the other inference providers we use.”

BigGo’s coverage of llm-proxy-tk, a lightweight FastAPI gateway, likens the setup to turning LLMs into “corporate tap water,” with the gateway as the building’s master valve.

But before you put that much trust in the plumbing, let’s look at what you’re installing.

What is an LLM gateway?

An LLM gateway is a reverse proxy for model calls. Your applications send requests to it instead of directly to OpenAI, Anthropic, Google, or whichever LLM providers you use. The gateway forwards those requests, applying your rules on the way through.

There's an API for every LLM provider: it's got its own request shape, its own authentication, and its own way of returning errors. Each also has separate rate limits and failure modes. Now, if there's no shared layer in front of them, each application that calls a provider does everything itself, and does it a little differently.

The tap-water analogy only gets you so far, but it gets you this far—providers get plumbed in once, at the gateway. From then on, each application gets its own metered connection, so you can see who’s using what and shut off any single line without touching the others.

LLM, AI, MCP, and API gateways: What’s the difference?

The gateway

What does it sit in front of?

What does it decide?

The limits of its view

API gateway

Your services and other backend APIs

Is this client authenticated, and is it under its rate limit?

Can inspect request bodies, but token accounting and other model-specific controls need added support.

LLM gateway

Model providers

Which model gets this request, who pays for it, and did it come back?

Sees tool calls proposed in model responses, but not their execution unless that traffic also passes through it.

MCP gateway

Tools and data sources an agent connects to through MCP

Can this agent call this tool, with these arguments?

Doesn’t automatically see the model call that produced the tool request.

AI gateway

Whatever the vendor has bundled: models, potentially tools and agent-to-agent traffic

A combination of the above, depending on the product

Depends on which connections it handles.

Follow one request from your application to the model and back.

• The request arrives at the gateway: Your application sends a normal-looking chat request, often in an OpenAI-compatible format, to the gateway’s endpoint. From the application’s point of view, this is the address it calls to reach the model.
• The gateway checks who’s asking and what they’re allowed to do: Before forwarding the request, it validates the credentials. Often, that’s a virtual key issued to a team or product. If you’ve wired in input guardrails, prompts can also be scanned for sensitive data or attempts to override the model’s instructions.
• The gateway checks whether it already has the answer: If caching is enabled, an eligible matching request can be answered from a saved response without another provider call. Some gateways also offer semantic caching, which looks for requests judged similar enough rather than identical. That can suit FAQ-style traffic.
• The gateway chooses where the request goes: The request might name a specific model or an alias. The gateway looks up the matching route in your configuration. You can also configure a backup route. If provider A times out, the gateway can try provider B.
• The gateway prepares the call and returns the response: A gateway converts your request into the chosen provider's format and authenticates the outgoing call. As soon as the response is received, it's converted into the format your application requires.
• The gateway records what happened: The gateway records details including who made the call, which model served it, token counts, latency, and cost when logging is enabled. This is the log the Reddit engineer from the introduction didn’t have.

The 4 things you need to know before you put a gateway into production

Everything in the section above is possible because the gateway sees every request routed through it. That's the pitch. But it's also a description of the attack surface and the vendor dependency you have to plan for.

The gateway is now the most valuable box in your AI stack

The gateway centralizes access. And therefore, it also puts a lot of credentials within reach of one service. Microsoft’s security team lists what a LiteLLM deployment may hold or retrieve: provider API keys, its own master key, virtual-key records, database connection strings, routing settings, and tenant policies.

In October 2026, Check Point Research disclosed two high-risk vulnerabilities in LiteLLM:

• CVE-2026-35029 let authenticated users, including read-only ones, reach administrative functions and take over the instance.
• CVE-2026-59823 let users redirect requests to a server they controlled and capture provider keys.

Both were fixed in version 1.83.9. An application that uses the gateway shouldn’t be able to change its settings.

Your guardrails need a failure policy

An October 4, 2026 review by Pillar Security documents third-party guardrail integrations across major gateways, including LiteLLM, Kong, TrueFoundry, Portkey, Azure API Management, and Apigee. (Pillar sells guardrails, so bear that in mind when reading its recommendations.)

A hook is a point in the request path, typically before the prompt goes to the model or after the response comes back, where the gateway can hand the content to an external scanner for a verdict. A blocking check waits for that verdict before letting traffic continue. Portkey’s documentation calls these before_request_hooks and after_request_hooks; TrueFoundry runs four, adding checks before and after MCP tool calls.

Now, two things to check:

• What happens if the scanner itself fails? When you block a request, you stop unchecked traffic but may interrupt your application. Let it through, and your application can continue without that check. The failure log shows you afterward that the check was missing, which is useful, but not the same as having run it.
• Can you replace the scanner? If another provider isn’t supported, you may have to update your application or use another proxy to connect.

TrueFoundry's MCP hooks extend the gateway's reach from model requests to tool calls. But it can only inspect traffic that passes through it. A coding agent running a command on a developer's machine may be outside that view.

Perplexity’s open-source Numbat works on that side. Numbat connects to supported agents through local hooks and can inspect planned actions, like shell commands or file writes, before they run. The blocking feature is optional and is turned off by default. The gateway sees the model traffic routed through it. Numbat adds visibility into what the agent does with the answer.

You need to know where the gateway’s control ends

Forkast News identifies three lineages approaching the gateway from different directions:

• LLM-focused gateways: LiteLLM and Portkey started with model calls; translating between providers, routing requests, and tracking tokens. They’re now adding controls for tool calls.
• API management platforms: Kong, Google’s Apigee, and Azure API Management started with authentication, permissions, and traffic management, then added support for models and AI-specific rules.
• Agent-native infrastructure: agentgateway, for instance, was built around agents communicating with models, tools, and other agents, including through MCP and A2A.

The sorting question is what you’re routing:

• Model calls only? An LLM-first gateway is a good starting point.
• Model calls inside an enterprise already full of API rules? Start with the platform you have.
• Agents calling tools calling models? Check that those connections are covered too.

You introduce a shared point of failure

Say your support chatbot uses OpenAI and your research tool uses Anthropic—both connect through the same gateway. But if that gateway stops working, both tools lose access to their models, even if OpenAI and Anthropic are running normally.

This has really happened. Cloudflare's AI Gateway's error rate peaked at 97% during its June 12, 2025 outage. The gateway relied on Cloudflare's Workers KV storage service for settings, and that service was failing.

If you run the gateway yourself, keep another copy running that can take over if one fails. If someone else hosts your gateway, ask how they keep it running during failures and what happens to your requests during recovery.

The key features to look for in LLM gateways

You’re going to give the gateway access to everyone’s credentials, so you should be demanding about what it does in return.

• One API in front of multiple providers: Your code speaks one format, often an OpenAI-compatible one, and the gateway translates. LLM providers meet it halfway. For example, Perplexity’s Agent API supports OpenAI’s Responses API interface, so teams can reuse existing OpenAI client libraries.
• Virtual keys with rules attached: Each team or product gets its own key, with rules governing allowed models, request limits, and spending.
• Routing and fallback: The gateway sends requests to models based on your settings, which might favour cost, speed, or a particular provider. If a call fails, it can try another. Perplexity’s Agent API lets you list up to five models, including models from different providers, in your preferred order.
• Caching: A saved response can spare you another provider call. Some gateways match identical requests; others can also match requests with similar meanings. Check whether cached responses can be separated by user or team, and whether you can turn caching off for personalized or time-sensitive answers.
• Streaming and consistent errors: The responses arrive as they’re generated, and provider errors get translated into a format your application can handle. Check what happens when you add AI guardrails.
• Logging and cost attribution: The gateway records who made a call, which model served it, how many tokens it used, how long it took, and its estimated cost. Check whether logs include prompts and responses and how long they’re kept.
• Guardrail hooks: These are points in the request path where the gateway can send content to an external scanner for a verdict. They commonly sit before a model request or after its response. Check whether requests stop or continue when the scanner fails, and whether you can connect a different scanner.
• An admin interface you can lock down: The gateway manages access to provider credentials and the rules everyone uses. Check that you can restrict admin access separately from application traffic and that read-only users cannot change settings. The master valve needs a lock.

What tools can you use to build your LLM gateway?

The feature lists overlap across the market. In one Reddit thread on internal model access, the poster asked: which gateway won't require a dedicated engineer?

That's the right question in our opinion, and it has four answers, depending on how much of the work you’re willing to own.

Write it yourself

A minimal gateway handles a short list of jobs—accept requests, check the caller’s key, add the provider credential, forward the call, and log the result. But supporting different provider formats adds work. As does keeping the whole thing running.

If you already run Envoy, its external-processing filter lets you connect a service that inspects and changes requests and responses. You still have to build and run that service.

In a r/LLMDevs thread on whether a gateway cuts costs, u/QuanTradin said logging tokens and a cache-hit flag on every call helped them find two jobs paying full price for identical prompts.

“...fixing that was worth more than any routing layer would have been.”

That’s a reported experience, and a good place to start. Find out what you’re paying for before building elaborate ways to route it.

Self-host an open-source gateway

LiteLLM is a common choice here: OpenAI-compatible, supports a wide range of providers, and has a large community. Bifrost is an alternative, built for low per-request overhead. Helicone is worth a look if observability tops your feature list.

LiteLLM, like any product, has limits for some teams. In a r/aiagents thread on alternatives to building your own, u/Own_Bar_920 said LiteLLM handled routing fine and fell short for them on prompt versioning across teams and continuous evals when swapping models, which pushed them to a separate tool.

Those go beyond basic gateway routing, which is the point—if routing is the main need, LiteLLM held up for them.

You get a ready-made feature list, with some features depending on the edition, and keep the gateway inside your network. You also own patching, which the Check Point disclosures earlier made concrete, and the gateway’s availability, which the single-point-of-failure section did.

But budget for someone to be on the hook for it. One worry in a r/LLMDevs thread on gateways for internal access was maintenance. u/Informal-Writer9685 wanted something that “doesn't require a dedicated engineer to maintain” and asked whether LiteLLM’s reported upgrade instability was a real problem at their scale or a production-only concern.

Use a managed gateway

Cloudflare AI Gateway, Vercel AI Gateway, OpenRouter, and Portkey’s hosted gateway run the service for you. For a compatible application, getting the first request through may take little more than a URL change and an API key.

The trade-off here is that every prompt now passes through a third-party before reaching the provider. That's one more data processor for your compliance paperwork, and the gateway's own outage is a vendor's incident—it's not yours to fix.

For example, on February 17 and 19, 2026, OpenRouter experienced outages lasting 38 and 35 minutes, respectively, after a caching dependency failed. Some customers received “User not found” errors and spent time debugging perfectly valid API keys. The fault was in the gateway’s infrastructure.

But to its credit, OpenRouter published a detailed postmortem and added protections to stop a cache failure from overwhelming its database. A managed gateway like OpenRouter can simplify access, but as u/QuanTradin points out on r/LLMDevs, consolidating the bills doesn’t automatically reduce the spend.

Extend the API gateway you already have

If Kong, Apigee, or Azure API Management already fronts your services, each has AI-specific capabilities: Kong AI Gateway, Apigee’s AI gateway policies, and Azure API Management’s AI gateway. You can build on the authentication, network rules, and operating procedures you already have, adding model controls through plugins or policies.

This can be the path of least organizational resistance in a large company. The AI features are younger than the standalone gateways’. Make sure to test token-based rate limiting and streaming through the full set of plugins or policies you plan to use.

So by all means, install the master valve—the almighty gateway. The engineer from the introduction has a point, though. A company’s model access shouldn’t depend on whoever happens to have the right key in a drawer.

A gateway fixes that by putting those keys under central control, which rather raises the stakes of leaving the drawer open. You have to know whose drawer it is.

LLM gateway FAQs

Is the LLM gateway free?

If you mean the category, some LLM gateways have free versions.

LLM gateways range from free open-source software to paid plans starting around $50 a month, with custom pricing for enterprise deployments. And some charge a percentage instead; for example, OpenRouter’s Standard plan adds 5.5% to credit purchases.

Which LLM gateway is the best?

What’s the work you’re willing to own?

• Take a look at Bifrost or Helicone if you want to self-host and control the configuration.
• A managed option like OpenRouter or Vercel AI Gateway suits teams that want someone else to run the service.
• If you already use Kong or Azure API Management, check their AI capabilities first.

Test your shortlist with your own requests, including what happens when a provider fails.

Does AWS have an LLM gateway?

Yes. Amazon Bedrock AgentCore Gateway supports model calls through what AWS calls “inference targets.” These let applications use one endpoint to reach providers including Amazon Bedrock, OpenAI, and Anthropic, with the gateway handling provider credentials and routing.