Dev48
Language
  • About
  • Services
  • Industries
  • Technologies
  • Articles
  • Contacts
Book a call
    Home/Articles/Endpoint candidates evaluate and promote inference endpoints
Dev48
info@dev48.org+375 29 320-40-96WhatsApp@isky_by

© 2026 · All rights reserved.

Endpoint Candidates: evaluate and promote inference endpoints

Source: Modal•October 12, 2026 at 05:59 PM

Endpoint Candidates: evaluate and promote inference endpoints

Test and promote inference endpoints to evaluate quality, latency, throughput, and cost.

Latency, throughput, and cost optimization are at the core of modern inference engineering. Without testing and measurement, these metrics cannot be reliably improved.

We’ve been focused on inference performance for some time now, and we’re building several new tools to help you optimize your own inference servers–starting with Endpoint Candidates.

Endpoint Candidates allow you to stage alternative deployments for your Dedicated Endpoints, so you can evaluate the impact on quality, latency, throughput, and cost. Each Candidate can mirror production, handling a sample of production traffic while Modal runs the underlying infrastructure. When you’re ready, promote your Candidate to production from the dashboard.

With Candidates, you can test new model checkpoints–or even entirely different models–without rolling your own traffic-mirroring and deployment infrastructure. Endpoint Candidates are generally available for all Modal workspaces.

Getting started

To create a new Candidate, start with an existing Dedicated Endpoint in your Modal environment. Select Create candidate to choose an open model from Modal’s catalog or bring your own weights from a Hugging Face repo or a Modal Volume.

Once your Candidate is ready, configure your traffic sample to start mirroring production traffic to it. Production continues to handle every original request and return the response your users see.

Use the dashboard to inspect the Candidate’s inference metrics, including latency and throughput, and run performance benchmarks to understand its behavior under load. Each Candidate also has its own URL, so you can run your existing evaluations against it to assess output quality.

When you’re ready, promote the Candidate from the dashboard. Modal starts a production rollout at the existing Endpoint URL. Your other Candidates remain available, so you can continue testing them independently.

Endpoint Candidates use standard Modal compute pricing, including compute used for mirrored requests.

Keeping production moving

One of the challenging parts of building traffic mirroring was keeping slow mirrors from holding up production. When several Candidates receive copies of a large request, they may read the data at different speeds. A proxy can end up buffering substantial amounts of data while waiting for the slower Candidates.

To address this, we extended our Pingora-based proxy to fork each request stream into a separate, bounded buffer for every Candidate receiving a copy. Each mirror reads independently: if a new chunk arrives and a buffer is full, the proxy cancels that mirrored request while production and the other mirrors continue.

But a bounded buffer for each mirror still leaves a problem: a busy proxy may be mirroring many requests at once. So we also cap the body data retained by mirrors within each proxy process. They share request-body chunks, so each chunk counts only once against this budget.

Removing a chunk from a buffer doesn’t necessarily free its memory, since a network write may still be holding it. We keep counting that chunk until the last mirror releases it. If adding the next chunk would exceed the budget, the proxy cancels that request’s mirrors while the original request continues.

Mirroring is best-effort by design: mirrored requests may be dropped under pressure, so production doesn't have to wait for a slow or failing Candidate.

What’s next

We’re expanding how you can create and evaluate Candidates, from more control over compute placement and serving recipes to richer metrics views for comparing deployments.

We’re working to automate more of that experimentation, too. When we introduced Auto Endpoints, we described our work on automating inference optimization. Candidates give those optimizations a place to prove themselves: a way for Modal to test promising changes against your traffic, then surface measured improvements in latency, throughput, or cost.

← All articles

What this article says

More in AI & Machine Learning

All →
MOPD: how frontier labs merge RL experts into one model and how to do it using TRL
Hugging Face

MOPD: how frontier labs merge RL experts into one model and how to do it using TRL

OpenAI bugging suspect Alfozan pleads not guiltyПресса
OpenAI

OpenAI bugging suspect Alfozan pleads not guilty

The Duo apps are coming, says AppleПресса
Apple

The Duo apps are coming, says Apple

OpenAI reportedly raises an additional $11 billion at a $1.25 trillion valuationПресса
OpenAI

OpenAI reportedly raises an additional $11 billion at a $1.25 trillion valuation

AI Accessibility: A Guide to Uses, Tools, and Limits
Perplexity

AI Accessibility: A Guide to Uses, Tools, and Limits

Introducing Inkling 2 Small
Thinking Machines Lab

Introducing Inkling 2 Small

More from Modal

How Snorkel runs realistic agent simulations with Modal
Modal

How Snorkel runs realistic agent simulations with Modal

Launching a trillion-parameter model at Sakana AI
Modal

Launching a trillion-parameter model at Sakana AI

Introducing our Training API
Modal

Introducing our Training API

Get started with LLM training on Modal
Modal

Get started with LLM training on Modal

Serverless routing made sticky
Modal

Serverless routing made sticky

NVIDIA Vera CPUs double throughput for Modal Sandboxes
Modal

NVIDIA Vera CPUs double throughput for Modal Sandboxes