BLOG

Product

AI Architecture

Services bring stable, managed endpoints to Haystack Enterprise Platform and lay the foundation for a broader AI Gateway

Published on

October 1, 2026

min read

TLDR

Key Metrics:

Agent development is a constantly changing, iterative process. Teams update prompts, switch models or data sources, refine retrieval strategies, and update business logic as they learn from testing and real-world usage. But the applications and workflows that depend on your agents need stability.

So how do you empower your AI engineering teams to work fast and test flexibly without the fear of breaking things? That’s where Services comes into play.

Services: stable endpoints for evolving AI

In Haystack Enterprise Platform, a Service is a stable, managed endpoint connected to a chosen pipeline version powering your agent. Apps and other agents connect to the Service rather than directly to the underlying pipeline. When a new pipeline version is ready, teams can publish it without changing the endpoint used by its consumers.

With Services, teams can:

Publish versions safely: Choose exactly which pipeline version is live, record what changed, and keep the last successful version serving if a new deployment fails.
Separate environments and consumers: Create independent Services for testing, staging, production, or individual applications and integrations.
Configure each runtime: Set replicas and idle timeouts for every Service, or use recommended system defaults.
Operate with visibility: Review traces, performance, connection details, and publishing history at the Service level.

Services create a reliable interface between an organization’s AI capabilities and everything that consumes them.

The foundation for our AI Gateway

AI Gateways are often associated with one endpoint for accessing multiple models, supported by routing, fallbacks, and spend tracking. That remains an important part of the category, but enterprise requirements are expanding.

AI applications increasingly depend on more than models. They use tools, pipelines, data, MCP servers, and other agents. Before you can control access, cost, security, or routing, you need a stable object those controls can apply to. Services provide that foundation.

In Haystack Enterprise Platform, Gateway is the product area where teams expose and operate the AI capabilities they build. Services are the first foundational capability inside it.

Get started

If you're running pipelines in production, Services are the recommended way to publish them. You create and manage them in Gateway, and existing deployed pipelines can be moved over when you're ready.

SeeGateway andCreate a Service in the docs to publish your first one.

Curious about building AI Apps and Agents?

meet the author

meet the authors

Loretta Shen

Senior Director, Product Marketing

Loretta leads product marketing at deepset, shaping product strategy and go-to-market while defining the market narrative for sovereign enterprise AI.

Volkan Arisli

Senior Product Engineer

Volkan builds products delivering the runtime foundations teams need to operate AI applications and agents reliably.

Table of Contents

GET STARTED WITH A PERSONALIZED DEEPSET DEMO

Book demo