Design scalable distributed inference systems for production AI applications, from classic ML models to large language models. Learn how Ray Serve delivers high-throughput, low-latency serving, and how to evaluate the performance tradeoffs in autoscaling, batching, routing, and inference-engine optimization.
Topics include:
- Building Ray Serve applications with deployments, composition, and load-aware request routing
Building Ray Serve applications with deployments, composition, and load-aware request routing
- Deploying and scaling LLM endpoints with Ray Serve LLM on engines like vLLM and SGLang
Deploying and scaling LLM endpoints with Ray Serve LLM on engines like vLLM and SGLang
- Optimizing the inference engine with the KV cache, paged attention, prefix caching, and continuous batching
Optimizing the inference engine with the KV cache, paged attention, prefix caching, and continuous batching
- Scaling frontier models with KV-cache-aware routing, prefill/decode disaggregation, and wide expert parallelism for mixture-of-experts (MoE) models
Scaling frontier models with KV-cache-aware routing, prefill/decode disaggregation, and wide expert parallelism for mixture-of-experts (MoE) models
- Deploying Custom LLMs on Anyscale and Integrating Them with Agentic Platforms (Claude Code, Cursor, Codex)
Deploying Custom LLMs on Anyscale and Integrating Them with Agentic Platforms (Claude Code, Cursor, Codex)
- Comparing costs across self-hosted GPUs, subscriptions, and token-based API billing.
Comparing costs across self-hosted GPUs, subscriptions, and token-based API billing.
