Dev48
Language
  • About
  • Services
  • Industries
  • Technologies
  • Articles
  • Contacts
Book a call
    Home/Articles/Roce and nvidia quantum infiniband and their quantified tradeoffs
Dev48
info@dev48.org+375 29 320-40-96WhatsApp@isky_by

© 2026 · All rights reserved.

RoCE and NVIDIA Quantum InfiniBand, and their quantified tradeoffs

Source: Lambda•November 20, 2026 at 02:53 PM

RoCE and NVIDIA Quantum InfiniBand, and their quantified tradeoffs

The GPUs you rent are just part of what determines your cluster's speed and efficiency. This comes into play through interconnect , the network that connects multiple nodes (machines) on the backend.

The GPUs you rent are just part of what determines your cluster's speed and efficiency. This comes into play through interconnect, the network that connects multiple nodes (machines) on the backend.

Historically, networking was viewed primarily as a cost center. Within AI factories, networking has become a business accelerator because higher GPU utilization, lower latency, improved synchronization, and predictable performance directly translate into greater productivity and improved return on AI investments.

Interconnects connecting GPUs typically come in two forms:

  • RDMA over Converged Ethernet (RoCE), which includes RoCEv1 and RoCEv2
  • NVIDIA Quantum InfiniBand

Here's a quick breakdown of how they work and why they matter:

RoCE operates at a networking layer called "layer 2." Meaning, it uses Ethernet and MAC addresses to communicate across the same network (broadcast domain). Nodes can communicate across multiple switches, but they must remain within the same "base" network.

RoCEv2 builds on this by using UDP/IP packets, allowing nodes to communicate through different subnets, buildings, or data centers. As a result, latency can increase since these nodes aren't guaranteed to be sitting in the same rack or data center.

NVIDIA Quantum InfiniBand uses a dedicated networking fabric rather than Ethernet/IP and can cross multiple switches and locations. However, it requires the foundations of the InfiniBand-enabled switches and fabric to be in place first to get the maximum performance possible with minimal latency. Guaranteed if you use a Lambda 1-Click-Cluster.

But what if you're not running on them? How much does interconnect affect performance?

At a high level, both InfiniBand and RoCE can achieve extremely low latency, often measured in just a few microseconds. InfiniBand generally delivers lower and more predictable latency, typically around 0.5–2 µs, while RoCE on a well-optimized local fabric can achieve roughly 1–3 µs.

However, RoCE is more sensitive to network configuration, congestion, and overall topology. In less optimal conditions (such as other traffic on the node, misconfigured switches, and non-optimized traffic), latency can be several times higher, potentially reaching tens of milliseconds under congestion. For now, we'll use these low-latency ranges as our baseline, bearing in mind that real-world performance will vary substantially, which we will see next.

The theory in practice

A 1-3 µs latency between communications doesn't seem like a large amount, but that's per transfer, and doesn't include anything about packet sizes, tensor allocation, and the like.

To see how these differences play out in practice, we ran an inference experiment:

  • 2 nodes of 1-Click-Cluster NVIDIA HGX™ B200 connected through NVIDIA Quantum InfiniBand switches and adapters
  • 2 nodes of HGX B200 off On-Demand, connected through RoCE.

To keep things fair, we spun up two HGX B200 On-Demand instances, ending up with two nodes in the same cloud region, but not per se in the same data center.

Here are the key details of how we ran our tests:

  • Model: nvidia/GLM-5.2-NVFP4, an NVIDIA Blackwell-optimized variant of Z.ai's GLM 5.2 model
  • Serving framework: SGLang
  • Parallelism: TP16/DP16/EP16
  • Benchmarking tool: NVIDIA's AIPerf
  • Workload: 8,192 input tokens and 1,024 output tokens per request
  • Concurrency: 10-300 simulated concurrent requests

Both servers used a core set of SGLang flags:

RoCE also used:

While IB used:

The data

Our results show that as concurrency reaches about 250 users (hardware saturation), RoCE delivers roughly 70% lower throughput than NVIDIA Quantum InfiniBand, with a 2.5x slower time to first token. As the number of users decreases, the effects of the different fabrics become more apparent, with low concurrency showing the slowest inter-token latency and tokens per second overall.

The lesson

The results show that the interconnect is not a background detail. At scale, it directly shapes GPU utilization, responsiveness, and the return on your infrastructure investment.

When serving at scale, communication can become the bottleneck. Every matrix multiplication needs to move across the fabric between multiple nodes, leading to large gaps in speed if your nodes don't have an excellent interconnect.

In a pinch? Spin up two Lambda On-Demand or Reserved Instance nodes, and temper your expectations accordingly.

Looking for a more viable solution? Lambda 1-Click-Clusters connected through a dedicated NVIDIA Quantum InfiniBand fabric is the obvious solution for your serving needs.

← All articles

What this article says

More in AI & Machine Learning

All →
WRITER AI Studio is now Control Center
Writer

WRITER AI Studio is now Control Center

Introducing harness adapters: adapt your favorite model to your harness to improve performance and cost
Hugging Face

Introducing harness adapters: adapt your favorite model to your harness to improve performance and cost

Anthropic announces closing of initial public offering, including full exercise of underwriters’ option to purchase additional shares, and closing of concurrent private placement
Claude

Anthropic announces closing of initial public offering, including full exercise of underwriters’ option to purchase additional shares, and closing of concurrent private placement

Dahua Technology Brings Buildings to Life at Frontier Innovation Event 2026
Dahua Technology

Dahua Technology Brings Buildings to Life at Frontier Innovation Event 2026

Continuously Measuring Cross-Sections Immediately After Extrusion: Measuring Wall Thickness, Width, and Corner Radius, and Feeding Back to Die Adjustment
Keyence

Continuously Measuring Cross-Sections Immediately After Extrusion: Measuring Wall Thickness, Width, and Corner Radius, and Feeding Back to Die Adjustment

Judging Exterior Deformation by Depth: Managing Swelling, Wrinkles, Dents, and Depressions With Thresholds
Keyence

Judging Exterior Deformation by Depth: Managing Swelling, Wrinkles, Dents, and Depressions With Thresholds

More from Lambda

Lambda announces the world’s first large-scale CPO deployment of NVIDIA Photonics switches
Lambda

Lambda announces the world’s first large-scale CPO deployment of NVIDIA Photonics switches

Lambda delivers faster AI training for language and image models
Lambda

Lambda delivers faster AI training for language and image models

TurboPairFormer: faster, more stable, more accurate OpenFold3 fine-tuning
Lambda

TurboPairFormer: faster, more stable, more accurate OpenFold3 fine-tuning

Lambda has a new address: lambda.com
Lambda

Lambda has a new address: lambda.com

Introducing identity and access management for Lambda Cloud
Lambda

Introducing identity and access management for Lambda Cloud

Run:ai open source comes to Lambda Kubernetes Service
Lambda

Run:ai open source comes to Lambda Kubernetes Service