Today, we are announcing the Special Interest Group (SIG) for Ray on hardware accelerators.

When Ray was created in UC Berkeley over 10 years ago, the computing landscape looked very different. At the time, “distributed computing” referred to CPU clusters running frameworks like Apache Hadoop and Apache Spark. GPUs were only just moving from graphics and scientific computing into machine learning.

Today, accelerated computing is front and center in the development and deployment of AI, and accelerators and their capabilities have become more specialized and diverse. For example:

NVIDIA’s RTX 6000 Pro supports flexible workloads spanning AI training, inference, graphics, and video

NVIDIA’s RTX 6000 Pro supports flexible workloads spanning AI training, inference, graphics, and video

NVIDIA B200s and AMD MI355X provide substantially greater memory capacity to support larger models, and GB200 NVL72 connects 72 GPUs for communication-intensive, large-scale training and inference.

NVIDIA B200s and AMD MI355X provide substantially greater memory capacity to support larger models, and GB200 NVL72 connects 72 GPUs for communication-intensive, large-scale training and inference.

Beyond GPUs, Google Ironwood TPUs target massive-scale AI workloads and Groq LPUs specialize in low-latency inference.

Beyond GPUs, Google Ironwood TPUs target massive-scale AI workloads and Groq LPUs specialize in low-latency inference.

The Ray community has been actively working on keeping up with this shift. Today, Ray supports more than a dozen kinds of hardware accelerators, from NVIDIA and AMD GPUs to TPUs, Trainium, Gaudi and others. We believe now is a crucial time to build on this foundation with a more ambitious, systematic effort across Ray to make accelerators easy to use, efficient at scale, and reliable in production.

The Accelerator Special Interest Group aims to bring together hardware vendors, Ray contributors, and application developers to define and advance a shared, vendor-neutral roadmap for accelerator support in Ray. This public interest group will be responsible for enabling Ray workloads to continue evolving in an era of fragmented, specialized accelerators.

Our initial roadmap focuses on 3 core areas: Development and Testing Infrastructure, Hardware Awareness Capabilities, and Networking/Transport, outlined in the following sections.

LinkDevelopment and Testing Infra

Reliable accelerator support requires dependable access to hardware for development, testing, and debugging. Maintainers and contributors need to validate changes on the devices they affect, reproduce failures, and catch regressions before they reach users. As agentic software engineering accelerates development, reliable testing and verification become even more important: both human contributors and coding agents need reproducible feedback from real hardware and representative workloads. We want to make this a routine part of contributing to Ray, with a clear path for hardware vendors and cloud providers to contribute resources and expertise.

Our initial priorities are:

Shared hardware access. Establish a documented process for bringing cloud and vendor-hosted accelerators into Ray’s development and testing infrastructure, including access for contributors to reproduce and debug failures.

Shared hardware access. Establish a documented process for bringing cloud and vendor-hosted accelerators into Ray’s development and testing infrastructure, including access for contributors to reproduce and debug failures.

Reproducible development and test environments. Provide maintained environments with explicit driver, runtime, and dependency versions, so contributors can reproduce CI results and validate changes consistently across providers.

Reproducible development and test environments. Provide maintained environments with explicit driver, runtime, and dependency versions, so contributors can reproduce CI results and validate changes consistently across providers.

Broader CI coverage. Extend existing pipelines with targeted checks for pull requests and broader nightly and release testing. Coverage should include distributed workloads, failure recovery, and performance regressions, alongside basic device functionality.

Broader CI coverage. Extend existing pipelines with targeted checks for pull requests and broader nightly and release testing. Coverage should include distributed workloads, failure recovery, and performance regressions, alongside basic device functionality.

Sustainable operation and ownership. Define responsibilities for maintaining environments, investigating failures, and keeping integrations working across releases. Make test results and infrastructure health visible, and manage capacity and costs so the infrastructure can grow with participation.

Sustainable operation and ownership. Define responsibilities for maintaining environments, investigating failures, and keeping integrations working across releases. Make test results and infrastructure health visible, and manage capacity and costs so the infrastructure can grow with participation.

LinkHardware-aware Scheduling

For applications to get the most out of an accelerator, Ray first has to know what hardware each node has and how it is structured. As the accelerator ecosystem evolves, accelerator nodes are no longer just a count of identical devices. Workload performance depends on the interconnectivity of devices, their relationship to CPUs and NICs, and node groupings into larger domains like racks and TPU slices.

Our goal is to give Ray an accurate and consistent picture of this structure across vendors and platforms. With that picture, Ray’s scheduler can place tasks and actors on the right hardware, and users can see what their cluster is made of and how it is connected.

Our initial priorities are:

Common hardware interfaces. Define vendor-neutral interfaces for labeling hardware, representing its architecture, and describing its specifications, so applications and libraries can target capabilities instead of writing vendor-specific code. This includes a consistent way to identify accelerator SKUs, including new ones at launch. It also includes a documented extension point and checklist for vendors adding a new accelerator.

Common hardware interfaces. Define vendor-neutral interfaces for labeling hardware, representing its architecture, and describing its specifications, so applications and libraries can target capabilities instead of writing vendor-specific code. This includes a consistent way to identify accelerator SKUs, including new ones at launch. It also includes a documented extension point and checklist for vendors adding a new accelerator.

Hardened discovery. Strengthen Ray’s discovery mechanism, with clear ownership for each accelerator’s integration, and back it with the CI coverage described in the previous section. Discovery should capture the topology and architecture of the hardware, not just the type and number of devices.

Hardened discovery. Strengthen Ray’s discovery mechanism, with clear ownership for each accelerator’s integration, and back it with the CI coverage described in the previous section. Discovery should capture the topology and architecture of the hardware, not just the type and number of devices.

Platform and cloud integration. Work with cloud providers and deployment platforms so that discovery reflects the underlying structure wherever a cluster runs. Some of this information, such as which rack or slice a node belongs to, is only available from the platform, not from the node itself.

Platform and cloud integration. Work with cloud providers and deployment platforms so that discovery reflects the underlying structure wherever a cluster runs. Some of this information, such as which rack or slice a node belongs to, is only available from the platform, not from the node itself.

Hardware visibility. Provide ways to view the accelerators in a cluster and how they are connected, alongside accelerator metrics on the Ray dashboard. This lets users confirm that Ray sees their hardware correctly and understand where their work was placed. It also gives contributors a way to verify new integrations.

Hardware visibility. Provide ways to view the accelerators in a cluster and how they are connected, alongside accelerator metrics on the Ray dashboard. This lets users confirm that Ray sees their hardware correctly and understand where their work was placed. It also gives contributors a way to verify new integrations.

Clear user guides. Document how Ray discovers and represents hardware, and how to use that information when placing work. Good guides help both application developers and the coding agents that increasingly write Ray code for them.

Clear user guides. Document how Ray discovers and represents hardware, and how to use that information when placing work. Good guides help both application developers and the coding agents that increasingly write Ray code for them.

LinkNetworking and I/O Transport

Moving data between accelerators is often just as important as running computation on the accelerator itself. Modern AI workloads, like large scale reinforcement learning, require moving large model weight tensors across devices and nodes from training to inference workers. Historically, Ray’s object store was built for CPU data, meaning accelerator tensors had to be copied to host memory, serialized, and sent over standard host networking before reaching another device.

To avoid that bottleneck, Ray Direct Transport (RDT) was introduced to enable actors to pass references to tensors through Ray while transferring the actual data directly between devices and using high performance transports like UCX and RDMA. Our goal is to make high-speed, direct data transfer work reliably across different accelerators, frameworks, and network fabrics, while keeping Ray’s familiar programming model.

Our initial priorities are:

Broad transport integrations. Support a wider range of accelerators, frameworks, and interconnects in RDT so Ray can select the fastest path between Actors, whether over intra-slice TPU ICI, same-node NVLink, or cross-node GPUDirect RDMA.

Broad transport integrations. Support a wider range of accelerators, frameworks, and interconnects in RDT so Ray can select the fastest path between Actors, whether over intra-slice TPU ICI, same-node NVLink, or cross-node GPUDirect RDMA.

Transport lifecycle and recovery. Ensure each hardware transport cleanly handles initialization, teardown, and error recovery on real devices, so RDT works reliably as actors scale or restart in dynamic workloads like reinforcement learning and disaggregated serving.

Transport lifecycle and recovery. Ensure each hardware transport cleanly handles initialization, teardown, and error recovery on real devices, so RDT works reliably as actors scale or restart in dynamic workloads like reinforcement learning and disaggregated serving.

Validated recipes and observability. Enable cloud and hardware providers to publish validated deployment configurations, benchmark high-throughput workloads, and surface clear diagnostics and transport metrics.

Validated recipes and observability. Enable cloud and hardware providers to publish validated deployment configurations, benchmark high-throughput workloads, and surface clear diagnostics and transport metrics.

LinkScope

While core scheduling and placement logic remain in Ray Core, SIG Accelerator is responsible for defining the plug-in interfaces, CI testing, and maturity tiers for accelerators in Ray. Hardware vendors and cloud providers contribute and maintain their own device plugins, transports, and day-zero support for new SKUs, while framework authors and ML practitioners ground our roadmap in real workload needs.

LinkGet Involved

Whether you are building a new accelerator integration, running large-scale training or inference on Ray, or providing cloud and CI infrastructure, we invite you to get involved!