EAGLE 4 on AMD Instinct GPUs

EAGLE-4, the newest drafter from the EAGLE team, runs on AMD Instinct™ MI355X GPUs from day 0, in SGLang and ATOM. - On Qwen3.8-27B, one MI355X GPU serves a single request at ~300 tokens/s in BF16: 3.5x plain decoding and 1.6x the model’s built-in MTP. - The gain holds as load grows, from 1 to 32 concurrent conversations, and scales to four GPUs. It works out of the box: the same checkpoint and the same serving flags as on other hardware.

Figure 1: EAGLE-4 at one request

What is new in EAGLE-4

Speculative decoding lets a small drafter guess several tokens ahead, and the large model checks them all in one pass. EAGLE-4 adds three ideas on top of the EAGLE family:

• One drafter for every batch size and every GPU. EAGLE-4 drafts one token at a time, so it can stop, deepen or widen its guess tree at every step. EAGLE-Harness measures what drafting and verifying cost on the actual hardware, once at start-up. During serving it picks the tree that gives the most accepted tokens per unit of cost: a wide, deep tree for a single user, short chains under heavy load.
• A cheaper LM head for the drafter. The drafter’s vocabulary projection used to be most of its cost. A low-rank head first shortlists candidates from the full vocabulary, then the full head scores only those exactly. This cuts the draft’s cost by more than 40% without truncating the vocabulary.
• A small drafter. One transformer layer, so it fits next to the model with little memory, from data-center GPUs down to edge devices.

EAGLE-4 runs on the MI355X GPU with the same checkpoint and the same launch flags as elsewhere:

To make it fast on day 0, we measured EAGLE-Harness’s cost tables on the MI355X GPU, so tree sizes fit this GPU; added tree verification for Qwen3.8’s linear-attention (Gated DeltaNet) layers: a guess tree is checked in one pass, and only the accepted branch’s state is kept; checked losslessness: every output token matches plain decoding, up to BF16 rounding at near-ties.

Results

Qwen3.8-27B, BF16 weights, one MI355X GPU, SPEED-Bench (the EAGLE-4 paper’s benchmark: thinking on, greedy).

Concurrent conversations

Plain decoding (SGLang)

292

872

1,285

Built-in MTP (SGLang)

178

617

1,743

2,551

EAGLE-4, SGLang

289

941

2,167

2,784

EAGLE-4, ATOM

309

976

2,309

3,243

Figure2: EAGLE-4 against load

• One request: EAGLE-4 gets ~4.7 tokens accepted per check, against ~2.9 for the built-in MTP. That is where the 1.6x comes from.
• Under load: EAGLE-4 shortens its guesses as the GPU fills up, so it stays ahead of MTP at 32 conversations.
• More GPUs: with tensor parallelism, one request reaches 403 tok/s on two MI355X GPUs (ATOM) and 463-489 on four (SGLang / ATOM).

Four GPUs

Tensor parallel across four MI355X GPUs, same benchmark:

Figure 3: EAGLE-4 on four GPUs

Concurrent Conversations

Plain decoding (SGLang)

156

539

1,655

2,331

Built-in MTP (SGLang)

303

1,036

2,711

3,916

EAGLE-4, SGLang

463

1,360

3,339

4,474

EAGLE-4, ATOM

489

1,347

3,381

4,796

With SGLang, EAGLE-4 is 3.0x plain decoding and 1.5x MTP for one request on four GPUs, and still 1.9x and 1.14x at 32 conversations.

What’s next

We would love to bring EAGLE-4 to more models, and we are combining it with our single-request megakernels for even lower latency(Link to blog). We welcome collaboration with the EAGLE team, model builders and the community.

Test configuration

• Hardware: AMD Instinct MI355X GPU (gfx950); TP1 unless stated. AMD ROCm™ software 7.2.4.
• Model: Qwen/Qwen3.8-27B in BF16; draft: the EAGLE-4 checkpoint for Qwen3.8-27B.
• Benchmark: SPEED-Bench qualitative split, every 10th conversation of each category (88 conversations), replayed turn by turn; thinking on, temperature 0; tok/s = generated tokens over wall-clock time.
• Engines: SGLang: the EAGLE-4 fork at main 2af5570 with the ROCm PR rebased on it; ATOM: branch eagle4_day0_support of proaditya/ATOM, on ATOM main 2479251d3, in rocm/atom-dev:nightly_202610131505 (ROCm 7.2.4, with aiter and FlyDSL).. SGLang uses EAGLE-Harness trees at every concurrency shown; ATOM uses fixed trees per batch size up to 16 conversations and chains at 32.

Reproducing the numbers

Both engines are measured with the same SPEED-Bench client and data from the EAGLE-4 repository, inference/scripts/client_speedbench.py and inference/dataset/speed_bench/question_multiturn.jsonl; for ATOM, the client reads token counts from the response’s usage field. The client replays each conversation turn by turn (thinking on, reasoning effort xhigh, temperature 0); the reported number is wall_tput, generated tokens over wall-clock time. --every 10 is the 88-conversation subset used in the tables; omit it for all 880.

ATOM: one command serves each concurrency with the recipe’s flags, runs the client and prints the results next to ours; the README has the options (TP size, concurrencies) and the recipe the flags behind them:

EAGLE4=<EAGLE-4 checkout> DRAFT=<EAGLE-4 checkpoint> bash blog_benchmark/eagle4/build_and_repro.sh

SGLang (the EAGLE-4 repository’s SGLang, on the ROCm SGLang image for the MI355X GPU):

The three SGLANG_GDN_TREE_* switches turn on the FlyDSL tree verification for the linear-attention layers (measure the cost tables with them set as well). For TP > 1, use --tp-size N in both commands and a cost table per TP size. Plain decoding: the same server command without the --speculative-* flags; built-in MTP: --speculative-algorithm EAGLE --speculative-draft-model-path $MODEL --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4.