Introducing FastFlow SDK

Say you're building an AI application and want local inference running fast: you point your app at an OpenAI-compatible endpoint, and you're ready. Now, say you need to tune models for a specific workload: you want to customize how they execute. Or say you need closer access to the metal to optimize the operators and kernels, such as GEMM, attention, and RoPE, that make your hardware fast in the first place. Different developers, different starting points, but all of them are trying to do real work on the same hardware.

Here's the catch: most tools only solve for the first case. Point-and-run inference engines get you to a working demo fast, but the moment you need to adapt how a model executes, or build a custom operator, you hit a wall. The serving API simply doesn't expose what's happening underneath it, so you're stuck choosing between an easy on-ramp and real access to the hardware.

FastFlow-SDK removes that tradeoff. It is one-stop platform by AMD for local AI software development, so you can start by running a supported model and keep going as far beneath the surface as your work demands, all within the same eco-system, without switching any licenses, vendors or tool chains.

This first blog in the series introduces FastFlow-SDK and the three technologies that make it possible: FastFlow, IRON, and HRX. The initial release targets Ryzen™ AI NPUs (XDNA2).

FastFlow-SDK meets you on the same stack for whatever advantage you are looking for:

• Run: Point your app at an OpenAI-compatible endpoint and get instant speed inference.
• Customize: Adapt model execution and build new capabilities using the tools underneath.
• Optimize: Build and tune operators and kernels directly at close-to-metal level.

The rest of this blog follows this workflow from FastFlow, to IRON, to HRX, showing how each layer carries you a little deeper into the stack.

FastFlow-SDK provides a single-entry point for developers targeting Ryzen AI NPUs. It includes support for running state-of-the-art GenAI models while exposing the underlying software stack for developers who need to customize model execution, operators, and kernels.

FastFlow-SDK assembles the following components:

• FastFlow (formerly FastFlowLM) - the application layer for running and serving AI models on the NPU.
• IRON - a close-to-metal Python API for programming Ryzen AI NPUs, built on an open-source MLIR-based compiler toolchain.
• Hip Runtime eXtended (HRX) - a lightweight runtime layer responsible for loading and executing workloads on AMD hardware across Windows and Linux.

FastFlow

FastFlow is an NPU-native inference engine integrated directly into Lemonade, which exposes an OpenAI-compatible streaming API and supports the context lengths of the models it serves. FastFlow also supports tool calling and reasoning models, enabling agentic and multi-step AI workflows on the NPU.

FastFlow supports a range of language, multimodal, speech, and embedding models:

Model Type

Model Families

Text LLMs

Qwen (2.5 / 3 / 3.5 / 3.6-MoE)

Mixture-of-Experts

Gemma 3/4, gpt-oss, Llama 3.1/3.2, Phi, LiquidAI/LFM2

Vision-Language

Gemma 3/4, Qwen3-VL, Qwen2.5-VL

Speech

Whisper large-v3-turbo

Embeddings / RAG

EmbeddingGemma 300m

Click here for complete list.

Together, these models provide a starting point for building chat applications, agents, multimodal experiences, speech workflows, and RAG pipelines targeting Ryzen AI NPUs.

IRON

Under the hood, the models supported by FastFlow are built from operators and kernels developed with IRON. While FastFlow provides ready-to-run model inference, IRON is the layer used to build and customize the NPU programs that power those models.

IRON is a close-to-metal Python API for programming AMD Ryzen AI NPUs, built on an open-source MLIR-based compiler toolchain. It takes Python-defined workloads and compiles them into programs that run on the AI Engine array, giving developers direct control over kernel implementation and data movement on the NPU.

Key capabilities include:

• Python-to-NPU compilation: The @iron.jit decorator compiles Python-defined workloads into executable NPU programs and caches the resulting artifacts.
• Low-level hardware control: Developers can place tiles, manage data movement, and write vectorized compute directly for the AI Engine array.
• Simplified data movement: ObjectFifo provides a declarative producer-consumer model that reduces the need to manually define DMA sequences and distribution patterns.
• Common AI operators: Examples include attention, GEMM, softmax, RMSNorm, LayerNorm, RoPE, SwiGLU, GELU, SiLU, Conv2D, and ResNet building blocks.

These operators form many of the building blocks used in modern transformer and vision models, including the operators used in the forward passes of models such as Qwen and Llama. With IRON, developers can build and optimize those building blocks directly for the NPU.

Hip Runtime eXtended (HRX)

HRX sits between compiled programs and the underlying hardware, acting as the runtime layer responsible for execution. It loads compiled programs and executes them alongside the device driver on Linux and Windows, handling memory management, asynchronous work queuing, synchronization, and command execution, thus, the layer that determines how fast everything above it runs.

HRX supports the two things you do with an NPU:

• Single dispatch: Run one program, check the results, and iterate. Useful when developing and tuning IRON kernels.
• Chained dispatch: Queue a sequence of operations for execution as a pipeline, reducing runtime overhead across a full model workload. This is the execution model used by FastFlow for end-to-end inference.

Getting Started

1. Run: Lemonade with the FastFlow backend

The easiest way to get started is through Lemonade, which includes FastFlow as a supported backend. If your goal is to build an AI application, benchmark a model, or evaluate NPU performance, this is the fastest path. You can begin running supported models immediately. To get started with Lemonade, visit the Lemonade website.

For more information on Lemonade backends, see the documentation here.

2. Customize: Build and modify with FastFlow-SDK

Once you're ready to move beyond running models, FastFlow-SDK provides access to the tools used to build and customize them. Developers can experiment with model configurations, integrate new capabilities, and explore how workloads are assembled for the NPU.

As your application requirements evolve, FastFlow-SDK serves as the entry point to the broader development stack and documentation. From there, you can begin working directly with the underlying projects and move deeper into the stack as needed.

To get started, visit the FastFlow-SDK repository for installation instructions, examples, and links to the underlying project documentation.

3. Optimize: Tune your performance with IRON and HRX

Performance tuning happens at the IRON and HRX layers. Use IRON to develop and optimize operators and kernels, then use HRX to control how they are executed on the NPU.

Single-dispatch execution is useful for validating individual IRON kernels, while chained dispatch reduces overhead across larger workloads by executing multiple operations as a pipeline. This gives you control over both kernel implementation and runtime execution. This is the same way the models that FastFlow supports were built.

Ready to try it?

Start with to run models on the Ryzen AI NPU, then explore FastFlow-SDK as your development needs expand to include model customization, operator development, and runtime optimization.

Visit the FastFlow-SDK repository for installation instructions, examples, and links to the underlying projects.

Stay Connected