In Part 1: Preparing AI models for deployment, we saw how our model left the training pipeline as a signed, optimized Open Container Initiative (OCI) artifact ready to travel to the edge. Now, our next question is: what will be the implications of running those AI workloads at the edge?
Running AI inference in a data center means using standardized hardware and abundant memory. At the edge, a single fleet might span multiple hardware platforms.
Before thinking about update strategies or how to manage the solution, there's a more fundamental question: what is required to run the AI model properly given a device with constrained hardware and shared resources?
This article follows that question through platform selection, hardware accelerator dependency management, shared memory and latency challenges, and the messaging patterns that connect sensors to inference and inference to downstream systems.
The hardware reality: Why edge AI deployment differs from cloud AI
When running AI at the "core" (cloud or on-premises data centers), you run on standardized hardware. You select a GPU instance type, the Compute Unified Device Architecture (CUDA) version is known, and memory is measured in tens of gigabytes. Edge AI fleets are the opposite: heterogeneous by nature, resource-constrained, and in some cases serving operational workloads that compete with AI inference for the same hardware resources.
A single organization's fleet might include NVIDIA Jetson Orin on aarch64, Intel Atom-based industrial gateways on x86_64 with or without GPU, AMD near-edge servers with Peripheral Component Interconnect Express (PCIe) discrete GPUs, and single-board computers with integrated neural processing units (NPUs). Each might require a different runtime format, a different model variant, and different resource management strategies.
Part 1: Preparing AI models for deployment addressed the runtime format question by showing how the recommended pipeline produces Open Neural Network Exchange (ONNX) as the intermediate representation and, if needed, compiles to hardware-specific formats (TensorRT per GPU SKU, OpenVINO for Intel, and ONNX Runtime for heterogeneous devices). In this article, we'll focus on another thing to take into account related to edge hardware usage: resource competition.
An edge AI inference process could be sharing memory and CPU with the applications' communication stack, the telemetry agent, and the watchdog service. And with edge hardware, this is even more challenging because this is true whether inference runs on the CPU, where all processes draw from the same memory pool, or on an integrated GPU, which is the most common GPU form factor in edge devices.
Platforms like NVIDIA Jetson unify CPU and GPU into a single shared memory pool, meaning operating system (OS) processes and the inference workload are literally competing for the same bytes. Resource contention at the edge doesn't produce slow responses the way it does in a cloud environment; it can cause the inference service to crash mid-inference. In some cases, this makes resource isolation a first-class design concern, not just an optimization.
Additionally, with this heterogeneity, maintaining a hardware compatibility matrix isn't easy. There could be incompatibilities that rarely surface at the beginning. They surface in production, across hundreds of devices, at the worst possible moment.Red Hat addresses this through a certified hardware program forRed Hat Device Edge andRed Hat OpenShift. For each device class in the fleet, a validated combination of OS baseline, kernel, runtime, and supported accelerators is documented and tested, turning hardware management from reactive to proactive and reducing the integration risk that AI inference workloads are particularly sensitive to.
Now that the hardware landscape is understood, the next question, given this hardware heterogeneity, is which software platform you should run on each device class.
Running the inference service: Choosing the right platform tier
The software platform that hosts the inference service determines what operational capabilities are available, how much resource overhead the platform itself consumes, and what tooling integrates with the management infrastructure. There's a spectrum of options, and choosing the right tier for each device class is important for both efficiency and operability. Red Hat knows that, and that's why it offers a broad spectrum of options here to help match your platform and use case requirements.
For the most constrained devices, where competing workloads and the OS itself leave roughly 1 core and 1.5 GB of memory available (3 GB / 4 GB are recommended if you deploy using the network) to the platform operating system,Red Hat Device Edge withPodman, simplified by the use ofQuadlets, provides container runtime capabilities with no Kubernetes overhead. Quadlet translates declarative container definition files into systemd unit files, so the inference service is managed by systemd directly: predictable startup ordering, automatic restart on failure, resource limits via systemd slice controls (MemoryMax and AllowedCPUs), and full integration with journald for logging. Inference servers like a minimal ONNX Runtime server or a custom lightweight inference container run as first-class systemd services. This is the right tier for far-edge devices where every megabyte counts. That said, if you want to stick to the use of container compose files to deploy your applications, that's also possible when using this platform.
For devices with at least 2 cores and 2 GB memory available (3 GB / 4 GB are recommended if you deploy using the network) and where an API is needed, you can use Red Hat Device Edge withMicroShift, which adds a lightweight Kubernetes. MicroShift supports Kubernetes-native workload manifests, health probes, and resource limits. It alsohelps with the edge AI model deployments thanks to KServe (tech preview) running in raw deployment mode (without serverless components) by providing standardized REST and gRPC inference endpoints while consuming significantly less overhead than a full serverless framework. This tier suits industrial PCs and inference appliances that need structured workload management based on Kubernetes without the resource cost of a full Kubernetes cluster.
For near-edge servers with more resources, there's the option of running OpenShift. There's a range of possible architectures available for edge use cases with OpenShift. Asingle node OpenShift (SNO) deployment requires approximately 4 cores and 16 GB of memory. As a side note, those resources are the base infrastructure for SNO; if you have a use case where you want to runRed Hat OpenShift AI on a SNO and not just the inference, you'll need at least 32 CPUs and 128 GB of memory for the infrastructure, and then you'll need to add the resources for the applications.
A compact 3-node cluster also requires 4 cores and 16 GB of memory per node but provides full high availability.
For sites that need high availability but can't justify 3 full nodes, OpenShift 4.20 introduced 2 compact topologies:Two-Node with Arbiter, which adds a lightweight third node whose only job is to maintain etcd quorum, andTwo-Node with Fencing, which achieves the same with only 2 nodes by using baseboard management controller (BMC)-based fencing via Pacemaker to isolate a failing node and prevent split-brain. In those cases, the control nodes need at least 2 cores and 16 GB of memory, while the Arbiter (if any) needs only 1 core and 8 GB.
Below, you can find a summary of the minimum resources required for each OpenShift architecture as it appears inthe official documentation. Note that each CPU core equals 2 virtual central processing units (vCPUs) when Hyper-threading is enabled, which is the common default if running deterministic workloads isn't part of the use case requirements.
Minimum requirements for each OpenShift edge architecture
Lastly, for deployments where even those baselines are too high,OpenShift's composable architecture allows cluster administrators to disable optional components at installation time, such as the console, the baremetal operator, or the storage operator, further reducing the resource footprint to match what the device provides.
While we've mapped some platform options to hardware resource tiers, the choice between Red Hat Device Edge and OpenShift involves more variables than memory and CPU headroom alone. You need to consider the operational model, update strategy, connectivity assumptions, and the degree of Kubernetes compatibility required—all play a role. If you're curious about making that decision, a dedicated decision framework covering those dimensions in depth is available in a previous article series.
Choosing the right platform for each device class is necessary, but not sufficient. Once the platform decision is made, the next challenge is verifying that the hardware acceleration those devices provide works reliably under it.
Hardware acceleration: Managing the dependency chain
Most edge AI deployments rely on hardware acceleration. NVIDIA devices use CUDA and TensorRT. Intel processors use OpenVINO and the oneAPI toolkit. AMD systems use Radeon Open Compute (ROCm). Other dedicated edge AI accelerators bring their own software development kits (SDKs). Each of these stacks works well in isolation, but the challenge specific to AI workloads is that the dependency chain between hardware, kernel driver, user-space toolkit, and inference framework is fragile in ways that can cause silent failures.
That fragility shows up at multiple layers. For example, at the application layer, TensorRT engines may need to be rebuilt when TensorRT, CUDA, driver, or GPU configurations change. At the system layer, OS or kernel updates can invalidate GPU driver modules, making the accelerator completely unavailable to inference workloads. This means, for example, an OS update intended to patch a Common Vulnerabilities and Exposures (CVE) can silently break inference the next time the model tries to load. Another example is that on NVIDIA Jetson platforms, containerized access to the integrated Tegra GPU depends on platform-specific device discovery and container-runtime configuration. CDI (Container Device Interface) specifications may need to be generated or refreshed to reflect changes in the host's accelerator configuration and drivers, and failures in this device exposure workflow can prevent inference containers from accessing the GPU even when the driver itself remains functional.
Addressing these challenges requires controls at 2 levels. At the device level, you must pin the entire software stack, including the kernel version and driver modules. And at the fleet level, you need a deployment pipeline that validates accelerator availability and runs a lightweight inference smoke test after each update to catch an error before it propagates across the fleet.
Red Hat addresses both levels. For the device layer, Red Hat Device Edge and Red Hat Enterprise Linux (RHEL) both can usebootc to manage the OS as an immutable, versioned image that pins the kernel, driver modules, and user-space toolkit together. In those systems, the updates are atomic and rollback is available if an update breaks an accelerator dependency, so the software stack the inference workload depends on never drifts silently between deployments.
On OpenShift-based deployments, the equivalent control comes from declarative node configuration. The Node Feature Discovery operator detects available accelerators and labels nodes accordingly, while theGPU Operator manages the full driver and toolkit lifecycle as a versioned resource. Both operators are configured through custom resources committed to Git, so the accelerator stack becomes part of the cluster definition and changes go through the same GitOps review and rollback process as any other configuration.
For the fleet layer, OpenShift (Tekton) andOpenShift AI (Kubeflow) Pipelines provide the deployment pipeline where accelerator availability can be validated and a lightweight inference smoke test executed after each update.For fleets of standalone RHEL devices that aren't part of a Kubernetes cluster,Red Hat Edge Manager takes on that role, rolling out bootc image updates across the fleet in a controlled, staged way and giving visibility into which devices are running which image version, so the same principles apply whether the fleet is Kubernetes nodes or bare RHEL devices in the field. Combined, these 2 controls close the gap between a working proof of concept (POC) and a deployment that stays working across OS updates, driver upgrades, and model changes.
One more thing needs to be taken into consideration: the dependency chain also creates a versioning lifecycle problem that extends beyond individual device updates. When new device classes are added to the fleet, the model continuous integration, continuous delivery and continuous deployment pipeline might need to produce new hardware-specific compiled variants for hardware that didn't exist when the current model was trained and validated. This isn't a 1-time activity; it must happen for every model version currently in the production and staging registries that needs to run on the new hardware class. Designing the CI/CD pipeline to produce all required hardware variants as a matrix output, tagged with both model version and hardware target, avoids this becoming a nightmare when new device types are procured.
But the accelerator dependency chain isn't the only execution challenge specific to AI workloads at the edge. Once the stack is stable and the accelerator is reliably available, 2 more constraints come into play that cloud deployments don't face in the same way. On many edge devices, CPU and GPU share a single physical memory pool, so any process spike affects the inference engine directly. If you plan to use these kinds of unified memory devices, continue reading the next section to see how it could affect your edge AI use case.
Resource contention and timing guarantees: Edge AI execution challenges
When you use unified memory devices, there are some usual execution challenges in edge AI deployments. The first and most common is resource sharing across AI and regular processes, but, depending on the use cases, there could be others, such as workload latency sensitivity. Let's explore both.
Edge devices and shared resources
On edge platforms with embedded AI hardware, CPU and GPU share a single physical memory pool. That means a memory spike from any of the processes running on the device directly reduces what's available to the inference engine, potentially causing an out-of-memory (OOM) failure mid-inference, or the other way around, the AI inference can get the memory that the OS and applications need to run.
The solution is explicit memory budgeting across all workloads on the device, treating the shared pool as a finite resource that must be partitioned by design rather than claimed opportunistically at runtime. This means setting hard memory ceilings per workload group at the OS level, so no single process can expand into the memory reserved for inference. On Red Hat Device Edge with Podman, this can be done by usingsystemd slice-based controls, and on MicroShift or OpenShift, using the Kubernetes memory requests and limits, which provide similar isolation at the orchestration layer.
Additionally, container memory limits alone may not provide complete protection against memory exhaustion on unified-memory edge platforms. Some inference frameworks and accelerator drivers manage additional memory pools whose consumption isn't always visible through standard container-level metrics. For this reason, specific memory monitoring and framework-level memory configuration are often required in addition to container limits.
The latency challenge
The latency challenge is common in edge AI use cases, and it has 2 distinct dimensions that often require different solutions.
The first is latency sensitivity. Many edge AI applications can't optimize for average response time the way cloud applications do. A conveyor belt inspection system where late detections mean missed defects, or a predictive maintenance system sampling vibration data at high frequency, needs consistently low latency even if an occasional spike is tolerable.
The second is determinism. Some applications require a guaranteed worst-case response time, not just a low average. A safety interlock that must respond within a bounded window, or arobotics controller where scheduling jitter breaks motion sequences, can't accept that the inference step will usually complete in 2 ms but occasionally takes 50 ms because the GPU scheduler deprioritized it or the memory allocator needed to reorganize. It could have physical consequences in that case.
Latency-sensitive inference
Starting with the latency sensitivity challenge, for applications with softer latency requirements using batch inference, the first configuration decision is server batch size configuration. Batching increases GPU utilization and throughput, but a batch size above 1 adds queuing latency that can violate response time requirements. So, for most edge AI use cases sensitive to latency, the batch size should be explicitly set to 1 rather than left to the serving framework default.
In addition, focusing on the AI service architecture and how it can help reduce latency, 2 inference patterns reduce effective resource consumption and inference time without sacrificing accuracy. The first is named "hierarchical" or "cascade" inference, where you run a small, fast model on the constrained end device for common easy cases, escalating to a larger model on a more capable gateway or cloud only when the confidence score is low or in cases where there are no hard latency delays.
The second pattern is "adaptive" inference, where it runs the full inference pipeline only when sensor input has changed meaningfully, using a lightweight change detection check first, making the system free of previous inference queuing when you need to use your model and, as an additional benefit, cutting average power consumption significantly in stable situations.
These patterns assume task-specific models with fast inference times such as object detection, defect classification, predictive maintenance scoring, or pose estimation. A different challenge appears when the edge workload involves a large language model (LLM)—for example, a field assistant running on a near-edge gateway that needs to answer technician queries or summarize sensor logs locally. LLM inference latency is dominated by token generation, not a single forward pass, so the standard optimization techniques we've discussed don't apply directly.
Two techniques are worth knowing for this case. The first isspeculative decoding, which can be seen as a special cascade inference case that uses a small draft model to generate candidate token sequences that a larger verifier model accepts or rejects in parallel, reducing the number of full forward passes needed and cutting time-to-first-token significantly on constrained hardware.
The second technique is key-value (KV) cache quantization that reduces latency and the memory footprint of the attention cache, which is an additional advantage for edge use cases since, as we've seen, on unified-memory edge platforms the inference directly competes with the same memory pool used by every other process on the device. It's important the inference server you use supports those techniques, and in that sense, those are supported in the vLLM inference server, which is available throughRed Hat OpenShift AI as part of the model serving stack.
Determinism and AI inference
For applications that require determinism, this tuning isn't sufficient because the scheduler, the memory subsystem, and the interrupt layer are each capable of producing delays regardless of how well the inference engine is configured. Addressing this requires additional features and configuration at the operating system level.
The first is thePREEMPT_RT real-time kernel configuration that replaces the standard scheduler with one that provides bounded worst-case latency. Besides the real-time (RT) kernel,CPU affinity and core isolation configuration via the isolcpus kernel parameter are a necessary precondition for those RT guarantees. Without them, other kernel threads can still be scheduled onto the inference cores and break the timing contract.
IRQ affinity (interrupt request affinity) extends the same principle to hardware interrupts. Network interfaces, storage, and sensor buses must be put away from the isolated cores because an incoming interrupt will preempt even an RT thread. At the memory layer,hugepages (2 MB or 1 GB pages via hugetlbfs) reduce TLB misses when large model weights and activation buffers are used on edge hardware. With all of these in place, the full control loop has a predictable worst-case execution time.
It's worth mentioning that these features aren't only available in RHEL and Red Hat Device Edge, it's also possible to configureCPU and IRQ affinity/isolation, memory hugepages, and other optimization capabilities in OpenShift.
It's also important to mention that the determinism controls we've described (real-time kernel, CPU isolation, IRQ affinity, etc.) are designed for single fast inference calls, and they don't apply to LLM inference, where latency is dominated by iterative token generation and is measured in hundreds of milliseconds regardless of scheduler configuration.
This section also doesn't cover non-uniform memory access (NUMA node)-specific tuning because multisocket hardware is uncommon at the edge. On the rare edge installation that does use multisocket hardware, those mechanisms are also available in RHEL, Red Hat Device Edge, and OpenShift, so those can be layered on top of the controls described here.
Connecting inference to data: Edge AI messaging patterns
We've described how to run inference efficiently on a device, but how does data reach the inference service, and how do inference outputs reach the systems that consume them? At the edge, this's an architectural choice with direct consequences for latency, reliability, and machine learning operations (MLOps) processes.
Most sensors and data sources don't speak REST or gRPC natively. They expose data through kernel drivers,Video4Linux2 (V4L2), serial interfaces, or proprietary SDKs, so a capture process almost always sits between raw source and inference service. The messaging pattern is therefore a choice about how that capture process, and the downstream consumers of inference results, coordinate with each other.
While designing data movement architecture, you'll need to define how data physically moves between processes (serialized copies over sockets versus zero-copy shared memory), and if the data exchange will be synchronous, asynchronous, or streaming. Simplifying the combination of these options, there are 3 patterns that cover the majority of edge AI deployments.
Data ingestion patterns during inference
Synchronous request-response
The capture process calls the inference service over REST or gRPC, waits for a result, and acts on it. This is simple to trace, but it creates tight coupling between the caller and the inference service since, if a model update changes how long inference takes or what the output looks like, the caller breaks. That means it has MLOps implications because when the model changes, you'll need to verify that it doesn't break the data pipeline.
Event-driven pub/sub
But there are multiple pub/sub solutions; which one should you use? The most common based on the broker architecture areMessage Queuing Telemetry Transport (MQTT) and Kafka, while you have ZeroMQ andData Distribution Service (DDS) that skip the broker entirely and connect publishers and subscribers directly.
/)
/)
