How Atos uses Red Hat AI to build smaller, specialized models for sovereign AI

For an enterprise, AI sovereignty starts with this primary requirement: Retain control of its own data and infrastructure. But as organizations move from experimenting with AI to deploying it in real business processes, control over the models themselves has become increasingly important. This includes the ability to fine-tune and adapt models to the specific work the enterprise needs them to do.

Atos is doing exactly that with Red Hat AI, in a project focused on parsing German-language social media content across 14 categories, such as indicators of harmful content. To remain reliable for downstream systems, the model must deliver its insights in a consistent, defined JSON structure. At the same time, the solution must meet strict sovereignty requirements, with control over the data, models and training pipeline.

For this to work, Atos had to find an approach that could deliver the required accuracy and reliability, while also remaining both sovereign and economically viable at scale.

When more context isn't the answer

RAG has become an important way for enterprises to connect AI models to proprietary information. But not every problem can be solved by giving a model more context.

For Atos, the challenge was getting a model to reliably perform a specific task: Interpret primarily German-language content, extract the right information, and consistently return it in the required structure. That requires changing how the model behaves rather than simply what information it has access to.

With Red Hat AI, Atos was able to use synthetic data generation to create the high-quality datasets needed to fine-tune small models for structured extraction.

This addresses the scarcity of domain-specific data (a common fine-tuning hurdle), allowing enterprises to generate synthetic training sets needed to specialize a model for a particular enterprise task.

This gives an enterprise another way to connect its data to AI. RAG can provide models with relevant enterprise knowledge, and fine-tuning can be used when the goal is to teach a model more specialized and consistent behavior.

Small models can change the economics of enterprise AI

Why do this rather than simply using a frontier model? At enterprise scale, model size matters. The Atos use case was designed with potentially millions of records per day in mind, making inference cost and latency important considerations.

The key challenge was to find how small a model could be and still perform the required tasks. Fine-tuning improved Phi-4 Mini's overall evaluation score by as much as 26.4% compared with its base-model baselines. The tuned model also learned to consistently produce valid JSON, reaching approximately 99% validity through fine-tuning alone, without relying on an additional harness to enforce the output structure. This is particularly important because it shows that the required output behavior was learned by the model itself.

More importantly, the smaller models were able to compete with GPT-4.1 on this specific task. The fine-tuned Qwen 2.5 7B model outperformed GPT-4.1 in the evaluation, while the fine-tuned Phi-4 Mini came very close to GPT-4.1 despite its much smaller size. Phi-4 Mini also outperformed NuExtract 2.0 8B, a model purpose-built for structured extraction and more than twice its size, by up to 22.6% in overall score.

This doesn't mean smaller models will outperform frontier models for every workload, rather it proves that the largest model isn't necessarily the right choice for every task.

For well-defined enterprise workloads, fine-tuning a small model delivers the right amount of precision without the overhead of a general-purpose frontier model. For high-volume production, this dramatically improves the economics of running AI at scale.

From data sovereignty to AI sovereignty

Sovereignty isn't only about keeping data within a specific geographical border. Enterprises increasingly need control over the models themselves: Where they run, how they are adapted, and how they integrate into business processes.

Open models provide an important part of that control. An open model gives you the ability to customize models for your own workloads, and to deploy it on infrastructure you control.

This becomes increasingly relevant with agentic AI. Enterprise agents often orchestrate many specialized tasks, such as extracting information, classifying requests, producing structured outputs and interacting with enterprise systems. These tasks don't always require the broad capabilities of a frontier model. They often require consistent and task-specific behavior.

Rather than sending every task to a massive frontier model, you can choose the right model for each workload and specialize smaller models where it makes sense.

For Atos, the results prove that a small, tuned model can match or exceed frontier model performance on specific enterprise tasks, while delivering complete control required for sovereign AI.

Sovereign AI is not simply about where AI runs. It is also about giving enterprises the control and flexibility to shape AI around their own requirements.

What next

Want to explore what smaller, specialized models could look like for your AI workloads?

• Talk to a Red Hatter: Explore Red Hat AI or Contact sales directly
• Explore the platform: Explore Red Hat OpenShift AI
• Try hands-on testing: Try our AI interactive demos
• Read our technical blog post: Smaller model, sharper insights: How Atos approached structured outputs
• Get technical quickstarts: View AI quickstarts