The latest MITRE ATT&CK® Enterprise Evaluations are out, featuring scenarios that emulate sophisticated actors like Scattered Spider and Mustang Panda. While every release of the findings is a significant event for the security community, this year’s evaluation highlights both new and recurring concerns for security professionals.
A Call for Clarity and Community
After years in the EDR/XDR industry, I’ve observed two concerning trends in the ATT&CK Evaluations.
First, vendor participation has dwindled from nearly 30 in past rounds to just 11 this year. This is a loss for customers, vendors, and the entire security community. As a company that integrates with all EDR vendors, we know their data is invaluable to the broader security narrative. It’s time to bring everyone back to the table. We all benefit from a deeper understanding of how these security clients behave, from their use of heuristics to their reliance on threat intelligence.
Second, the presentation of the results changes from year to year, making them difficult for even seasoned professionals to interpret. This inconsistency fuels the spread of misinformation based on cherry-picked data. The inaccessibility of the raw JSON file at the time of writing only compounds the issue. In its current state, each vendor interprets the data like a Rorschach test.
To restore clarity and authority, the MITRE ATT&CK Foundation should make the results more readable for the casual observer and host a webcast on release day to present its official findings.
The High-Stakes Game of EDR Evaluations
Why are vendors stepping back? These evaluations are high-pressure, high-stakes events. A great result offers a golden marketing opportunity; a poor one means months of damage control. This pressure exists because many customers accept vendor marketing claims without digging into the nuances of the results.
Participating also requires significant investment in engineering resources. Teams spend months preparing their product and working with the MITRE ATT&CK Foundation to interpret results and make configuration changes during the test. The process can feel like an 18th-century debutante ball. Everyone arrives in their finest attire, hoping to win favor.
This pressure often leads to a “MITRE Mode,” a version of the product specially tuned for the test, rather than the standard, “off-the-shelf” version a customer would use. For example, during the Turla evaluation in 2023, many vendors incorporated a leaked version of MITRE’s testing tool into their threat intelligence modules. Many are also accused of deploying resource-intensive configurations that would be unacceptable in a real-world environment.
These evaluations could foster greater transparency by adding a performance impact as a metric, ensuring high-fidelity detection does not come at the cost of business operations. This would be similar to how AVTest evaluates performance and false positives. Another step would be to test an off-the-shelf version from a reseller, a practice proven effective by researchers at the University of Piraeus in Greece.
MITRE must address the perception that some vendors are “cheating,” which discourages others who can’t commit similar resources from participating.
13 Tips to Be Your Own Analyst
Selecting a vendor based on a small sample of tests is ill advised. The evaluations break down only two malware variants into individual attacks, which does not accurately reflect how a product will perform in your environment or against other unknown threats. It’s like choosing a family dog based on which breed won the last Westminster agility test.
Use these 13 tips to interpret the results and select the right EDR vendor for your organization.
Our strongest defense is one built on a common language. The ATT&CK framework provides that language by mapping threats to specific tactics, techniques, and procedures (TTPs), allowing the security community to move forward together.
At Exabeam, this principle is fundamental. We use TTPs to enrich every alert with context, helping analysts understand the why behind a threat, not just the what. The log data from our EDR partners is the basis of that security story, which is why we are so committed to supporting both the MITRE Foundation and the EDR vendor community.
To our EDR partners: Let’s rejoin the MITRE ATT&CK Evaluations and recommit to the transparency that makes our industry stronger.
This commitment to a TTP-based framework is also why we built Outcomes Navigator. It allows our customers to visualize their security coverage for either use cases or directly against the ATT&CK framework and identify gaps. Understanding your defense is the first step to improving it. Exabeam Nova provides further insight into how your coverage has improved and where to focus next.
To learn more about how CISOs are using agentic AI like Exabeam Nova, download our white paper, A CISO’s Guide to the New Era of Agentic AI.
