Can AI diagnose like a team of doctors? New research puts it to the test

Rethinking How AI is Built and Tested

AI has the potential to transform health and healthcare, and progress is accelerating rapidly. The challenge now is to build and rigorously evaluate AI systems that translate advances in their clinical reasoning capabilities into meaningful impact.

Today’s paper, building on earlier work, introduces two foundations for that shift: the Sequential Diagnosis Benchmark (SDBench), which tests step-by-step diagnostic reasoning, and the Microsoft AI Diagnostic Orchestrator (MAI-DxO), our leading AI system on the benchmark.

Medical licensing exams such as the United States Medical Licensing Exams (USMLEs) are designed to assess physicians’ knowledge – not the realities of making a clinical diagnosis. In these tests, all information is presented upfront and a single answer is selected; in practice, clinicians must build the picture step by step, deciding what to ask, which tests to order and how each new finding changes the likely diagnosis. Such exams have become the common benchmarks for evaluating the performance of health AI.

That’s why we developed SDBench, which evaluates performance across two complementary sets of cases: 304 complex cases taken from The New England Journal of Medicine (NEJM) case proceedings and a set of more typical emergency-department presentations from MIMIC-IV-ED. Each becomes an interactive case in which AI models and/or physicians start with limited information and choose what to request next. The benchmark measures diagnostic accuracy, evidence requested and estimated test costs – assessing both the answer and the path taken.

We built MAI-DxO and tested it within SDBench. Rather than relying on one model for an immediate answer, it operates like a panel of virtual physicians: each proposing diagnoses, selecting useful questions or tests, and challenging the emerging view. MAI-DxO coordinates this exchange over several rounds of debate, updating its assessment as evidence arrives and weighing each test’s likely value against its cost.

What we found:

• On complex NEJM cases, MAI-DxO achieved 83% diagnostic accuracy. Physicians achieved 22% without external reference resources and 36% with them, making MAI-DxO about 3.7 and 2.3 times more accurate, respectively.
• On more typical emergency-department cases from MIMIC-IV-ED, MAI-DxO achieved 80% accuracy versus 47% for physicians – more than 1.7 times higher.
• When optimised more heavily for cost, MAI-DxO performance remained strong while choosing more cost-effective workups. On NEJM cases, one configuration achieved 81% accuracy at less than half the cost of comparative systems.
• Across both datasets, no other human or AI approach was both more accurate and less costly than MAI-DxO.
• These gains persisted – and improved – when we repeated the evaluation with newer models, including GPT-5.5, without changing the orchestration method, suggesting the approach is not tied to one model generation.

We are open-sourcing SDBench for select NEJM cases – including the benchmark, Gatekeeper, Judge and cost-estimation code – so researchers can reproduce the evaluation, test other models, and challenge or extend the findings.