Microsoft Says Its Diagnostic AI Outperforms Physicians on Curated Cases
Microsoft says it has built an AI system that diagnoses conditions up to four times more accurately than human doctors, according to The Verge. Microsoft's findings on the system have been published in the journal Science.
The system, called the Diagnostic Orchestrator, runs several AI agents, each handling a different step. They work through a patient's symptoms and medical history, decide which tests to order, and converge on a diagnosis. It is powered by OpenAI's GPT-5. Rather than answering a single prompt, it runs as a sequential, agentic loop: each first-pass hypothesis shapes the next test, much as a clinician works through a case.
Microsoft reports the following accuracy figures. On case reports from the New England Journal of Medicine, the system reached around 83% diagnostic accuracy. Generalist physicians with access to reference materials scored roughly 36%, and those without access scored 22%. On emergency department cases from Beth Israel Deaconess Medical Center, the system scored 80%, against an average of 47% for human physicians.
These are case-based comparisons, not prospective clinical trials, and that distinction matters. Written case reports are curated and contain enough detail for a diagnosis to be reached in hindsight. Real patients arrive with incomplete information, and a wrong call carries different costs across specialties.
Microsoft compiled the test cases into its own benchmark, SDBench, and says it plans to open-source it. If that happens, outside groups could rerun the evaluation, change the case mix, and check whether the margins hold.
The work has a predecessor. Microsoft Research published its MAI-DxO findings in 2025 under the title "Sequential Diagnosis with Language Models." In that paper, MAI-DxO paired with OpenAI's o3 model reached 80% diagnostic accuracy, against a 20% average for generalist physicians. The current announcement extends that research to a different system configuration and a broader set of cases.
Suleyman was explicit that the system needs clinical validation before wide deployment. That qualifier carries most of the weight. A benchmark score says little about calibration, failure modes, performance across demographic subgroups, or what happens when clinicians over-trust the output.
In my view, the more useful question for hospital technology teams is whether the system's reasoning can be audited and its uncertainty surfaced at the point of decision. A tool that orders the right test and explains why is easier to deploy than one that simply arrives at the right label.
The broader context here is that diagnostic support has long been one of the more plausible clinical uses for large language models, since the task centres on information gathering and differential reasoning. If validation holds in prospective settings, the likely gain is fewer missed diagnoses in high-volume emergency departments, where the 47% physician average suggests considerable room for support tools. Whether that gain materialises depends on the validation Suleyman described, not on the benchmark figures alone.
