Microsoft AI tool hits 83% diagnostic accuracy versus 36% for doctors

Key Points

1. Microsoft researchers built a stepwise diagnostic AI that more than doubled generalist physicians' accuracy on complex cases.
2. The system was tested on 304 complex cases drawn from New England Journal of Medicine reports and emergency departments.
3. Researchers say it also picked cheaper, more efficient testing paths than conventional AI benchmarks measure.

The latest:

A Microsoft-built artificial intelligence tool reached roughly 83% accuracy on complex medical cases, against 36% for participating general physicians who had medical references and internet access, according to a study published in the journal Science. The tool works sequentially, gathering patient history, ordering tests and revising its diagnostic hypotheses as results arrive.

Details:

• The design: According to the study, the tool simulates a team of clinical experts rather than a single responder. It collects the patient’s medical history, determines which tests are required, evaluates the results, then continuously updates its diagnostic hypotheses until it settles on the most appropriate diagnosis.
• The test set: Researchers ran the system against 304 complex medical cases built on reports published in the New England Journal of Medicine, alongside additional cases taken from emergency departments. The two groups were scored separately rather than pooled into a single accuracy figure.
• The numbers: On complex cases the tool scored close to 83% accuracy versus 36% for the physicians, a gap of more than double. In emergency cases it recorded roughly 80% against 47% for doctors, a narrower but still substantial margin.
• The comparison group: The physicians measured against the system were generalists, and researchers noted they were allowed to consult medical references and the internet while working the cases. No specialist comparison group was reported.
• Cost factor: Researchers said the system does not optimize for accuracy alone. It weighs the cost of the tests it requests, and during the trials it demonstrated an ability to select diagnostic pathways that were both more efficient and less expensive than the alternatives available to it.
• Why the method differs: Conventional evaluations of medical AI usually rely on multiple-choice questions and information supplied in advance. Researchers said this approach instead reproduces real-world diagnosis, where evidence is gathered and assessed in stages before any conclusion is reached.
• Publication: The findings were published in the journal Science. The researchers did not announce a timeline for clinical deployment, nor did they identify any hospital systems involved in testing the tool beyond the emergency department cases used in the evaluation.

Background:

Medical AI systems have typically been benchmarked on exam-style questions with all relevant information disclosed upfront, a format that rewards recall rather than the sequential evidence-gathering that defines actual clinical diagnosis.

Between the lines:

The cost result may matter as much as the accuracy result. A system that reaches 83% while also choosing cheaper testing routes targets two separate pressures on health systems at once. But the benchmark compares the tool against general physicians, not specialists, and the complex cases were drawn from published journal reports — material with a documented, settled answer, unlike an undiagnosed patient.

What’s next

Watch for independent replication of the 83% and 80% figures, any peer response in Science, and whether Microsoft names a clinical partner or regulatory pathway for testing the tool in live hospital settings.