AI diagnostic accuracy falls up to 39 points in patient conversations

Key Points

1. A new MBZUAI-led benchmark found AI diagnostic accuracy drops 13 to 39 percentage points in conversation.
2. MedRoundsQA tests 15 models across 1,387 cases and 17 specialties over up to 20 rounds.
3. Multiple-choice scores overstate what diagnostic assistants can do with real, talking patients.

The latest:

AI model accuracy in medical diagnosis fell by 13 to 39 percentage points when the diagnosis had to be reached through back-and-forth conversation rather than a single-turn test, according to researchers at Mohamed bin Zayed University of Artificial Intelligence and partner institutions. Their benchmark, MedRoundsQA, was presented at the 2026 EMNLP conference in Budapest.

Details:

• The benchmark: MedRoundsQA was built to test whether language models can diagnose patients through multi-turn dialogue instead of multiple-choice questions, according to MBZUAI. It covers 1,387 cases across 17 medical specialties, each graded easy, medium or hard, so performance can be compared across difficulty levels rather than reported as a single score.
• How it works: Fifteen models were placed in the doctor role and allowed up to 20 rounds of questions and answers with a simulated patient, according to the researchers. The design forces models to decide what to ask, rather than selecting an answer from options already written for them.
• The patient personas: Simulated patients were varied by education level, emotional tone, language proficiency and occupation, the researchers said. That variation is what allows the benchmark to separate a model’s medical knowledge from its ability to extract information from a person who is anxious, imprecise, or unfamiliar with clinical vocabulary.
• The headline drop: Compared with single-turn evaluation, accuracy in the conversational setting fell between 13 and 39 percentage points across the models tested, according to the study. The size of the gap is the study’s central finding: the same model can look competent on a quiz and much weaker in dialogue.
• Best performer: Kimi-K2.5 recorded the highest score in the multi-turn evaluation at 42%, according to the researchers. Even the leading result leaves most conversational cases undiagnosed, placing the top of the field well below what the single-turn format suggests.
• Versus human doctors: On a subset of 60 difficult cases, three human doctors averaged 48% accuracy against 27% for the models, the study found. The comparison is narrow in scope but sets the hardest end of the benchmark as the clearest point of separation between clinicians and systems.
• The equity finding: Lowering the simulated patient’s education level cut model performance by seven to eight percentage points, according to the researchers, who described this as an equity risk that single-turn benchmarks do not surface. The effect appears without any change in the underlying medical case.
• Where it was presented: The work was presented at the Conference on Empirical Methods in Natural Language Processing in Budapest in 2026, according to MBZUAI, which credited researchers from the university alongside other institutions. The university did not name the partner institutions involved.

Background:

Medical AI capability has largely been measured on exam-style multiple-choice question sets, where models select from supplied answers. MedRoundsQA is designed around the opposite condition: the model must elicit the information itself before committing to a diagnosis.

Between the lines:

Two of the study’s results point the same way. A model that scores far lower when it must ask its own questions, and that loses seven to eight points when the patient is less educated, is being measured on interviewing skill rather than stored knowledge. That distinction matters for health systems evaluating diagnostic assistants, because the 48%-to-27% gap on hard cases appears exactly where errors are most costly.

What’s next

Watch whether other developers run their models against MedRoundsQA and publish multi-turn scores, and whether health regulators and hospital procurement teams begin requiring conversational evaluation alongside multiple-choice benchmark results.