Microsoft’s AI Doctor Claim: How To Read The Study Results

🔍 Read the full analysis: Microsoft’s AI Doctor Claim: How To Read The Study Results on ThorstenMeyerAI.com

Before you orderOffer from Amazon

Get hardware and tech essentials delivered free with Prime

• Fast, free delivery on millions of items
• Prime Video, Amazon Music and more included
• Member-only deals all year

Start your free Prime trial →Free trial for eligible customers · Cancel anytime

As an affiliate, we earn on qualifying purchases.Hide

Microsoft’s MAI-DxO diagnostic AI scored 83.2% on a set of 304 difficult medical cases, compared with 36.1% for generalist physicians using textbooks and the internet, according to a peer-reviewed paper published in Science on October 22. The study tested historical written cases in a simulator, not diagnosis or treatment of real patients, so its results do not establish how the system would perform in routine clinical care.

The paper, titled “Sequential diagnosis with language models,” evaluates Microsoft’s Sequential Diagnosis Benchmark. Each case begins with a short description. The system or doctor must request further information—such as symptoms, test results or imaging details—from a simulated gatekeeper holding the case record. The benchmark scores the final diagnosis and estimates the cost of the tests requested along the way.

Microsoft reported 83.2% diagnostic accuracy for MAI-DxO on the 304 New England Journal of Medicine cases. The paper’s comparison figures were 36.1% for generalists with reference materials and 22.3% without them. On a separate set of 325 emergency department cases, Microsoft reported scores of 80.3% for the AI and 46.8% for doctors. These are results reported by the authors, who are all affiliated with Microsoft.

The AI system combines a language model with an orchestration process in which virtual agents propose diagnoses, choose follow-up questions or tests, and challenge one another. But the paper’s figures also indicate that the underlying model accounts for much of the performance: GPT-5 alone scored 78.9% on the hard cases. Microsoft’s chart estimated test costs at $8,409 per case for GPT-5 alone; the supplied source says the orchestration layer added about four percentage points and reduced estimated costs, but does not give the corresponding final cost figure.

What the Scores Mean for Care

The results matter because diagnostic AI is often tested on questions where all the relevant information is already provided. This benchmark instead asks the system to decide what information to seek next, a decision that is part of clinical work. A system that can ask useful questions and avoid unnecessary tests could eventually support clinicians, but this study does not show that it can safely do so in practice.

The reported gap should not be read as a direct comparison between AI and the full range of medical care. The physician group consisted of generalists working individually, while the AI used a multi-agent setup. The cases were selected as difficult teaching examples, and the comparison did not include specialists or physician teams. Those choices shape what the scores can tell hospitals, insurers and patients.

Cost estimates also require caution. The benchmark priced tests using billing codes and a 2023 U.S. hospital price list, along with a fixed physician-visit charge. The authors describe these as first-order approximations. They do not show what a hospital would actually spend if it adopted the system, including staffing, integration, follow-up care or the cost of errors.

How the Benchmark Was Built

Microsoft’s paper describes a different test from a standard medical knowledge exam. In the Sequential Diagnosis Benchmark, information is released only when requested, and tests carry estimated costs. That structure is intended to reflect part of the diagnostic process: deciding which question or test could help distinguish between possible explanations.

The Science paper was published on October 22, 2026. An earlier, non-peer-reviewed version appeared on arXiv in June 2025 and reported a more dramatic comparison, using an earlier AI model. The source material notes that some methodological details come from that preprint because the full final paper is behind a paywall; details may differ in the published version.

Microsoft’s blog framed the new findings as evidence of strong performance relative to physicians. Peer review means a paper has been evaluated for its methods and reporting; it is not independent replication of the results. The authors’ Microsoft affiliations are relevant when interpreting claims about Microsoft’s own system.

Limits the Study Leaves Open

The benchmark did not involve real patients, physical examinations or live imaging. It therefore cannot establish whether MAI-DxO would make safe decisions in a clinic, how it would handle incomplete or contradictory information, or whether clinicians would accept its recommendations. The study also did not test the AI against specialist teams, which may be involved in care for complex cases.

Other limits affect how accuracy should be interpreted. In the benchmark, a judge model scored diagnoses against the published final diagnosis; physician re-grading of a sample showed good but imperfect agreement, according to the source material. When a requested test result was absent from the case record, the gatekeeper generated a synthetic result. The cases also lacked healthy patients and harmless explanations, meaning the test could not measure false alarms in a representative patient population.

It remains unclear whether independent researchers will reproduce the scores, how performance would vary across patient groups and common conditions, and whether cost estimates would hold in ordinary clinical workflows. The study does not show that the system improves patient outcomes or reduces total healthcare spending.

The Planned Patient Study

Microsoft says a prospective study at Beth Israel Deaconess Medical Center in Boston is planned. Consenting patients are expected to interact with the system alongside their usual care. That would move evaluation beyond historical written cases, although the source material does not provide a start date, enrollment target or detailed study protocol.

Results from that research would need to clarify how the system behaves with real patient information and how its suggestions relate to clinicians’ decisions and patient outcomes. Until then, the Science scores are best understood as benchmark results on selected cases, not proof that MAI-DxO can replace doctors or is ready for unsupervised use.

Key Questions

What did Microsoft’s MAI-DxO score on the difficult-case benchmark?

Microsoft reported 83.2% accuracy on 304 difficult New England Journal of Medicine cases. Generalist physicians scored 36.1% with reference materials and 22.3% without them, according to the paper.

Was the system tested on real patients?

No. The study used historical, text-based cases in a simulator. It did not test diagnosis or treatment during real patient care.

Were specialists included in the comparison?

The comparison group was made up of generalist physicians working alone. The supplied study details say it did not include specialists or physician teams.

Does peer review mean the results have been independently confirmed?

No. The paper was published in the peer-reviewed journal Science, but the authors are affiliated with Microsoft. Peer review is not independent replication of the reported results.

What is expected to happen next?

Microsoft says a prospective study is planned at Beth Israel Deaconess Medical Center, where consenting patients would use the system alongside their usual care. Timing and further study details were not provided in the supplied material.

Source: ThorstenMeyerAI.com