AIThis post was created with the assistance of artificial intelligence (AI).

🔍 Read the full analysis: Microsoft Says Its AI Beats Doctors, But What Did The Study Test? on ThorstenMeyerAI.com

Buying for a business?Offer from Amazon

Get business pricing on tech for your team

  • Business-only prices and quantity discounts
  • Tax-exempt purchasing
  • Multiple users, one account, clear invoices
As an affiliate, we earn on qualifying purchases.

TL;DR

Microsoft’s peer-reviewed study reports that MAI-DxO achieved 83.2% diagnostic accuracy on 304 difficult medical cases, compared with 36.1% for generalist doctors using reference materials. The test used historical, text-based cases in a simulator; it did not compare the system with specialists or test patient outcomes in clinical care.

Microsoft’s MAI-DxO diagnostic system scored 83.2% on 304 difficult medical cases in a study published October 22 in the peer-reviewed journal Science, compared with 36.1% for generalist physicians using textbooks and internet resources. The result is a benchmark comparison, not evidence that the system outperforms doctors in treating patients: researchers tested historical, text-only case records in a simulator, without specialists, physical examinations or real-time clinical care.

The paper, titled Sequential diagnosis with language models, describes Microsoft’s Sequential Diagnosis Benchmark, which gives a doctor or AI a short case summary and requires it to request additional information. A simulated gatekeeper supplies details from the case file. The benchmark scores the final diagnosis and the estimated cost of the questions and tests used to reach it. Microsoft says MAI-DxO works as a panel of virtual physicians that proposes diagnoses, selects further questions or tests, and evaluates options over several rounds.

For the 304 complex cases drawn from the New England Journal of Medicine, Microsoft reports 83.2% accuracy for MAI-DxO. The reported physician averages were 36.1% with textbooks and internet access and 22.3% without those references. In a separate set of 325 emergency department cases, the AI scored 80.3%, compared with 46.8% for physicians. These are results reported by the paper’s Microsoft-affiliated authors; publication in a peer-reviewed journal is not the same as independent replication.

The study also suggests that much of the performance came from the underlying language model rather than the orchestration system alone. Microsoft’s chart gives GPT-5 a 78.9% score on the difficult cases without MAI-DxO, at an estimated $8,409 in tests per case. The source account says the orchestration layer added about four percentage points and reduced estimated test costs. These figures concern performance and modeled costs within the benchmark, not observed savings in hospitals.

At a glance
reportWhen: Published October 22, 2026; a prospecti…
The developmentA Science paper published October 22 reports Microsoft’s results for MAI-DxO, a diagnostic AI system evaluated on difficult medical cases.
Thorsten Meyer AI
Science · 22 Oct 2026

Infographic 01 · The scoreboard

83% vs. 36%: the number, and who it was measured against.

Microsoft’s diagnostic system beat individual generalist physicians by a wide margin on a simulated, step-by-step test. The comparison group decides how much that says.

83.2%MAI-DxO on 304 hard NEJM cases
36.1%generalist physicians with textbooks and internet
22.3%generalist physicians without references
80.3 / 46.8emergency cases: MAI-DxO vs. physicians, in %

Hard cases (NEJM)

Diagnostic accuracy, 304 case records

  • MAI-DxOno test budget83.2%
  • GPT-5 alonesame benchmark, no orchestration78.9%
  • Physicians, open booktextbooks and internet36.1%
  • Physicians, closed bookno references22.3%

Physician averages rest on 662 (open book) and 764 (closed book) case attempts, per Microsoft’s chart data.

Who was compared

The fine print behind “beats doctors”

  • Generalists only: primary care doctors and hospital generalists. No specialists, no teams.
  • Working alone: no colleagues; in the preprint also no search engines or language models.
  • Text-only, historical cases: no real patients, no physical exam, no imaging.
  • Hard by design: rare teaching cases; no healthy patients, so false alarms were not measured.
  • Self-reported: all 15 authors are listed with Microsoft affiliations; peer-reviewed, not independently replicated.

Sources: Science 394, 423–428 (22 Oct 2026); Microsoft AI, “Building Medical Superintelligence” (22 Oct 2026) and its charts; physician cohort details from the arXiv 2506.22405 preprint (2025).
Powered by Thorsten Meyer AI

What the Accuracy Gap Shows

The reported gap matters because diagnostic errors can affect treatment decisions, delays and the use of medical tests. The benchmark also attempts to model a part of diagnosis that many exam-style AI evaluations leave out: deciding what information to seek next, not just selecting an answer after every fact has been provided. Its cost scoring reflects the practical trade-off between gathering more information and reaching a diagnosis.

But an accuracy score on selected cases cannot by itself show that an AI system improves care. The physicians were individual primary care doctors and hospital generalists; the comparison did not include specialists, colleagues working as a team or doctors using a hospital’s full diagnostic resources. The cases were unusually challenging, and the source account says the collection did not include healthy patients or harmless explanations. That limits what the results say about false alarms, unnecessary testing and routine cases.

For hospitals and patients, the findings are best read as evidence that MAI-DxO performed well on a particular diagnostic benchmark. They do not establish that it is safe, cost-effective or more accurate than a clinician in day-to-day care. Those questions require testing in clinical settings, where patient histories may be incomplete, examination findings matter and a wrong answer can have consequences.

Products Worth Considering

Bose TV Speaker Compact Soundbar with Bluetooth and HDMI-ARC, Black

Bose TV Speaker Compact Soundbar with Bluetooth and HDMI-ARC, Black

Bose TV Speaker (2nd Gen) Compact Soundbar with Bluetooth and HDMI Connectivity

Bose TV Speaker (2nd Gen) Compact Soundbar with Bluetooth and HDMI Connectivity

As an affiliate, we earn on qualifying purchases.

How the Benchmark Worked

In conventional medical question-answering tests, a system often receives a complete case and chooses a diagnosis. Microsoft’s benchmark instead starts with a summary and makes each additional question or test a decision. The design aims to imitate the sequential nature of diagnosis, while keeping the case information inside a simulated environment rather than involving patients.

The source account describes several methodological details from the 2025 preprint, and cautions that the final Science paper may differ. In that earlier version, a separate model judged diagnoses against the published answer on a five-point scale; physician re-grading showed good, but imperfect, agreement. When a requested test result was absent from a record, the gatekeeper generated a synthetic result. Test prices were estimates mapped to billing codes using a 2023 U.S. hospital price list, with a fixed $300 physician-visit charge. The authors described these costs as first-order approximations.

The comparison also changed between the earlier preprint and the published study. The preprint reportedly compared an earlier model with doctors on a smaller held-out set and presented a larger performance gap. The October 2026 Science paper reports the results cited here, including an open-reference physician group. Readers should use the published figures rather than treating earlier preprint results as the final comparison.

“The system was reported as 1.7 times as accurate as physicians on emergency department cases and more than twice as accurate as physicians using textbooks and the internet on complex cases.”

— Microsoft, in its blog post, “Building Medical Superintelligence”

Limits Beyond the Simulator

The published results do not establish how MAI-DxO would perform with real patients, changing symptoms, physical examinations, imaging or records from different health systems. The source account says the comparison group consisted of generalists working alone, while difficult cases may be referred to specialists or handled by teams. It is also unclear from the supplied material whether outside researchers have independently reproduced the Science results.

Other open questions include how often the system would make harmful errors or recommend unnecessary tests in a broader patient population. The case set described in the source did not include healthy people or benign explanations, so it could not measure false alarms. Estimated benchmark costs are not evidence of actual savings, and the reported diagnostic accuracy does not show whether patients would recover more quickly or experience better outcomes.

The source reports that a prospective study is planned at Beth Israel Deaconess Medical Center, where consenting patients would interact with the system alongside normal care. Its design, size, timeline and results are not provided here, so the effect of that research remains unknown.

Testing With Patients in Boston

The next reported step is a prospective study at Beth Israel Deaconess Medical Center in Boston. According to the source account, consenting patients are expected to interact with MAI-DxO while receiving their usual care. That setting could provide evidence about how the system performs alongside clinicians, beyond the historical records and simulated test requests used in the published benchmark.

Details of the study’s protocol and schedule were not included in the source material. Until results from that work or independent evaluations are available, the current evidence remains a peer-reviewed benchmark report by Microsoft-affiliated researchers—not a demonstration that MAI-DxO can replace doctors or improve outcomes in ordinary medical practice.

Key Questions

What did Microsoft’s study test?

It tested MAI-DxO on historical medical cases, asking the system to request information and tests before giving a diagnosis. The comparison included generalist physicians working on the same type of cases.

Did the study show that the AI is better than doctors in hospitals?

No. The study reported higher scores on its simulator benchmark. It did not test whether MAI-DxO provides better care than doctors treating real patients in hospitals.

How did the physicians score?

On 304 difficult cases, Microsoft reported 36.1% accuracy for generalists using textbooks and internet resources and 22.3% without them, compared with 83.2% for MAI-DxO. On a separate set of 325 emergency department cases, the reported scores were 46.8% for physicians and 80.3% for the AI.

Was the study independently verified?

The paper appeared in the peer-reviewed journal Science, but the supplied source says all 15 authors were affiliated with Microsoft. Peer review does not establish that independent researchers have reproduced the results.

What evidence comes next?

The source reports a planned prospective study at Beth Israel Deaconess Medical Center in Boston, where consenting patients would use the system alongside normal care. The study’s timing, protocol and findings have not been provided.

Source: ThorstenMeyerAI.com

You May Also Like

AI Hospital Command Centers Are Closer Than You Think

The future of healthcare is evolving rapidly with AI hospital command centers, and understanding their impact could transform how hospitals operate—discover more inside.

AI for Predictive Migraine Management

AI for Predictive Migraine Management analyzes your data to forecast attacks and personalize treatment, offering potential relief—discover how it can transform your care.

AI-Designed Drug Moves to Human Trials in Record Time

An AI-designed drug advances rapidly into human trials, promising a new era in medicine, but questions remain about safety and regulatory adaptation.

What to Look for in a Sleep Tracker With AI Coaching

AIThis post was created with the assistance of artificial intelligence (AI).To find…