Microsoft AI Beat Generalist Doctors on Hard Cases, but Only in Simulation

A Microsoft AI system correctly diagnosed 83% of complex medical cases in a simulated test, compared with 36% for generalist physicians who could consult reference materials, according to a study published in the journal Science. The cases came from The New England Journal of Medicine, and the test was run on written case files, not on live patients.

The system, called MAI-DxO, has not been used in clinics. Microsoft is now preparing a study with Beth Israel Deaconess Medical Center in Boston to see how it performs alongside doctors treating real patients.

The central numbers

• 83% vs. 36%: MAI-DxO's accuracy on complex cases, against generalist doctors working individually with reference materials, as reported by AI Insider and International Business Times.
• 22%: The share of cases generalists diagnosed correctly without outside references, according to Microsoft AI's announcement as relayed by International Business Times.
• 304: The number of published New England Journal of Medicine cases used to build the test.
• 80% vs. 47%: Accuracy on emergency department cases drawn from a database of deidentified hospital records, for the AI and for generalist physicians respectively.
• $5,431 and $2,241: Estimated diagnostic costs for two versions of the system. The first reached 83.2% accuracy. A version with tighter cost limits cost less and kept 80.9% accuracy. The physicians' workups averaged $4,725.

How the test worked

Many medical AI tests are multiple-choice questions that hand over all the clinical facts at once. Microsoft's researchers built a different benchmark, which they call the Sequential Diagnosis Benchmark. The AI starts with a short description of a patient and must ask for history, examination findings, lab tests or imaging, much as a clinician would.

Inside MAI-DxO, several AI agents act like a panel of doctors. According to Gadget Review, one proposes possible diagnoses, one chooses tests, one challenges assumptions and one weighs the cost of each next step. Each test carries an estimated price, so the benchmark can track accuracy and spending together.

What is behind the results

The system works with different underlying language models. Microsoft tested models from OpenAI, Google, Anthropic, Meta and DeepSeek. International Business Times reported that GPT-5 reached about 79% on its own and just over 83% inside MAI-DxO. AI Insider reported that GPT-5.5 rose from 72% to nearly 87%, and that a weaker model, Gemini 2.5 Flash, improved from 46.4% to nearly 69%. According to AI Insider, much of the overall improvement comes from stronger base models, with the orchestration adding gains on top. The same outlet said GPT-5 and GPT-5.5 saw no meaningful gains on emergency department cases.

The cost figures are modeled estimates. International Business Times and Gadget Review both stressed that they are not savings shown in hospitals. Gadget Review also noted that Microsoft's research page lists other configurations, including 80% accuracy with one OpenAI model setup and 85.5% under a maximum-accuracy setting. The 83% figure therefore reflects one specific comparison, not a single universal result.

What this means for patients

For now, nothing changes in the exam room. The cases were historical and text-based, and the comparison involved individual generalists, not the specialist teams that usually handle such difficult cases. AI Insider and Gadget Review both noted that specialists and physician teams were not part of the comparison.

Gadget Review said the results do not show that the system is safer or more reliable than doctors treating actual patients, since curated cases cannot capture communication problems or fast-changing conditions in a real emergency department. It also reported that Microsoft executives have said clinical testing is required before any deployment.

Dominic King, vice president of Health at Microsoft AI, said the biggest challenge in health care is the widening gap between demand for medical expertise and its supply. Mustafa Suleyman, CEO of Microsoft AI, called the result a step toward “medical superintelligence.” Those are statements from Microsoft's own leaders, not findings of the study.

What happens next

The Beth Israel Deaconess study will test MAI-DxO in emergency and urgent care settings. Consenting patients will keep receiving standard care, and the system's assessments will be compared with those of treating physicians. King said he expects it to be the largest study of its kind, according to AI Insider. Researchers hope to learn where the system does well and where it struggles. Microsoft did not disclose a start date in the reports reviewed.

AI Insider also reported that Google recently published a similar Beth Israel study of its own diagnostic AI, AMIE, in The Lancet. Suleyman said Microsoft is working with the Mayo Clinic on a health care foundation model and aims to lead in clinical decision support tools.

Around Prague, based on reporting by AI Insider, International Business Times, Gadget Review