Microsoft's AI diagnoses twice as well as doctors?

A patient, hand sanitizer and one question

A man is hospitalized for alcohol withdrawal, and he's getting worse. An AI, OpenAI's o3 model used off the shelf, suspects antibiotic toxicity. It orders a brain MRI and an EEG, then gets it wrong. Simulated bill: $3,431.

Microsoft's system floats the idea of poisoning acquired in the hospital from the very first round. Before ordering anything, it asks the patient a question: have you been drinking hand sanitizer? The patient admits it. A targeted blood test confirms it, for $795.

That case appears in the first version of the study, posted on arXiv in June 2025. It shows what Microsoft is trying to measure: not an answer to a multiple-choice quiz, but a doctor, human or machine, who starts from three lines, asks questions, orders tests one at a time and pays for each. The team calls it sequential diagnosis.

The number making the rounds

On October 22, the work was published in the journal Science, co-signed among others by Mustafa Suleyman, the head of Microsoft AI. The paper's abstract gives the figure now repeated everywhere: the system, named MAI-DxO, finds the right diagnosis in 83.2% of cases drawn from the New England Journal of Medicine (NEJM), versus 36.1% for generalist doctors.

Suleyman tells TechCrunch he sees it as a step toward "medical superintelligence." According to TechCrunch, Microsoft defines it, in a text published alongside the paper, as the ability to outperform the best human health systems. Note the wording: health systems. Not a doctor.

Who the doctors were

The Science abstract says "generalists." TechCrunch is more specific: primary care physicians and general internists, each working alone. In the 2025 version there were 21 of them, 17 in primary care, with a median of about twelve years of experience. How many in 2026? Microsoft's blog post doesn't say, and we couldn't access the full text of the Science paper.

The cases themselves are NEJM clinicopathological conferences: clinical puzzles published for teaching, which the authors said they picked for difficulty and educational value. Faced with a cancer they can't name, a generalist refers the patient to an oncologist. The authors made that point themselves in 2025, noting that their participants would probably have sent the hardest cases to specialists. Specialists aren't part of the comparison.

It's a bit like a trivia final for experts where only general-knowledge contestants were invited. The score is true, but mostly it tells you who was on stage.

With or without a library

What's new in the Science version shows up in Microsoft's blog post. With no outside resources, doctors score 22%. With textbooks and internet access, 36%. Microsoft draws two ratios from that: the AI would be 3.7 times more accurate in the first case, 2.3 times in the second.

In 2025, only the harsher setup existed. Doctors were told not to use search engines, AI or online sources, because the NEJM cases can be found on the web. They topped out at 19.9%, and the abstract at the time claimed an AI four times better. Handing doctors back part of their tools took that ratio from nearly four to a little over two.

The blog post doesn't say whether they could ask a colleague, which is the first thing most people do with a hard case.

What the orchestrator really does

MAI-DxO isn't an AI model. It's an orchestrator: it has a language model play several virtual doctors who propose hypotheses, pick tests and argue with each other. Microsoft tested it on models from OpenAI, Google, Anthropic, Meta and DeepSeek.

Now, according to the study's figures as reported by TechCrunch, GPT-5 alone, with a short prompt, already hits 78.9% on the complex cases. The orchestrator adds about four points and trims the bill by around $3,000. The 2025 version said much the same with o3: 78.6% alone, 81.9% orchestrated. With those models, most of the gap with doctors comes from OpenAI's model, not from the layer Microsoft added on top.

The picture changes with other models. Again per TechCrunch, GPT-5.5 goes from 72% alone to nearly 87% with the orchestrator, and Gemini 2.5 Flash from 46.4% to nearly 69%. Microsoft's contribution is real, but it runs from four to more than twenty points depending on the engine underneath: a copilot, not the pilot.

Tests, not servers

TechCrunch puts the cost of GPT-5 alone at about $8,400 per case and presents it as the cost of running the AI. That's not what Microsoft describes. Its blog post talks about estimated test costs. Every consultation and every test ordered for the virtual patient adds to a fictional bill, calculated in 2025 from a US health system's price list. Computing, by contrast, came to roughly fifteen cents per case at the time.

The emergency room, and what's left to prove

Science adds a second testbed, closer to everyday practice: emergency department admissions drawn from MIMIC-IV-ED, a US hospital database. There, the AI stays clearly ahead: 80.3% versus 46.8% for doctors, according to the abstract. But per TechCrunch, the orchestrator adds almost nothing there compared with the bare model, in accuracy or in cost. With GPT-5.5, both versions land around 85%.

The Science abstract itself adds a caveat: these results were obtained in a controlled testing setup. In 2025, the authors spelled out what that setup can't see. No healthy patients among the cases, so no way to measure false alarms. And when a requested test wasn't in the published case, a model made up a plausible result.

The real test is announced with Boston's Beth Israel Deaconess Medical Center. Consenting patients will receive their usual care while also interacting with the system, whose recommendations will be compared with those of the doctors.

The question the authors asked

In 2025, the Microsoft team ended its discussion with a blunt question: should these systems be compared with individual doctors, or with whole teams of generalists and specialists, as in a hospital? Its own definition of medical superintelligence answers it: it's health systems they will have to be measured against. The study published in Science hasn't done that yet.