Skip to main content
← Back to Blog
LLM Tech7 min read

Microsoft’s Diagnostic Agent Passed a Harder Test. The Hospital Test Comes Next.

Microsoft’s diagnostic agent outperformed physicians in sequential case simulations, but its real clinical test is only beginning in a prospective study.

Microsoft's diagnostic-agent research has moved from a preprint into Science, with a broader evaluation that includes ordinary emergency-department presentations rather than only medicine's most difficult teaching cases. The system, called MAI-DxO, does not receive a complete patient record and answer in one shot. It asks for history, selects examinations and tests, updates a differential diagnosis, and decides when enough evidence exists to commit.

Microsoft reports that MAI-DxO reached 83 percent diagnostic accuracy on complex cases from The New England Journal of Medicine, compared with 36 percent for physicians who could use textbooks and the internet. On cases derived from the MIMIC-IV-ED emergency-care database, it reports 80 percent accuracy versus 47 percent for physicians. Those are striking benchmark results, not proof that an autonomous system is ready to diagnose patients.

The benchmark makes information gathering part of the test

Most medical language-model tests are static. They present a polished vignette containing the relevant history, examination findings, laboratory results, and imaging, then ask for an answer. Real diagnosis is an information problem before it is an answer problem: the clinician must decide what to ask, which tests are worth their cost and burden, and which early hypothesis should be abandoned.

Microsoft's Sequential Diagnosis Benchmark, or SDBench, converts cases into interactive encounters. A physician or AI begins with limited information. A gatekeeper model reveals findings only after a specific request and can synthesize case-consistent results for tests absent from the published narrative. The evaluation then scores the final diagnosis, the evidence requested, and the estimated cost of the workup.

The original benchmark used 304 clinicopathological cases published by the New England Journal of Medicine. These cases are deliberately selected for diagnostic difficulty and instructional value. Microsoft held out 56 cases published in 2024 and 2025 until its method was finalized, reducing the chance that either model memorization or framework tuning explains the result. The newer Sciencepublication also reports testing on more typical emergency-department cases from MIMIC-IV-ED.

Orchestration changed the path, not the underlying model

MAI-DxO is an orchestration layer rather than a newly trained medical foundation model. One underlying language model plays five clinical roles. A hypothesis role maintains the differential; other roles propose questions and tests, challenge the emerging diagnosis, and consider cost and invasiveness. The orchestrator coordinates several rounds and exposes new evidence between them.

That separation matters. In the earlier study, the same structure improved models from multiple providers, with an average accuracy gain of 11 percentage points. The strongest reasoning models gained less raw accuracy but became more cost-efficient. Microsoft says the updated evaluation repeated the method with newer models, including GPT-5.5, without changing the orchestration design.

On the difficult NEJM cases, Microsoft now reports 83 percent accuracy for MAI-DxO, compared with 22 percent for physicians without external references and 36 percent for physicians allowed to use them. A cost-optimized configuration retained 81 percent accuracy at less than half the estimated cost of comparison systems. On the emergency-department set, the reported 80 percent versus 47 percent result narrows the gap but addresses a more representative clinical setting.

The comparison is important and still artificial

The physician comparison should not be read as “AI replaces doctors.” The NEJM cases are rare, confusing, and written after the outcome is known. Each participating physician completed only a subset of cases. The benchmark's gatekeeper is itself a language model, which means the evaluated system interacts with a simulation of the record rather than a patient, a nurse, a laboratory, or an imperfect electronic chart.

Estimated test prices are useful for comparing strategies inside the benchmark, but they do not capture waiting time, false-positive cascades, local availability, insurance rules, patient preferences, physical examination quality, or the harm of delaying a time-sensitive treatment. Diagnostic accuracy also leaves out calibration: a system that reaches the right answer must still communicate uncertainty and recognize when the case is outside its competence.

The researchers acknowledge the central limitation. Their results do not establish clinical efficacy. Microsoft is beginning prospective work with Beth Israel Deaconess Medical Center in which patients will interact with the system and its performance will be compared with physicians' independent assessments. That is the evidence needed before claims about better care can replace claims about better benchmark performance.

Persistent state is becoming part of clinical reasoning

The technical lesson extends beyond medicine. MAI-DxO performs well because it maintains a changing working state: candidate diagnoses, evidence for and against them, tests already ordered, costs incurred, and unanswered questions. Each new result must revise that state without erasing the provenance of earlier observations.

A production clinical agent would need a stronger version of that memory. It would have to distinguish patient statements from verified measurements, preserve timestamps and sources, record corrections, expose contradictions, and enforce access and retention policies. A plausible diagnosis cannot be allowed to harden into a patient fact merely because an agent proposed it during deliberation.

Microsoft's study therefore says less about a single “doctor model” than about system architecture. The model supplies medical reasoning, while the orchestrator controls how evidence is acquired, challenged, priced, and remembered. The publication offers strong evidence that this structure improves performance in simulation. The forthcoming prospective study must determine whether it remains useful when the evidence is incomplete, the consequences are real, and the patient is not a benchmark.

Sources

Your AI remembers everything. Everywhere.

Unimatrix gives you a shared, durable memory layer across Claude Desktop, Cursor, ChatGPT, and Gemini. Setup in 2 minutes. Free and paid plans available.

Looking for developer resources?

Browse our catalog of 500+ tested AI prompt profiles covering DevOps, data modeling, agent behaviors, and API wrappers. Have a prompt to share? Submit your own for review by our librarian to be featured. Completely free, no registration required. Browse prompt libraries →

Keep reading