Large language models may ace medical licensing exams, but they stumble at the real work of medicine, according to a trio of new benchmarks from Stanford researchers. MedHELM maps 121 common clinical tasks and finds top models posting their strongest win rates in note generation and patient communication, while lagging in clinical decision support and administrative workflows—areas where hospitals most need relief. Two execution-grounded tests underscore the gap between talk and action. In PhysicianBench, the best model completed just 46% of multi-step consult tasks on a single run and only 28% across repeated runs, often writing the “right” plan in a note but failing to place the corresponding EHR orders. HealthAdminBench shows similar shortcomings in back-office work: models can pass most subtasks yet falter on end-to-end completion due to coordination and memory lapses. The results suggest that progress will depend as much on agent architecture, tool use, and memory as on bigger models. By open-sourcing data, code, and leaderboards, the team invites health systems to contribute workflow-level tests that better mirror the realities of care.
Related articles:
Ethics and governance of artificial intelligence for health
DECIDE-AI: reporting guideline for the early-stage clinical evaluation of AI-based decision support systems
HL7 FHIR: standard for healthcare data interoperability
