A new study compares the performance of human and AI doctors working in the ER.
Science Journal published a study led by physicians and computer scientists at Harvard Medical School, Stanford University, the University of Minnesota Medical School, Beth Israel Deaconess Medical Center, and more. The study compared how Open AI’s o1 large language model (LLM) performed against various physicians “at different levels of training and experience on a variety of clinical cases, ranging from published patient vignettes to evaluations of brand-new emergency room patients, as well as on clinical tasks including both diagnosis and planning of clinical management.”
The Guardian reported that one experiment examined 76 patients at the emergency room of Beth Israel Deaconess Medical Center in Boston, MA, comparing diagnoses made by AI and human doctors. Both were given the same electronic health records, including vital signs, demographic data, and patient notes. The AI identified the exact or near-correct diagnosis in 67% of cases, outperforming the doctors, who were correct about 50% to 55% of the time. Additionally, with more information, AI’s accuracy increased to 82%, compared to human doctors, who ranged from 70% to 79%.
“At each diagnostic touchpoint, o1 either performed nominally better than or on par with the two attending physicians,” the study says.
The study also noted the differences “were especially pronounced at the first diagnostic touchpoint (initial ER triage), where there is the least information available about the patient and the most urgency to make the correct decision.”
“We tested the AI model against virtually every benchmark, and it eclipsed both prior models and our physician baselines,” said Arjun Manrai, one of the study’s lead authors who works at an AI lab at Harvard Medical School, per TechCrunch.
The study also showed that AI outperformed a larger group of human doctors when outlining longer-term care plans, including end-of-life considerations and antibiotic treatment strategies, per The Guardian. The study also notes that AI only examined patient data shared via text, and physical appearance or distress levels were not considered.
“I don’t think our findings mean that AI replaces doctors,” said Manrai, per The Guardian. “I think it does mean that we’re witnessing a really profound change in technology that will reshape medicine.”
Dr. Adam Rodman of Boston’s Beth Israel Deaconess Medical Center, who is also a lead author of the study, told the outlet that AI LLMs are some of “the most impactful technologies in decades” and predicts AI will not replace human doctors but will work in an updated “triadic care model … the doctor, the patient, and an artificial intelligence system”.
