Compiled by the editorial desk with reference to the JAMA Network Open study and public statements from the researchers.

In a head-to-head comparison, an AI chatbot demonstrated superior diagnostic accuracy compared to a group of 50 physicians, according to a study published in JAMA Network Open. The findings, which surprised even the researchers, have sparked a fresh debate about the role of artificial intelligence in clinical decision-making.

The experiment involved 50 doctors who were asked to diagnose medical conditions based on case reports. Some were randomly assigned to use ChatGPT as a decision-support tool. Their performance was evaluated not only on the correctness of their final diagnosis but also on the clarity of their reasoning. Physicians working alone scored an average of 74 percent, while those collaborating with the AI scored 76 percent. In contrast, ChatGPT operating independently achieved an average score of 90 percent, surpassing both groups by a significant margin.

Dr. Adam Rodman, an internal medicine specialist at Beth Israel Deaconess Medical Center in Boston and a coauthor of the study, expressed his astonishment at the outcome. “I was shocked at the results,” he said during an interview on the New York Times podcast Hard Fork. “My hypothesis going in was that people using [ChatGPT] would be the best. So I am surprised by this.”

The study was small in scale—only six case studies were examined—but the implications are far-reaching. The cases were intentionally challenging, based on real patients, and yet the AI model prevailed. Rodman suggested that the results may reflect not only the AI's capabilities but also human doctors' reluctance to accept its suggestions. Physicians using ChatGPT may have dismissed its second opinions, sticking with their initial diagnoses.

Why Doctors Lagged Behind

Another possible explanation is that the doctors were not sufficiently familiar with using the AI tool. However, Rodman cautioned against concluding that ChatGPT is inherently more competent than human physicians. He noted that the case reports were meticulously prepared by expert clinicians, who organized the information in a way that made it easier for the AI to process. “The difference is that the people who put the case together, like, the information, if you want to think about the prompts, were expert clinicians,” he explained. “We organize it in such a way.”

In other words, the human effort involved in gathering and presenting medical data was crucial to the AI's success—a task that AI cannot yet perform on its own. Rodman compared the situation to a chef taking credit for a meal when the recipe was provided by someone else.

The study's primary aim was to assess whether ChatGPT could effectively assist doctors, not to prove AI superiority. Indeed, previous research has shown ChatGPT to be unreliable in diagnostic tasks. Notably, this experiment used an older version of the AI model, GPT-4 Turbo, which may yield even more impressive results if repeated with current technology.

“Maybe AI models are better at making diagnoses than human doctors. But I don’t think that’s the case with GPT-4 Turbo, which was the model that was used here,” Rodman said. “But it’s going to be true at some point, and we’re quickly approaching that.”

As AI continues to evolve, its integration into healthcare raises important questions about the balance between technological assistance and human expertise. While this study offers a glimpse into the potential of AI, it also underscores the need for careful evaluation and training before such tools can be reliably deployed in clinical practice.