The Dark Side of Medical Chatbots: Why AI Diagnoses Can't Replace Human Doctors

Can Large Language Models Replace Human Doctors?

<p Large language models, like those behind ChatGPT, have been making waves in the medical community with their impressive performance in medical exams. However, a team of researchers from the Technical University of Munich (TUM) has raised concerns about the reliability of these models in making accurate diagnoses.

In a recent study published in Nature Medicine, the researchers tested the capabilities of large language models in making diagnoses and found that they often failed to follow treatment guidelines, ordering unnecessary examinations that could put patients' lives at risk.

The Test: Simulating Real-World Scenarios

The researchers used anonymized patient data from a clinic in the USA to test the models. They selected 2,400 cases of patients who came to the emergency room with abdominal pain. Each case description ended with one of four diagnoses and a treatment plan.

The models were tasked with simulating the decision-making process of real doctors, from ordering blood tests to making a diagnosis and creating a treatment plan. However, the results were far from impressive.

Failures in Diagnosis and Treatment

The researchers found that none of the large language models consistently requested all the necessary examinations. In fact, the models' diagnoses became less accurate the more information they had about the case.

In one extreme case, a model correctly diagnosed gallbladder inflammation in only 13% of cases. The models also failed to follow treatment guidelines, sometimes ordering examinations that would have had serious health consequences for real patients.

Comparison with Human Doctors

The researchers compared the AI diagnoses with those made by four human doctors and found that the latter were correct in 89% of cases, while the best large language model achieved only 73%.

The models also lacked robustness, with the diagnosis depending on the order in which the information was received. Linguistic subtleties also influenced the result, making the models unreliable in real-world scenarios.

Why Open-Source Models Are the Way Forward

The researchers explicitly did not test commercial large language models like ChatGPT due to data protection concerns and the importance of using open-source software in the healthcare sector.

Only open-source models provide hospitals with sufficient control and knowledge to ensure patient safety. With open-source models, researchers can know what data was used to train them, ensuring fair evaluations.

Rapid Progress, But Limitations Remain

Despite the current shortcomings, the researchers see potential in the technology. Large language models could become important tools for doctors, but only if their limitations are acknowledged and addressed.

Frequently Asked Questions

Can large language models replace human doctors?

No, current large language models are not reliable enough to replace human doctors in making accurate diagnoses and creating treatment plans.

Why are open-source models important in healthcare?

Open-source models provide hospitals with control and knowledge to ensure patient safety and allow for fair evaluations.

What are the limitations of large language models in healthcare?

Large language models often fail to follow treatment guidelines, order unnecessary examinations, and lack robustness in their diagnoses.

Conclusion

While large language models have made impressive strides in medical exams, they are not yet ready to replace human doctors in making accurate diagnoses and creating treatment plans. It's essential to acknowledge their limitations and address them before integrating these models into everyday clinical practice.