A new artificial intelligence (AI) system called ChatGPT has passed the United States Medical Licensing Exam (USMLE), a highly standardized and regulated series of exams required for medical licensure in the US. This impressive feat was achieved through a study conducted by Tiffany Kung and her colleagues at AnsibleHealth in California. The USMLE assesses knowledge spanning most medical disciplines, including biochemistry, diagnostic reasoning, and bioethics.
ChatGPT is a large language model (LLM) that generates human-like writing by predicting word sequences using its internal processes. Unlike most chatbots, it cannot search the internet but instead relies solely on its internal processes to generate text. The authors of the study tested the software on 350 out of 376 public questions from the June 2022 USMLE release.
The results of the study were outstanding, with ChatGPT scoring between 52.4% and 75% across the three exams. The passing threshold each year is approximately 60%, and ChatGPT has surpassed this benchmark with ease. Furthermore, ChatGPT demonstrated 94.6% concordance across all its responses and produced at least one significant insight, something that was new, non-obvious, and clinically valid, for 88.9% of its answers.
ChatGPT outperformed its counterpart, PubMedGPT, a model trained exclusively on biomedical literature, which scored 50.8% on an older dataset of USMLE-style questions. The relatively small input size used in the study did limit the depth and range of analyses, but the authors believe that their findings provide a glimpse into ChatGPT’s potential to enhance medical education and eventually, clinical practice.
Clinicians at AnsibleHealth have already begun utilizing ChatGPT to rewrite jargon-heavy reports, making them easier for patients to comprehend. The authors of the study believe that ChatGPT’s ability to reach the passing score for the USMLE, a notoriously difficult exam, without any human reinforcement, marks a notable milestone in the maturation of clinical AI.
In fact, ChatGPT played a significant role in the writing of the study manuscript. The co-authors interacted with the AI system much like a colleague, asking it to synthesize, simplify, and offer counterpoints to drafts in progress. All of the co-authors valued ChatGPT’s input and found it to be a valuable asset in the research process.
In conclusion, the study conducted by Tiffany Kung and her colleagues at AnsibleHealth has shown that ChatGPT has the potential to revolutionize medical education and eventually, clinical practice. The AI system has surpassed the passing score for the USMLE, a highly standardized and regulated exam, without any human reinforcement, marking a notable milestone in the maturation of clinical AI.
For the latest tech news, follow Techbar Nepal on Facebook, and Instagram.
For Digital Products and Services: Maurisys Software.
