Chat Generative Pretrained Transformer Fails the Multiple-Choice American College of Gastroenterology Self-Assessment Test : Official journal of the American College of Gastroenterology | ACG

Secondary Logo

Journal Logo

Advanced Search
BRIEF COMMUNICATION

Chat Generative Pretrained Transformer Fails the Multiple-Choice American College of Gastroenterology Self-Assessment Test

Suchman, Kelly MD1; Garg, Shashank MD2; Trindade, Arvind J. MD1,3

Author Information

1Zucker School of Medicine at Hofstra/Northwell, Long Island Jewish Medical Center, New Hyde Park, New York, USA;

2Arkansas Gastroenterology, North Little Rock, Arkansas, USA;

3Institute of Health Innovations and Outcomes Research, Feinstein Institutes for Medical Research, Northwell Health, Manhasset, New York, USA.

Correspondence: Arvind J. Trindade, MD. E-mail: [email protected].

The American Journal of Gastroenterology ():10.14309/ajg.0000000000002320, June 9, 2023. | DOI: 10.14309/ajg.0000000000002320

Abstract

INTRODUCTION: 

Chat Generative Pretrained Transformer (ChatGPT) is a natural language processing model that generates human-like text.

METHODS: 

ChatGPT-3 and ChatGPT-4 were used to answer the 2022 and 2021 American College of Gastroenterology self-assessment tests. The exact questions were inputted in both versions of ChatGPT. A score of 70% or higher was required to pass the assessment.

RESULTS: 

Overall, ChatGPT-3 scored 65.1% on 455 included questions and GPT-4 scored 62.4%.

DISCUSSION: 

ChatGPT did not pass the American College of Gastroenterology self-assessment test. We do not recommend its use for medical education in gastroenterology in its current form.

© 2023 by The American College of Gastroenterology

Full Text Access for Subscribers:

Log In Here
ACG Member Access
Log In Here

Not a Subscriber?

Buy
Request Permissions

You can read the full text of this article if you:

Access through Ovid