Will ChatGPT Match to Your Program?

Abstract

ChatGPT and other artificial intelligence word prediction large database models are now readily available to the public. Program directors should be aware of the general features of this technology and consider its effect in graduate medical education, including the preparation of materials such as personal statements. The authors provide a sample ChatGPT-generated personal statement and general considerations for program directors and other graduate medical education stakeholders. The authors advocate that programs and applicants will be best served by transparent expectations about how/if programs will accept application materials created using artificial intelligence, starting with this application cycle. Graduate medical education will have many additional factors to consider for the innovative use and safeguards for the ethical application of artificial intelligence in clinical care and educational processes. However, the exponential increase in the application of this technology requires an urgent review for appropriate management of program procedures, iteration of policies, and a meaningful national discussion.

TECHNOLOGY AND ARTIFICIAL INTELLIGENCE IN MEDICAL EDUCATION

Effective use of technology in graduate medical education is an important consideration for program applicants, trainees, faculty, program directors, sponsoring institutions, and other stakeholders. Technology supporting training programs and associated processes can vary in usage. Use can be scheduled, such as program Website updates sharing a range of information, including selection criteria.¹ Sometimes, the technology use is unexpected, such as changes to rotations and interview conditions during the COVID-19 pandemic.² Process changes and technology choices have consequences, and programs must consider evidence-based options for supporting fair, equitable, and inclusive recruitment and mitigating bias in interview and recruitment practices.^3–5

The application of artificial intelligence (AI) in physiatry and medical education is not new. Broad applications already exist, from building physician skills in rehabilitation research and practice,⁶ clinical training and decision making,^7,8 to models screening applications to residency programs,⁹ and beyond. What is new is the release of ChatGPT by OpenAI in November 2022, a chat model “which interacts in a conversational way”¹⁰ triggering an impressive pace of discussion, individual and organizational applications, and the full range of possible reactions, including excitement and grave concern.¹¹ Within the first 3 mos of its release, ChatGPT has already been used to model answering US Medical Licensing Examination questions,¹² generating discharge summaries,¹³ and credited as a coauthor on multiple scientific publications.¹⁴ Search frequency for ChatGPT is exponentially increasing, and program directors should be aware of this topic.

The authors acknowledge a wide range of potential opportunities and drawbacks in the application of AI in medical education and do not intend to provide a global recommendation on its use. However, with the rapidly evolving application of ChatGPT and similar models across many contexts, AI is a topic with considerations important for program directors and their work. Text output from ChatGPT can draft abstracts that may not be reliably identified as AI-generated to human reviewers,^15,16 and a quick internet search will demonstrate to readers that applicants to medical school, residency, and fellowship programs are already asking questions such as, “Should I use ChatGPT to write my personal statement?” We recommend that programs not leave applicants guessing. This article aims to encourage program directors and other stakeholders to increase their awareness and explore considerations for AI in training program application materials, processes, and policy needs, starting with this application cycle.

PERSONAL STATEMENTS AND THE USE OF ARTIFICIAL INTELLIGENCE

Personal statements are among the top factors residency program directors consider in applicant review.¹⁷ In the authors’ professional experience, personal statements are also frequently used to support the program’s understanding of the applicant’s interests in the program, values, and professional goals, as well as to assess the general clarity, organization, and effectiveness of the applicant’s written communication in English. The authors encourage program directors to use this opportunity to reflect on the purpose, objective, and the current uses of the personal statement in applications to their program and to consider alternative approaches if the current practice does not support the program goals.

Models such as ChatGPT are specifically designed for successful written output, rather than other functions such as complex mathematical computations. Program directors should be aware of how applicants may use such tools, especially in the written elements of program application materials. In drafting, refining, and finalizing a personal statement, applicants are generally expected to use feedback and edits from other people and apply computer-based spelling and grammar checks. The authors anticipate that physiatry faculty and program directors would be surprised if applicants submitted a personal statement without using “spell check.” Will these same program faculty and specialty leaders now be surprised if applicants do, or do not, utilize the features of ChatGPT or other AI in their personal statement writing process? The high-stakes nature of residency applications has supported the growth of for-profit services offering a range of materials and preparation activities.¹⁷ We agree with Johnstone et al.¹⁷ that program application processes would likely be improved by having programs explicitly “define what application help is allowed.” In preparation for this article, one author (CJ) reviewed brain injury medicine fellowships program Website materials. As of the date of this writing, none of the 27 Accreditation Council for Graduate Medical Education accredited brain injury medicine fellowship programs provide guidance for applicants about using AI in personal statements or other application materials.

One suggestion also offered by Johnstone et al.¹⁷ is to encourage applicants to avoid editing and other application support services that require a fee. However, there is no guarantee that a “free” service will give good advice, act in the best interest of applicants, or avoid other nonmonetary direct or indirect costs to users. Emerging technology, including free services such as ChatGPT, may still conceal an understanding of application material authorship, intentionally or unintentionally. In addition, applicants may derive value from their ChatGPT usage and consider the model’s output to optimize their application. Simply using the model assists the company with AI optimization.

Per OpenAI, prose such as a personal statement written using the ChatGPT model is coauthored by the human and the AI, as reflected in the API Terms and Policies, Sharing and Publication Policy:

People should not represent API-generated content as being wholly generated by a human or wholly generated by an AI, and it is a human who must take ultimate responsibility for the content being published.

Here is some stock language you may use to describe your creative process, provided it is accurate:

“The author generated this text in part with GPT-3, OpenAI’s large-scale language-generation, model. Upon generating draft language, the author reviewed, edited, and revised the language to their own liking and takes ultimate responsibility for the content of this publication.”¹⁸

We argue that programs can support applicants and future processes by proactively considering and sharing with applicants their stance on using AI assistance in the application process, including personal statements. After ChatGPT’s launch, many colleges and universities are swiftly adding verbiage to policies. If available, the programs should follow their sponsoring institution’s existing policies and guidance when determining their stance on the usage of AI, including broader considerations such as letters of recommendation. Some educators are changing their course, examination, and grading structure and updating their definition of plagiarism to include, “using text written by a generation system as one’s own (eg, entering a prompt into an AI tool and using the output in a paper).”¹⁹ Furthermore, medical education programs shoulder consider opportunities to evolve their educational experiences and innovate with AI. For example, senior researcher at Harvard Graduate School of Education, Chris Dede, advises that AI should be incorporated into education to improve and raise the human standard.²⁰

SAMPLE CHATGPT-GENERATED PERSONAL STATEMENT FOR FELLOWSHIP

It is essential for programs to be aware of ChatGPT’s capabilities. To demonstrate, below is a sample personal statement generated by ChatGPT in less than 30 secs.

Author prompt: “Write a strong 1-page personal statement for a Physical Medicine and Rehabilitation resident applying to Brain Injury Medicine fellowship.”

ChatGPT-generated text: “As a physical medicine and rehabilitation resident, I have developed a passion for working with patients who have sustained traumatic brain injuries. Through my training, I have gained extensive knowledge and experience in managing complex medical conditions and addressing the physical, cognitive, and emotional consequences of brain injuries. I am now eager to pursue a fellowship in brain injury medicine to further enhance my skills and knowledge in this field.

My interest in brain injury medicine began during my residency, where I was exposed to patients with a wide range of brain injuries. I witnessed the significant impact these injuries have on the lives of patients and their families, and I was motivated to help them regain their independence and improve their quality of life. I have learned how to manage acute and chronic brain injuries, including developing comprehensive care plans to address physical, cognitive, and emotional issues. I have also had the opportunity to work with a multidisciplinary team to provide personalized and holistic care to each patient.

During my residency, I have also conducted research on brain injuries, including examining the effects of exercise on brain injury recovery. My research has deepened my understanding of the challenges faced by brain injury patients and the potential for rehabilitation to make a positive impact on their lives. I am excited to continue conducting research in this field and contributing to the knowledge base of Brain injury medicine.

I am confident that a fellowship in Brain Injury Medicine will provide me with the opportunity to further refine my skills in managing brain injury patients, working collaboratively with other healthcare professionals, and conducting cutting-edge research. I am eager to learn from experienced professionals in this field and contribute to the advancement of the field of Brain Injury Medicine. I am committed to providing the highest quality of care to my patients, and I am confident that a fellowship in Brain Injury Medicine will enable me to do so.”

Author prompt: “Update the personal statement to be for sample program and include examples of pertinent patient encounters.”

ChatGPT-generated text: “As a physical medicine and rehabilitation resident, I have had the privilege of working with a diverse patient population, including those with traumatic brain injuries, stroke, spinal cord injuries, and other neurological disorders. Throughout my training, I have developed a deep passion for improving the lives of patients with brain injuries and believe that a Brain Injury Medicine fellowship with Sample Program is the perfect next step in my career.

During my residency, I have managed several patients with brain injuries, each with unique challenges and complexities. One such patient was a 35-year-old man who sustained a traumatic brain injury following a motor vehicle accident. Upon admission to our inpatient rehabilitation unit, he had significant deficits in cognition, communication, and mobility. Through a comprehensive rehabilitation program that included physical, occupational, and speech therapy, as well as neuropsychology and recreational therapy, we were able to make significant improvements in his functional abilities. Witnessing his progress and the impact it had on his quality of life was truly inspiring and reinforced my commitment to the field of Brain Injury Medicine.

Another patient encounter that stood out to me was a 50-year-old woman with a history of multiple concussions. Despite receiving appropriate medical care, she continued to experience persistent headaches, cognitive difficulties, and fatigue. Through an individualized treatment plan that incorporated medication management, cognitive rehabilitation, and graded exercise therapy, we were able to improve her symptoms and quality of life. This experience highlighted the importance of a patient-centered approach to care and the need for a multidisciplinary team to address the complex needs of individuals with brain injuries.

I am drawn to the Sample Program Brain Injury Medicine fellowship because of its reputation for excellence in patient care, research, and education. I am eager to learn from experts in the field and to gain exposure to the latest advancements in research and technology. I am also excited about the opportunity to work with a team of professionals with a shared passion for improving the lives of individuals with brain injuries.

In conclusion, my experiences as a Physical Medicine and Rehabilitation resident have reinforced my passion for improving the lives of patients with brain injuries. I am confident that a Brain Injury Medicine fellowship with Sample Program will provide me with the knowledge, skills, and experience necessary to become a leader in the field. I am committed to making a meaningful contribution to the field of Brain Injury Medicine and am excited about the prospect of working with a team of professionals dedicated to improving the lives of those affected by brain injuries.”

Authorship: This text was generated on February 20, 2023, and March 1, 2023, with GPT-3 (ChatGPT February 13 Version), OpenAI’s large-scale language-generation, model. For the purposes of this publication, the language has been submitted as generated, without additional human edits. The authors take ultimate responsibility for the content of this publication.

CONCLUSIONS

Future conversations will be needed in our specialty, and more broadly in medical education, regarding innovation, and appropriate use of AI. Considerations will be needed across formative and evaluative medical education activities at all levels, from appropriate use of AI as a resource for administrative tasks and project reports, to defining the broader needs and training of current and future physiatrists in the ethical and effective application of AI to advance our specialty and the patients we serve. The availability and impact of ChatGPT and similar models is a rapidly evolving, complex, and multifaceted discussion with elements beyond the scope of this article. However, an excellent next step for physiatry is the definition by programs of the characteristics of the appropriate use of AI in graduate medical education training applications, such as personal statements and recommendation letters, associated awareness of ChatGPT and other AI, and ongoing national discussion.