We thank Chen and colleagues1 for highlighting some of the issues medical students face during the residency application and selection process. Their comments addressing Step 1 of the United States Medical Licensing Examination (USMLE) provide a helpful contemporary perspective, clarify some concerns about the USMLE, include perspectives of stakeholders other than the students, highlight some of the benefits associated with a potential shift to pass/fail scoring for Step 1, and suggest potential solutions to improve the residency application process. In this Invited Commentary, we offer additional context and perspective, while supporting further discussion of potential solutions and options to improve the residency application and selection process.
USMLE Program
A full description of the USMLE program appears at www.usmle.org. USMLE Step exams are one set in a series of key milestones during a physician’s journey of education and training to serve the public. One of the most important aspects of the USMLE program is that it is an independent, high-quality measure that is strongly valued by state medical boards, is developed by mission-based organizations focused on serving the public, and provides a uniform standard of competency.
The primary purpose of the examination, which is only administered during the early stages of a physician’s career, has always been to support the initial licensure decisions made by the nation’s state and territorial medical boards (SMBs). All SMBs recognize the examination—administered in the United States and around the world—as a high-quality, standardized tool for assessment. Some SMBs also value numerical USMLE scores in their subsequent postlicensure review.
Numerical score data inform several secondary uses, including providing content-specific feedback to examinees, curricular assessment for medical school faculties, medical school accreditation by the Liaison Committee on Medical Education, pedagogical and psychometric research, and evaluation by residency program directors.
Besides its robust assessment of medical knowledge and scientific concepts, the USMLE assesses elements of patient care knowledge and skills; communication and interpersonal skills; legal and ethical professionalism issues; and systems-based practice, with a focus on patient safety and quality improvement. The vast majority of content on the examinations is written by U.S. medical school faculty; SMB representatives, including physicians and members of the public; and nonfaculty experts who supplement faculty-derived content. In addition, U.S. medical school faculty, SMB members and staff, and the CEOs of the National Board of Medical Examiners (NBME), Federation of State Medical Boards (FSMB), and Educational Commission for Foreign Medical Graduates (ECFMG) are involved in every aspect of the oversight and governance of the program.
The Residency Application Experience
In previous professional roles, both of us have participated in the residency application process. As a student affairs dean, P.J.K. sought to balance an accurate description of a student for residency selection with the desire to present her or him in the best possible light, adhering to Association of American Medical Colleges (AAMC) guidelines about preparation of the dean’s letter, and subsequently the medical student performance evaluation (MSPE). As a preclinical education dean, H.J.C. recognized the value of the dean’s letter, clerkship evaluations, and the availability of numerical licensing exam scores in helping students obtain training in hospitals and in specialties of their choice. These experiences have helped sensitize us to the complexity and multiple perspectives of different stakeholders in the residency application process.
Almost all U.S. medical students, and many graduates of international medical schools, pursue postgraduate specialty training in the United States leading to licensure and medical practice. The selection process for residency training—like admission to college, medical school, and the pursuit of scholarships and research grants—is inherently competitive. Residency programs use data from multiple sources to evaluate applicants and depend on a reliable, valid national standard to compare candidates as part of their holistic review of applicants. Residency programs typically review hundreds (in some cases, thousands) of very heterogeneous applications, in addition to fulfilling their education, research, administrative, and patient care duties. A program’s evaluation processes for candidate selection seek to optimize the balance between effort and results. Many programs have found that scores from the USMLE provide a useful, consistent, and high-quality benchmark to allow comparisons between applicants.
Does Step 1 performance predict residency success? To our knowledge, no study has been done to answer this question. However, performance on Step 1 has been shown to correlate highly with similar licensing exams, and these exams have been correlated with quality metrics of potential interest to secondary score users like residency programs, including cardiac morbidity and mortality,2 likelihood of state board disciplinary action,3 and measures of preventive care and acute and chronic disease management.4 Thus, performance on Step 1 can provide useful information to inform part of a residency selection decision.
Varied Stakeholders, Varied Perspectives
We value the importance of U.S. medical students’ perceptions of the role that USMLE scores play in residency selection processes. We also recognize that there are numerous additional stakeholders whose perspectives are both important and potentially different from those identified by Chen and colleagues. Residency programs recognize that USMLE scores are but one data point in an applicant’s file, and National Resident Matching Program (NRMP) data indicate that residency programs’ weighting of Step 1 scores varies. Similarly, use of Step 1 scores varies across specialties, and looking only at the mean score of accepted applicants may be a misleading marker of its use. In 2018, virtually every specialty accepted applicants scoring 200 or below; only applicants to interventional radiology needed scores of 210 or above.5 USMLE scores provide a national, standardized, objective measure to compare individuals from different medical schools, backgrounds, and countries.
Students and U.S. medical graduates (USMGs) from elite medical schools may feel that their school’s reputation ensures their successful competition in the residency application process and, thus, may perceive no benefit from USMLE scores. However, USMGs from the newest medical schools or schools that do not rank highly across various indices may feel that they cannot rely on their school’s reputation, and have expressed concern in various settings that they could be disadvantaged if forced to compete without a quantitative Step 1 score. This concern may apply even more for graduates of international medical schools (IMGs) that are lesser known, regardless of any quality indicator; graduates of these schools make up almost one-third of NRMP participants and almost one-quarter of successfully matched applicants. Information provided by a nationally administered examination such as the USMLE may be critically important in evaluating IMG residency applicants.
Many changes to the residency application process, such as standardizing the MSPE and enhancing the Electronic Residency Application Service, are designed to facilitate efficient, effective review of residency applicants. Nonetheless, the task of a program director who receives hundreds or thousands of applications every year makes comprehensive review of every application nearly impossible. Residency program faculty note with some dismay the increased number of applications their programs receive every year as students increasingly employ a scattershot approach to residency application. Many bemoan their reliance on Step 1 scores in screening applicants but feel that alternative approaches, where objectivity, standardization, quality, and consistency are lacking, would be worse. To improve the residency selection process will require better understanding of the potentially conflicting and competing interests of all involved.
Anticipated Benefits and Risks of Step 1 Pass/Fail Scoring
Some argue that a shift to pass/fail scoring would reduce student stress with concomitant improvement in student mental and behavioral health. Student stress, burnout, and other mental health problems absolutely need careful consideration. For some students, a change to pass/fail scoring may be a significant stress reducer, while others may feel increased stress if they perceive this change as diminishing their ability to demonstrate academic excellence and to compete effectively for a desired or competitive residency training position.
If students reduce time and effort devoted to preparing for Step 1, they may indeed devote attention to other activities that will prepare them to be good physicians. This would arguably be an ideal outcome of such a change. However, it is possible that if students were to devote more time to activities that make them less prepared to provide quality care, this could negatively impact residency performance and ultimately patient safety. We know that assessment drives learning, so another concern resulting from a shift to pass/fail scoring may be a less knowledgeable physician population.
Elimination of numeric Step 1 scores may also lead to development of inferior, less psychometrically sound, and expensive alternative assessments in the information vacuum that would be created, with potentially increased costs for students and little guarantee of the reliability and validity of the substituted assessment. Faced with the elimination of a Step 1 score, it is conceivable that residency programs would encourage applicants to submit scores from another assessment, either one that already exists or one that is created because of the change. Even if programs do not adopt and require an alternative assessment, students may nonetheless feel pressured and compelled to undertake it, shifting the angst currently associated with Step 1 to a new assessment.
Research demonstrates some differences in USMLE scores attributable to race and ethnicity, with self-identified black, Asian, and Hispanic examinees showing score differences compared with self-identified white examinees.6 Some cite this as evidence in support of eliminating Step 1 scores, at least for residency selection. However, the majority of these observed differences disappear when controlling for undergraduate grade point average and Medical College Admission Test scores. The presence of a national, standardized, objective measure such as the Step 1 score may actually serve as an antidote to implicit bias, counteracting some of the subjectivity inherent in evaluating other aspects of an applicant’s record.
Potential Improvements to the Process of Residency Application
To broaden the discussion beyond Step 1 scores, we offer the following additional ideas, admittedly an incomplete list of possibilities, with a sampling of potential pros and cons to illustrate the complexity of the issue. Many of these ideas are not mutually exclusive. Virtually all require careful consideration of both the intended and unintended consequences because the process of residency application represents an important, complex system by which medical school graduates are trained.* As such, it is highly likely that a significant perturbation such as changing the approach to Step 1 scores could have unanticipated effects that ripple through the system in ways that are counterintuitive and unhelpful. With this in mind, we outline several potential options to stimulate further analysis and discussion.
Make Step 1 pass/fail
We remain committed to high-quality assessments of physicians’ preparation for practice. If thoughtful, evidence-based analysis by relevant stakeholders identifies a better approach to quantifiable USMLE scoring, we would be open to such an option and would present it to USMLE governance, and the governance of our two organizations, for consideration.
Release scores only to examinees
Some have suggested that providing scores only to examinees would retain scores’ formative benefit while eliminating their undesirable influence in residency application. One concern with such an approach would be the possibility (if not probability) that residency programs would exert some degree of pressure on applicants to provide their score data. Rules to prevent such information exchange would help, unless they lead to a catch-22 situation for applicants, who may feel pressured to provide score data yet be at risk for censure if they do so.
Alter the timing of score release to follow submission of Match lists
Delaying release of numeric scores until after NRMP Match lists have been submitted could remove consideration of USMLE scores in the application process yet ultimately still provide programs and other stakeholders with information that could facilitate residency training. The absence of USMLE score data during the applicant evaluation process, however, may lead to some of the undesirable consequences described earlier.
Enhance residency program data
If residency programs were to improve the data available about their programs (e.g., by centralizing, standardizing, and increasing the information available to applicants), then applicants may be better able to identify suitable programs. Targeting their applications more effectively could potentially reduce total application volumes, which in turn may reduce residency program workload and enable more holistic review of more applications.
Develop other numerical scores of information important in residency selection
A USMLE score uniquely meets a residency program’s desire to have a national, standardized measure of the competencies assessed by the USMLE sequence. Other important characteristics (e.g., research, volunteerism, resilience, commitment to underserved populations) are not presently reported in a manner that enables efficient comparison of applicants. Development of one or more additional metrics could balance the weight presently assigned to USMLE scores. One example of an attempt to accomplish this is the AAMC’s Standardized Video Interview.7
Develop an MSPE rating system
The AAMC has developed, revised, and promulgated standards for the MSPE. These could form the basis for a third-party evaluation of the quality and fidelity of medical schools’ MSPEs. Publishing the results of such evaluations could provide residency programs with an independent assessment of each school’s MSPE, which could then facilitate greater faith in, and weighting of, the MSPE. Such a system would also likely increase incentives for medical schools to conform to guidelines.
Conclusions
The residency application process is an important decision point in a physician’s career that is inherently and understandably stressful. Policy changes to mitigate such emotional impact should be supported by adequate evidence, solid reasoning, and informed discussion so as not to worsen the status quo. Simply eliminating high-quality information presently available during residency application risks worsening the process and desired outcomes.
Interpretation of USMLE scores should be done in the context of other important applicant characteristics, as well as residency program strengths and characteristics. A USMLE score should provide useful input into the holistic consideration of a candidate’s application. Clinicians will recognize that this approach is similar to the interpretation of tests that are routinely performed in the day-to-day practice of medicine, when a single test result must be interpreted in the context of a patient’s overall evaluation.
It is helpful to discuss the important issues raised by Chen and colleagues within a broader context, to facilitate comprehensive and evidence-based improvement in residency selection that thoughtfully considers desired and unintended consequences. There should be more conversation about the role of licensing examination scores in the residency application process. To this end, the AAMC, American Medical Association, ECFMG, FSMB, and NBME are jointly sponsoring an invitational conference in 2019 to convene representatives of the stakeholders identified earlier, who are directly or indirectly impacted by this issue, to facilitate further discussion of the possibility of any potential changes to the status quo.
References
1. Chen DR, Priest KC, Batten JN, Fragoso LE, Reinfeld BI, Laitman BM. Student perspectives on the “Step 1 climate” in preclinical medical education. Acad Med. 2019;94:302–304.
2. Norcini JJ, Boulet JR, Opalek A, Dauphinee WD. The relationship between licensing examination performance and the outcomes of care by international medical school graduates. Acad Med. 2014;89:1157–1162.
3. Cuddy MM, Young A, Gelman A, et al. Exploring the relationships between USMLE performance and disciplinary action in practice: A validity study of score inferences from a licensure examination. Acad Med. 2017;92:1780–1785.
4. Tamblyn R, Abrahamowicz M, Dauphinee WD, et al. Association between licensure examination scores and practice in primary care. JAMA. 2002;288:3019–3026.
5. Charting Outcomes in the Match: U.S. Allopathic Seniors, 2018. 2018.Washington, DC: National Resident Matching Program.
6. Rubright JD, Jodoin M, Barone MA. Examining demographics, prior academic performance, and United States Medical Licensing Examination scores. Acad Med. 2019;94:364–370.
