Secondary Logo

Journal Logo

Invited Commentaries

Anticipating the Impact of the USMLE Step 1 Pass/Fail Scoring Decision on Underrepresented-in-Medicine Students

McDade, William MD, PhD; Vela, Monica B. MD; Sánchez, J.P. MD, MPH

Author Information
doi: 10.1097/ACM.0000000000003490
  • Free

Abstract

With the alignment of required Liaison Committee on Medical Education and Accreditation Council for Graduate Medical Education (ACGME) diversity standards for medical schools and graduate medical education programs,1,2 there is great opportunity to develop a diverse workforce to lead equitable health care in the United States. However, in 2018, African American/black-, Hispanic/Latinx-, and American Indian/Alaskan Native-identified individuals represented approximately 34% of U.S. residents while these individuals represented only 14%–16% of U.S. allopathic medical students and 10% of residents in ACGME-accredited specialty and subspecialty training programs.3–5 Also striking is the low proportion of these underrepresented-in-medicine (UiM) groups in residency programs by specialty for the 2017–2018 academic year (pediatrics: 9% UiM residents; emergency medicine: 10%; orthopedics: 6%; dermatology: 6%).5

In February 2020, the National Board of Medical Examiners and the Federation of State Medical Boards announced that the United States Medical Licensing Examination (USMLE) Step 1 would transition to pass/fail grading beginning in 2022.6 The negative consequences of the current practice of providing a 3-digit numeric score for Step 1 have been discussed at length, including rising medical student anxiety related to test performance, associated financial costs for proprietary preparation materials, deterioration of preclinical medical student class participation, a shifting of medical school curricula to support a heavier emphasis on tested material to the exclusion of other formative subjects, an overwhelming number of applications submitted to residency programs, and prevalent use of Step 1 score cutoffs rather than holistic review to screen residency applicants. To further facilitate discourse on the impact of the Step 1 pass/fail grading decision on UiM applicants, we discuss several potential advantages and disadvantages of this change and call for close, rigorous evaluation of subsequent efforts to assess this change, especially regarding UiM individuals.

Potential Advantages for UiM Trainees

Addressing disparities in high-stakes testing

UiM students consistently score lower on standardized exams compared with white and Asian students.7 It is widely accepted by experts that these differences are due to consequences of structural racism; stereotype threat; and inequities in education, housing, and household income.7,8 On the whole, UiM students have more limited financial means and are thereby unfairly disadvantaged compared with others who have the ability to afford the expensive test preparation services and mentoring that have been shown to be beneficial to test takers.7,8 Cumulatively, these factors may adversely influence UiM students’ performance on high-stakes examinations such as the Medical College Admission Test and Step 1, unfairly limiting their chances of matriculating into medical school or gaining acceptance to a residency program in a preferred specialty. Pass/fail scoring rather than a numeric score for high-stakes examinations may help to flatten the effects of these inequities.

Dispelling the myth that a higher Step 1 score equates to greater competence

Residency program directors continue to select candidates with higher Step 1 scores despite lacking reliable evidence that these scores predict clinical skills acquisition or a candidate’s success as a physician.9 The persistent use of Step 1 scores as a screening tool by program directors is driven by a perception that focusing on a higher standardized numeric score ensures future competence. However, Step 1 only assesses trainees’ understanding of important concepts of the sciences basic to the practice of medicine, with special emphasis on principles and mechanisms underlying health, disease, and modes of therapy. A higher Step 1 score does not predict other physician competencies such as interpersonal and communication skills, professionalism, and practice-based learning and improvement. The passing score set by the examiners should be sufficient to define minimum expectations for the practice of medicine, and additional metrics can be used to determine suitability for a given program. Redefining what constitutes the “best” candidate involves looking at the total potential a given candidate can contribute to the field of medicine and to the care of patients. Producing a physician who is excellent in high-stakes testing but who will not serve the underserved or cannot effectively practice in minority communities may not be the “best” physician for the future. Competence is more than an examination score.

Allowing space for flexibility in the curriculum

Because of the emphasis placed on Step 1 scores in residency selection, the preclinical medical school curriculum has been oriented to improve students’ Step 1 performance. Focusing on preparing students for this single examination, however, occurs to the detriment of formative elements that are essential to becoming a physician but are not tested on Step 1. Students recognize that such formative curricular content often competes with fact-based Step 1 preparation and vote with their feet by not attending classes that are not directly pertinent to success on the examination.10

More elite and prestigious schools trade on their histories of having produced successful graduates in competitive residencies and still retain some emphasis on important curricular elements not tied to Step 1 preparation (e.g., cultural humility, patient advocacy, health disparities). Newer or less highly regarded medical schools, with a higher proportion of UiM students, do not have the luxury of such curricular offerings, and their students view a high Step 1 score as their sole opportunity to be selected for a competitive residency program or specialty. This undermines the uniqueness of a given medical school and the rationale as to why one might choose to attend it, as opposed to any other, as a student. Pass/fail scoring thereby can better support the diversity in medical school culture, curriculum, and mission that allows medical schools to educate to their strengths.

Reducing the competitive culture between residency programs

Some program directors continue to select candidates with high Step 1 scores in an effort to increase the competitiveness of their residency program. They believe that potential applicants and their advisors consider a program to be more desirable compared with other programs if their cohorts boast higher average Step 1 scores. The positive correlation between Step 1 scores and first-time board pass rate may suggest to applicants that they will benefit if they attend a program with a higher average Step 1 score among previous cohorts. The mere correlation between these test outcomes may suggest that there are qualitative and quantitative differences in program experiences, didactics, and resources, which may or may not be true. Correlation is simply an observation, not causation, and, as such, makes the contributory factors uninterpretable.

Additionally, as residency program faculty face increasing pressure to see more patients, selection committees may attempt to identify applicants who are perceived to require less didactic support based on Step 1 performance. Formerly, ACGME Residency Review Committees (RRCs) used only first-time specialty board pass rate to evaluate programs, which could differ by discipline. Because of this, many program directors felt pressure to select applicants felt most likely to pass specialty board exams on the first try in order to maintain their accreditation status. This may have contributed to the use of Step 1 as a surrogate for specialty board performance. Recent changes in ACGME common program requirements, however, mitigate the concern about first-time specialty board pass rate to a large degree.11 All disciplines now have a uniform pass rate requirement, and RRCs are beginning to evaluate graduates’ ultimate pass rate over the entire eligibility period. Through removing the 3-digit Step 1 score and its perceived predictive power, the ability of a given program to boast an uninterpretable competitive advantage over another may thereby be removed as well. It also reduces the likelihood of overreliance on standardized examination scores on which UiM candidates historically underperform, which has historically constructed a barrier that discounts the overall value of a diverse resident workforce.

Avoiding the improper use of Step 1 scores

As the gap narrows between the number of postgraduate year 1 positions and U.S. medical school graduates, medical students, especially those with lower Step 1 scores, are applying to a growing number of residency programs, increasing the workload for program staff having to screen a larger number of applications and for selection committees in interviewing and ranking more candidates in the matching process. Using Step 1 scores as a screening tool to impose a cutoff simply reduces the workload by winnowing the number of applications to review. This was unintended by the examination makers, who pleaded with the graduate medical education community to refrain from using Step 1 scores for that purpose.10

Easing anxiety about specialty selection

The use of Step 1 scores as a screening tool for resident selection may be restricting the entry of UiM candidates into select specialties and academic medicine. It is common for students who seek counsel after scoring lower than anticipated to be filled with concerns that their dream to practice dermatology, orthopedics, or another specialty is now out of reach. The stereotype threat engendered by a low Step 1 score may often drive lower performance on subsequent shelf exams and amplifies differences in future accomplishments12 that determine opportunities to enter competitive specialties. Conversely, students who score well on Step 1 and remain interested in primary care specialties are made to feel that they are wasting their talent and score. These not-so-hidden messages are fueled by Step 1 scores driving selection into each of the specialties.

Reducing isolation

In the period that students spend intensely studying for this exam, sometimes from the start of medical school, many students engage in isolating study practices. Isolation causes significant mental health morbidity and compounds the isolation experienced by UiM students already marginalized as minorities in their communities. As previously reported, Step 1 preparation has created a climate that contributes to a loss of student well-being.13 This unintended impact is likely more greatly felt among UiM medical students, who report a lower sense of personal accomplishment, higher stress, and lower quality of life than nonminority students.14 In contrast, pass/fail scoring inspires collaborative learning with colleagues who can all view passing as a collective good instead of fostering needless competition, which is counterproductive in medicine. When students are free to share their cognitive weaknesses without fear of comparison or of ridicule for not having known something that most peers know, it strengthens teamwork, reduces marginalization, and undermines the imposter syndrome.

Fostering participation in community activities

As students become less marginalized in their schools, their participation in extracurricular and curricular activities should increase. Medical students have acknowledged the value of a diverse student body to their own education and to the care they anticipate providing to their patients.15 Participation in student-led organizations may also increase in the preclinical years. The idealism that led many students into medicine is often lost to the self-interest of preparing for the individual benefit of doing well on Step 1. Often this self-imposed exile for the benefit of one’s own aspirations begins a career-long choice wherein the initial idealism with which medical students begin their education is lost and hard to reestablish. Bringing students back from the margins and integrating them into learning communities will benefit everyone.

Potential Disadvantages for UiM Trainees

Introducing the risk for discrimination

Negative effects in the change to pass/fail reporting for Step 1 scores could include a potential risk for discrimination against UiM candidates that parallels employment discrimination. A relevant example is the “Ban the Box” initiative, whose goal was to remove questions regarding prior criminal conviction from employment applications. Sociologists unexpectedly discovered that banning the declaration of a felony conviction adversely impacted all African American male applicants. Employers tacitly assumed that, in the absence of information, all such applicants presented a risk for conviction history.16 This stigmatized all black men, even ones who had no felony conviction. Could the absence of a 3-digit Step 1 score lead program directors to assume that all UiM candidates present a risk for future testing deficits?

Affecting future test scores

In the current scoring system, UiM students tend to do slightly better on USMLE Step 2 Clinical Knowledge (CK) 3-digit scoring than they do on Step 1.17 This could be because during rigorous Step 1 preparation, students who were formerly poor test takers learn to become better testers, and their subsequent test scores, including Step 2 CK, reflect this. However, without Step 1 to motivate behavior and sharpen test taking skills, we might very well find that the increased achievement of UiM students on Step 2 CK may be lost.

Eliminating a competitive edge

UiM students who are matriculated at schools considered to be less prestigious may have less of a competitive edge for certain residencies without a high numeric Step 1 exam score. Although the medical schools in Puerto Rico and historically black medical schools graduate the greatest proportion of black and Hispanic medical students, these schools are rarely touted as the most prestigious medical schools.18 A high 3-digit Step 1 score may aid some students at less prestigious schools to stand out in a competitive residency application process.

Increasing stress

Student response to the announced scoring change has been variable, but the announcement has created a great deal of stress among current and incoming students. UiM students report higher stress and lower quality of life during medical school,19 and the uncertainty of this new scoring system may worsen their wellness and performance during their preclinical training, potentially leading to a higher fail rate on Step 1. With reduced emphasis on high performance on Step 1, students may underprepare, which would have the unintended consequence of increasing the failure rate for UiM students. Going forward, the penalty for failure in a pass/fail system may be perceived as greater for those who have initially failed the examination. Currently, a larger proportion of UiM students fails Step 1 compared with peers.20 There is also the possibility that, in the future, an argument may be made to raise the bar for passing in the interest of patient protection. Combined, these suggest that the pass/fail system could result in greater challenges to diversity in graduate medical education.

Shifting the focus without solving the problem

Program directors may begin to use other metrics (school ranking, clerkship grades, Step 2 CK scores) to differentiate student performance. For example, schools with the greatest proportion of UiM trainees are often newer schools that lack recognition, stature, or a broad alumni network. Some of these metrics might adversely affect UiM trainees. Additionally, program directors may also feel the impetus to create subject exams that lack the reliability and rigorous field-testing of the USMLE.

The Need to Closely Monitor the Step 1 Scoring Change

The Step 1 scoring change has numerous potential advantages and disadvantages for UiM trainees. As academic health centers “engage in practices that focus on mission-driven, ongoing, systematic recruitment … of a diverse and inclusive workforce of residents [and] fellows,”2 it is imperative to take a rigorous approach to tracking and understanding the impact of this change on the representation of UiM individuals as well as other underrepresented groups (women, sexual and gender minorities, people with disabilities, and/or first-generation college students). The call for this rigorous approach is a shared responsibility for national organizations (e.g., accrediting, licensing, regulatory, professional, honor, student, faculty associations), hospitals, residency programs, and patient advocacy groups.

Conclusion

By reporting only the pass/fail status for Step 1, the stakes become considerably lower for this examination. The decreased pressure means students can focus simply on passing this examination, which was created to ensure minimal competence in medical knowledge. Medical schools can focus more attention in subject areas for which they have unique strengths and teach to the ideals that underlie their mission. Without a narrow focus on Step 1 preparation, crucial topics like cultural humility, patient advocacy, and health disparities can now find a place in curricula, addressing what often drives UiM students into the profession in the first place. Such a shift in focus has the potential to contribute to preparing all students to care for marginalized groups. Reporting numeric Step 1 scores and using them in residency selection decisions, on the other hand, is complicit in supporting inequities and disparities that keep UiM students from competitive specialties and perpetuate health inequities in society. Given that diversity among health care providers improves the health outcomes of patients,21,22 it is imperative that greater diversity exist at all levels of care, leadership, practice, and education. Shifting to pass/fail scoring of Step 1 is potentially an important step toward this goal to achieve greater diversity throughout the medical profession.

References

1. Liaison Committee on Medical Education. Functions and structure of a medical school. Standards for accreditation of medical education programs leading to the MD degree, Effective Academic Year 2021-22. Standard 3.3 diversity/pipeline programs and partnerships. Published March 2020. https://lcme.org/publications. Accessed April 29, 2020.
2. McDade WA. Increasing graduate medical education diversity and inclusion. J Grad Med Educ. 2019;11:736–738.
3. United States Census Bureau. Quick facts. May 2020. https://www.census.gov/quickfacts/fact/table/US/PST045219. Accessed April 2020.
4. Association of American Medical Colleges. Medical School Graduation Questionnaire: 2018 all schools summary report. July 2018. http://www.aamc.org/download/490454/data/2018gqallschoolssummaryreport.pdf. Accessed April 29, 2020.
5. Accreditation Council for Graduate Medical Education. GME Data Resource Book, Academic Year 2017-2018. Published 2019. https://www.acgme.org/About-Us/Publications-and-Resources/Graduate-Medical-Education-Data-Resource-Book. Accessed April 29, 2020.
6. United States Medical Licensing Examination. Invitational Conference on USMLE Scoring. Change to pass/fail score reporting for Step 1. Published 2020. https://www.usmle.org/incus/#decision. Accessed April 29, 2020.
7. Lucey CR, Saguil A. The consequences of structural racism on MCAT scores and medical school admissions: The past is prologue. Acad Med. 2020;95:351–356.
8. Cohen JJ. The consequences of premature abandonment of affirmative action in medical school admissions. JAMA. 2003;289:1143–1149.
9. McGaghie WC, Cohen ER, Wayne DB. Are United States Medical Licensing Exam Step 1 and 2 scores valid measures for postgraduate medical residency selection decisions? Acad Med. 2011;86:48–52.
10. Prober CG, Kolars JC, First LR, Melnick DE. A plea to reassess the role of United States Medical Licensing Examination Step 1 scores in residency selection. Acad Med. 2016;91:12–15.
11. Accreditation Council for Graduate Medical Education. ACGME common program requirements (residency). July 2019. Section I.C. https://www.acgme.org/Portals/0/PFAssets/ProgramRequirements/CPRResidency2019.pdf. Accessed April 29, 2020.
12. Teherani A, Hauer KE, Fernandez A, King TE Jr, Lucey C. How small differences in assessed clinical performance amplify to large differences in grades and awards: A cascade with serious consequences for students underrepresented in medicine. Acad Med. 2018;93:1286–1292.
13. Chen DR, Priest KC, Batten JN, Fragoso LE, Reinfeld BI, Laitman BM. Student perspectives on the “Step 1 climate” in preclinical medical education. Acad Med. 2019;94:302–304.
14. Dyrbye LN, Thomas MR, Huschka MM, et al. A multicenter study of burnout, depression, and quality of life in minority and nonminority US medical students. Mayo Clin Proc. 2006;81:1435–1442.
15. Saha S, Guiton G, Wimmers PF, Wilkerson L. Student body racial and ethnic composition and diversity-related outcomes in US medical schools. JAMA. 2008;300:1135–1145.
16. Doleac JL, Hansen B. The unintended consequences of “ban the box”: Statistical discrimination and employment outcomes when criminal histories are hidden. October 2017. http://jenniferdoleac.com/wp-content/uploads/2015/03/Doleac_Hansen_BanTheBox.pdf. Accessed April 29, 2020.
17. Rubright JD, Jodoin M, Barone MA. Examining demographics, prior academic performance, and United States Medical Licensing Examination scores. Acad Med. 2019;94:364–370.
18. Mullan F, Chen C, Petterson S, Kolsky G, Spagnola M. The social mission of medical education: Ranking the schools. Ann Intern Med. 2010;152:804–811.
19. Grbic D, Sondheimer H. Personal well-being among medical students: Findings from an AAMC pilot survey. AAMC Analysis in Brief. April 2014;14. https://www.aamc.org/system/files/reports/1/april2014aib_personalwell-beingamongmedicalstudents.pdf. Accessed April 29, 2020.
20. Andriole DA, Jeffe DB. A national cohort study of U.S. medical school students who initially failed Step 1 of the United States Medical Licensing Examination. Acad Med. 2012;87:529–536.
21. Smedley BD, Stith Butler A, Bristow LR. In the Nation’s Compelling Interest: Ensuring Diversity in the Health-Care Workforce. 2004.Washington, DC: National Academies Press;
22. Alsan M, Garrick O, Graziani G. Does diversity matter for health? Experimental evidence from Oakland. Am Econ Rev. 2019;109:4071–4111.
Copyright © 2020 by the Association of American Medical Colleges