How Small Differences in Assessed Clinical Performance Amplify to Large Differences in Grades and Awards: A Cascade With Serious Consequences for Students Underrepresented in Medicine : Academic Medicine

Secondary Logo

Journal Logo

Advanced Search
Perspectives

How Small Differences in Assessed Clinical Performance Amplify to Large Differences in Grades and Awards: A Cascade With Serious Consequences for Students Underrepresented in Medicine

Teherani, Arianne PhD; Hauer, Karen E. MD, PhD; Fernandez, Alicia MD; King, Talmadge E. Jr MD; Lucey, Catherine MD

Author Information

A. Teherani is professor, Department of Medicine, education researcher, Center for Faculty Educators, and director of program evaluation, University of California, San Francisco, School of Medicine, San Francisco, California; ORCID: http://orcid.org/0000-0003-2936-9832.

K.E. Hauer is professor, Department of Medicine, and associate dean for assessment, University of California, San Francisco, School of Medicine, San Francisco, California; ORCID: https://orcid.org/0000-0002-8812-4045.

A. Fernandez is professor, Department of Medicine, University of California, San Francisco, School of Medicine, San Francisco, California.

T.E. King Jr is professor, Department of Medicine, and dean, University of California, San Francisco, School of Medicine, San Francisco, California.

C. Lucey is professor of medicine and vice dean for education, University of California, San Francisco, School of Medicine, San Francisco, California.

Funding/Support: None reported.

Other disclosures: None reported.

Ethical approval: The University of California, San Francisco Committee for Human Research Institutional Review Board approved the research protocol as exempt, May 2016, IRB no. 16-19486.

Previous presentations: The results from the analysis were presented institution-wide at the University of California, San Francisco.

Supplemental digital content for this article is available at https://links.lww.com/ACADMED/A568.

Correspondence should be addressed to Arianne Teherani, Office of Medical Education, University of California, San Francisco, School of Medicine, 1855 Folsom St., Suite 200, San Francisco, CA 94143; telephone: (415) 509-4988; e-mail: [email protected]; Twitter: @arianneteherani.

Academic Medicine 93(9):p 1286-1292, September 2018. | DOI: 10.1097/ACM.0000000000002323

Abstract

While students entering medical schools are becoming more diverse, trainees in residency programs in competitive specialties and academic medicine faculty have not increased in diversity. As part of an educational continuous quality improvement process at the University of California, San Francisco, School of Medicine, the authors examined data for the classes of 2013–2016 to determine whether differences existed between underrepresented in medicine (UIM) and not-UIM students’ clinical performance (clerkship director ratings and number of clerkship honors grades awarded) and honor society membership—all of which influence residency selection and academic career choices.

This analysis demonstrated differences that consistently favored not-UIM students. Whereas the size and magnitude of differences in clerkship director ratings were small, UIM students received approximately half as many honors grades as not-UIM students and were three times less likely to be selected for honor society membership.

The authors use these findings to illustrate the amplification cascade, a phenomenon in which small differences in assessed performance lead to larger differences in grades and selection for awards. The amplification cascade raises concerns about opportunities for UIM students to compete successfully for competitive residency programs and potentially enter academic careers. Using a fishbone diagram, a continuous quality improvement root cause analysis tool, the authors contextualize their institutional results. They describe potential causes of group differences, drawing from the education disparities literature, and propose interventions and future research. They also share countermeasures adopted at their institution and encourage other medical schools to consider similar exploration of their institutional data.

Increasing racial and ethnic diversity in the U.S. physician workforce is a national priority.1 Improvement in workforce diversity enhances access to health care, reduces health care disparities, and improves quality of care for underserved populations.2 Yet progress toward achieving a diverse medical workforce has been slow. Racial and ethnic groups underrepresented in medicine (UIM) include African Americans and Latinos, each constituting 4% of the physician population, and Native Americans and Alaska Natives, constituting 0.4% of the physician population, compared with 13%, 17%, and 2%, respectively, of the general population.1,3,4 To address the need for a more diverse workforce, medical schools have embraced initiatives such as pipeline programs and holistic admissions processes, which balance consideration of academic and nonacademic predictors of success in medical school and beyond.5,6 These initiatives have been successful at increasing the matriculation rate of UIM students.7 Unfortunately, while medical schools are becoming more diverse, residency programs in competitive specialties have not shown a similar increase in the diversity of trainees.8,9 Additionally, the diversity of faculty in academic medical centers has remained low.10 In essence, while the pipeline of UIM matriculants to medical school is expanding, the pipeline of UIM physicians from medical school to competitive residencies and careers in academic medicine is leaking.

Some UIM students choose to pursue residencies in primary care and careers practicing medicine outside academic medical centers. However, researchers have raised concerns that the cause of this leaking pipeline is differential opportunities, created by medical school assessments, for UIM students compared with their peers from groups not underrepresented in medicine (not-UIM; i.e., from racial and ethnic groups that have comparable or greater numbers in the physician population compared with their numbers in the general population).11 In a survey of residency program directors about metrics used to select applicants for interviews and to rank candidates, respondents demonstrated a preference for those with high scores on standardized examinations.12 This preference exists despite inconsistent evidence about the ability of exams to predict future success as a physician.13 Further, higher United States Medical Licensing Examination (USMLE) Step 1 scores are related to careers in academic medicine, which may partially explain the limited number of minority physicians entering faculty career paths.14 UIM students consistently receive lower scores on major standardized gateway exams, outside and within the medical education pathway, compared with not-UIM students.15–17 Experts believe that these group differences represent the long-term consequences of structural racism, which have created inequities in education, housing, economic, and social opportunities.16,18 Thus, assessment or residency selection strategies that rely on standardized exam results systematically disadvantage UIM students, which may make it more challenging for them to earn entry into competitive specialties and residency programs and, later, faculty careers.

Residency program directors also reported relying on the number of honors grades earned in medical school clerkships and membership in the Alpha Omega Alpha Honor Medical Society (AOA) to select future residents.12 New literature documents that group differences exist between UIM and not-UIM students in AOA membership that are preferential to not-UIM, white students, independent of exam scores and honors grades.19 Hence, the post-medical-school pipeline for diversifying residency programs and faculty may be affected by differential opportunities to earn honors grades and become eligible for AOA membership. Yet, to the best of our knowledge, the published literature does not document this type of exploration or discuss the potential causes of observed differences and possible interventions.

In this article, we describe our use of educational continuous quality improvement (ECQI) to evaluate the relationship between medical students’ UIM status and clinical performance (clerkship director ratings and the number of clerkship honors grades awarded) and AOA membership. We used a root cause analysis ECQI framework designed to answer the following three-part question: What happened, why did it happen, and what can be done to prevent it from happening again?20 We describe what happened through describing our ECQI method and results. To contextualize our findings, we discuss why it happened through use of a fishbone diagram,21 a root cause analysis tool. We also describe the potential causes of the observed differences, drawing from the literature on educational disparities. Finally, we share ideas on how to prevent inequities by describing possible interventions. Our intent is to prompt others to consider similar exploration of how their institutional procedures may perpetuate existing inequities. This work will, we hope, contribute to a nationwide conversation and further a research agenda on how medical schools can initiate change to equalize opportunities for all learners.

ECQI: Setting and Data Analysis

The University of California, San Francisco, School of Medicine (UCSF) is a research-intensive, urban public medical school. For this ECQI analysis, in 2016 we examined data for all medical students who matriculated at UCSF from 2009 through 2012 (graduating classes of 2013–2016). During the analysis period, UCSF employed holistic review of applications. This holistic review process did not use cutoff scores for grade point averages or Medical College Admission Test (MCAT) scores; instead, committees composed of faculty and students decided which applicants to interview and select on the basis of a balanced evaluation of academic performance, extracurricular activities, personal statements, letters of recommendation, and, finally, interview scores.

A total of 670 UCSF students were included in the analysis. Overall, 360 (53.7%) were women and 177 (26.4%) were UIM. We defined UIM status as described in the Association of American Medical Colleges’ Medical Minority Applicant Registry: “member of a racial or ethnic group historically underrepresented in medicine—African American/Black, Hispanic/Latino, American Indian/Alaska Native or Native Hawaiian/Pacific Islander.”22 UCSF designates applicants as UIM if they select any of the above race and ethnicity categorical values on their medical school application. If the applicant does not select any of these categorical values (e.g., selects another value such as white or Asian), the applicant is considered not-UIM. During the study period, the percentages of male and female medical students enrolled at UCSF were similar in the UIM and not-UIM groups. (Detailed demographics are available in Supplemental Digital Appendix 1 at https://links.lww.com/ACADMED/A568.) UIM students were much more likely than not-UIM students to be the first in their family to attend college.

We compared UIM and not-UIM students’ performance on the following outcome measures:

  • high-stakes multiple-choice question (HSMCQ) exams: mean composite score of MCAT exam’s Physical Sciences, Biological Sciences, and Verbal Reasoning section scores; first-attempt score on the USMLE Step 1 exam (taken before clerkships); and score on the USMLE Step 2 Clinical Knowledge (CK) exam (taken after clerkships).
  • clinical performance: clerkship director ratings in eight domains across seven required clerkships and average number of honors grades received across the seven clerkships; and
  • AOA status: eligibility and selection.

At the time of the study, per school policy, a maximum of 25% to 30% of students per clerkship were assigned an honors grade. Clerkship grades were awarded based primarily on faculty and resident ratings of students and to a smaller extent on clerkship examination scores. AOA eligibility was defined based on number of weeks of honors in required clerkships. A committee of faculty and residents reviewed the AOA-eligible group and selected up to one-sixth of the class for AOA membership based on service, humanism, and overall contributions to UCSF.

Our main objective was to investigate clinical performance and AOA selection, but we began by examining HSMCQ exam scores to determine whether our UIM students’ performance was similar to what was reported in the literature and to contextualize students’ clinical performance within their overall pre-medical-school and preclinical performance. Similar to existing evidence, we identified differences in HSMCQ exam scores by UIM status at our institution. Thus, when we compared clinical performance and AOA status by UIM status, we conducted two analyses. In the second, we adjusted for USMLE Step 1 score (covariate) because it reflected preclerkship knowledge assessment.23 All data were available from the UCSF Educational Data unit, who merged and deidentified data in preparation for our analysis. The UCSF Committee for Human Research Institutional Review Board approved the research protocol as exempt.

Below, we highlight our significant findings and describe the strength of those findings as computed by effect size calculations.

ECQI: Comparison of UIM and Not-UIM Student Performance

On each of the HSMCQ exams, the overall mean score was lower for UIM students than for not-UIM students. The effect sizes for the differences between UIM and not-UIM students were large for MCAT scores and medium for USMLE Step 1 and Step 2 CK scores. (Detailed results are available in Supplemental Digital Appendix 2 at https://links.lww.com/ACADMED/A568).

On average, compared with not-UIM students, UIM students received slightly lower clerkship director ratings (one-tenth of a point on the clerkship assessment scale) and, subsequently, half as many honors grades across all clerkships. Whereas the magnitude of the relationship between UIM status and clerkship director ratings was small, the magnitude of the relationship between UIM status and clerkship honors grades was medium.

When we adjusted the analysis by controlling for USMLE Step 1 scores, the relationships between UIM status and clerkship director ratings and honors grades were no longer significant. However, the relationships between USMLE Step 1 scores and clerkship director ratings and honors grades were significant, and their magnitude was medium. (Detailed results are available in Supplemental Digital Appendix 3 at https://links.lww.com/ACADMED/A568.)

Following the UCSF policies and AOA criteria detailed above, not-UIM students were approximately four times more likely to be eligible for AOA compared with UIM students. Not-UIM students were approximately three times more likely to be selected for AOA compared with UIM students.

The Amplification Cascade

Our single-institution examination of the association of UIM status and clerkship director ratings, honors grades, and AOA selection demonstrated differences that consistently favored not-UIM students. We observed that group differences narrowed between MCAT scores and subsequent USMLE Step 1 and Step 2 CK scores. Furthermore, group differences between performance on HSMCQ exams as a whole and clerkship director ratings narrowed even more. We believe this supports the idea that HSMCQ exam results tend to represent the long-term consequences of structural inequities in education. These favorable summative outcomes demonstrate that students with a wide range of backgrounds and exam scores can succeed at a research-intensive medical school, and they support UCSF’s use of holistic admissions.

UIM status accounted for a small amount of the unadjusted (for USMLE Step 1 score) difference in clerkship director ratings, and the magnitude of the difference was small. Unfortunately, this difference was amplified by institutional grading policies and AOA eligibility criteria. As a consequence, UIM students received approximately half as many honors grades across all clerkships compared with not-UIM students and were three times less likely to be selected for AOA membership. This amplification cascade, in which small differences in assessed performance lead to larger differences in grades and selection for awards, raises questions about policies for grading at UCSF and those driving AOA eligibility and selection nationally.19 The downstream consequences of differences in honors grades and AOA selection on opportunities for UIM students to compete successfully for competitive residency programs and potentially enter academic medicine careers are concerning.24

Differences between UIM and not-UIM students in clerkship director ratings and honors grades were no longer observed after we controlled for USMLE Step 1 scores, suggesting that student attributes linked to performance on HSMCQ exams are responsible for the differences in clinical performance assessment. These attributes may include medical knowledge, but they may also include other characteristics that predict higher exam scores, such as socioeconomic status and educational advantage.25,26 Socioeconomic status plays a vital role in student performance.25,27 However, the impact of the interplay between the closely related constructs of socioeconomic status and UIM status in medical education remains unclear.28,29 Further examination of currently used measures of socioeconomic status and the coalescence of socioeconomic status and UIM status should be at the forefront of future research on race/ethnicity and medical education.

Below, we return to our ECQI framework to address the two remaining questions in the root cause analysis: Why did it happen, and what can be done to prevent it from happening again?20 We discuss the potential evidence-based reasons for our findings in the next section on causes and consequences and then consider how to prevent future occurrences in the subsequent section on countermeasures.

Causes and Consequences of Differential Attainment

Differential attainment is a concept that encompasses how educational outcomes vary for different demographic groups assessed in the same way.28,30 In exploring possible causes of the group differences in assessed clinical performance that initiated the amplification cascade at UCSF, we employed a continuous quality improvement tool known as a fishbone diagram21 to illustrate the potential causes, effects, and consequences of individual (student or faculty/resident rater), interpersonal (student and faculty/resident rater), and cultural and structural factors that may have an impact on the student’s performance and/or the accuracy of the assessment of the student’s performance. The diagram (Figure 1) recognizes that all students, regardless of UIM status, may be subject to factors that negatively affect their performance or the assessment of their performance. Here, we focus on those factors, drawn from the literature, that have a disproportionate negative impact on UIM students and may contribute to the observed differences in assessed clinical performance.

F1
Figure 1:
Fishbone diagram illustrating the causes, effects, and consequences of lower assessed performance in underrepresented in medicine (UIM) students compared with all students.

Personal and interpersonal factors

Although personal and interpersonal factors influence all students’ performance, UIM students shoulder additional burdens as they strive to learn and demonstrate their abilities in the clinical environment.31–35 Microaggressions or overt racism directed at the student or others in the environment precipitate the stress response, resulting in impaired critical thinking, lower speech fluency, and sleep disorders that interfere with long-term memory.36–40 Activation of stereotype threat (i.e., environmental cues that make salient negative stereotypes about an individual’s status as a member of a group) has the potential to lead to diminished working memory capacity and performance overmonitoring, resulting in less engagement, lower cognitive risk taking, and less acceptance of feedback.31,41 Also, UIM students are often keenly attuned to and feel personally responsible for addressing issues of disparate care provided for minority patients, as a manifestation of race-conscious professionalism.35 Further, the difficulty of finding race- and ethnicity-concordant senior trainees and faculty contributes to feelings of isolation and low social capital.42 The performance of a student preoccupied by these issues may underrepresent his or her true abilities. Over the long term, the cumulative impact of these stressful distractions has the potential to interfere with UIM students’ overall competency development despite their capabilities and contributions.

Personal and interpersonal factors involving interactions with faculty and residents may also contribute to UIM students’ lower clerkship performance assessments. Because unconscious bias manifests in multiple aspects of academic medicine, from hiring to health care delivery, it is likely that unconscious bias also interferes with the accurate assessment of UIM learners by some faculty and residents.43,44 Faculty and residents may be sources of microaggressions in their interactions with UIM students and patients. Although these slights are often unintended, they can be impactful. In addition to distracting the student in the midst of important clinical discussions, microaggressions from a supervisor damage the trusting relationship needed to engage in workplace learning.32,40,45 In addition, faculty and residents who espouse a colorblind ideology may fail to recognize opportunities to address issues such as stereotype threat, racism, and the impact of microaggressions on performance.46 Ignorance about race-conscious professionalism may lead a not-UIM faculty member or resident to interpret a UIM student’s advocacy for minority patients as uninformed, disrespectful, or unprofessional. Taken collectively, these issues may result in faculty and resident assessments of a UIM student’s performance that are lower than the student’s true level of performance or ability.

Structural and cultural issues

Structural and cultural issues in the clinical learning environment may contribute to group differences in assessed performance. Rapidly rotating team assignments—particularly when teams lack racial or ethnic diversity other than that provided by the student—may exacerbate UIM students’ sense of isolation and lack of belonging.47,48 UIM students with proficiency in another language may preferentially be assigned to work with patients with low English proficiency49 without institutional recognition that this work often takes more time, particularly if the patients also have low health literacy or lack social support. These structural and cultural issues increase the work of learning and patient care for UIM students to an extent that often is not recognized by faculty and not accounted for in standard assessment strategies.

Moreover, structural issues in the conduct of grading may influence performance assessments. Given the known observed differences between UIM and not-UIM populations in performance on many standardized exams,15–17 institutional policies that afford sizeable weight to standardized knowledge-based clerkship exams (e.g., National Board of Medical Examiners subject exams) in determining honors grades may further perpetuate differences. Unless data exist that correlate clinical performance with exam scores, inclusion of exam results in clerkship grades could be viewed as inequitable. Employing normative grading policies (e.g., awarding honors grades to the top 25% of the class), rather than criterion-based grading policies (e.g., awarding honors grades to all who exceed a specified level of competency), may amplify small differences in assessment scores to large differences in achievement of honors grades—particularly when range restriction exists in the use of clerkship assessment scales.50

Countermeasures to Differential Attainment

As we have discussed above, UIM students shoulder substantially more burdens than their not-UIM peers, and their increased cognitive and emotional loads of learning and performing are neither commonly addressed in the educational environment nor captured and considered in assessment and grading. Thus, the paradigm of medical school as a colorblind, equal opportunity educational environment for learning and performing must be challenged. Truly equitable educational environments must take into account the differential experiences of UIM students in contrast to their not-UIM peers.

In an effort to begin to create an equitable educational environment, the leaders of the UCSF Bridges curriculum renewal51,52 (focused on preparing learners for practice in complex systems to improve health care and advance science) began in 2016 to work collaboratively with our institution’s Differences Matters Initiative53 (focused on efforts to increase equity and inclusion throughout the institution). This collaboration has enabled educators to work with experts in health care disparities and critical race theory as well as to engage a broader community of UIM and not-UIM faculty and staff in identifying potential solutions to the problem and causes of differential attainment identified in our internal study (Figure 1). Achieving a more equitable and inclusive educational environment requires an integrated systems approach to change.54 We have approached change through redesign of the medical school curriculum, learning and assessment methods, and faculty development programs so that all constituents (i.e., faculty, leadership) are equally responsible for (1) understanding the issues of racism, bias, and privilege, and how they manifest in the educational and health care environment; (2) recognizing and addressing microaggressions when they occur in our environments; (3) mentoring effectively across racial, ethnic, and other group differences; and (4) redesigning health care delivery systems and educating individuals to address health care disparities. Table 1 summarizes the countermeasures or components of our strategic responses to the issues identified that were developed and started to be implemented in 2016. We are presently investigating the outcomes of these internal strategies as a part of the ECQI process and anticipate that our findings will inform long-term solutions.

T1
Table 1:
Countermeasures Adopted, Starting in 2016, by the University of California, San Francisco, School of Medicine to Begin to Address Differential Attainment by Underrepresented in Medicine (UIM) Students in the Clinical Learning Environment

Future Research

Given the importance of diversity to quality of care in the U.S. health care system, federal funding should support a research agenda that builds an evidence base for equity in health professions education, not only for UIM learners but also for learners from diverse backgrounds. Future research should examine and seek prevention or intervention strategies for issues such as racism, microaggressions, and stereotype threat triggers that create learning environments unfavorable to optimal performance; for unconscious bias and colorblind ideologies that lead to inaccurate assessments; and for normative ranking and selection strategies that risk amplifying small performance differences of uncertain significance into large consequences that affect careers. Similar to research into prevention of medical errors, research into systems designs that protect vulnerable students from fallible faculty will be critical.

Conclusion

Efforts to diversify the nation’s physician workforce cannot focus only on pipeline programs and holistic admissions processes. Ultimately, mitigating the impact of differential attainment by UIM students on selection for competitive residencies and faculty careers will require national conversations about inclusive learning environments; equity in performance assessments at medical schools; and equitable, not-colorblind strategies in candidate evaluation and selection at residency programs. Further work is needed to ensure that learning environments are inclusive; instructional methods and assessment tools are equitable; and selection strategies for residency, fellowship, and faculty positions provide fair access to all candidates who are broadly qualified to succeed. A collective commitment by medical schools and teaching hospitals to seek out and address issues of inequity in learning, assessment, advancement, selection, and career opportunities is urgently needed. Successfully addressing these educational challenges will catalyze efforts to create equity in medical education and enable educators to deliver on our social obligation to improve the health of our communities.55

Acknowledgments:

The authors wish to thank the University of California, San Francisco, School of Medicine students, staff, and faculty who participated in discussions about and provided feedback on the project and manuscript. They would like to thank Mark Lovett, Elaine Allen, and Judy Quan for their data collection and methodological expertise. They would like to thank Patricia O’Sullivan and Christy Boscardin for their thoughtful feedback on the manuscript.

References

1. Association of American Medical Colleges. Diversity in the Physician Workforce: Facts and Figures 2014. 2014. Washington, DC: Association of American Medical Colleges; https://www.aamc.org/data/workforce/reports/439214/workforcediversity.html. Accessed May 9, 2018.
2. U.S. Department of Health and Human Services. HHS Action Plan to Reduce Racial and Ethnic Disparities: A Nation Free of Disparities in Health and Health Care. 2011. Washington, DC: U.S. Department of Health and Human Services; https://minorityhealth.hhs.gov/npa/files/Plans/HHS/HHS_Plan_complete.pdf. Accessed May 9, 2018.
3. United States Census Bureau. Facts for Features: Hispanic Heritage Month 2014. Release no. CB14-FF.22. https://www.census.gov/newsroom/facts-for-features/2014/cb14-ff22.html. Published September 2014. Accessed May 9, 2018.
4. Association of American Medical Colleges. Diversity in Medical Education: Facts and Figures 2012. 2012. Washington, DC: Association of American Medical Colleges; https://members.aamc.org/eweb/upload/Diversity in Medical Education Facts and Figures 2012.pdf. Accessed May 9, 2018.
5. Andriole DA, McDougle L, Bardo HR, Lipscomb WD, Metz AM, Jeffe DB. Postbaccalaureate premedical programs to promote physician-workforce diversity. J Best Pract Health Prof Divers. 2015;8:10361048.
6. Association of American Medical Colleges. Advancing holistic review initiative. https://www.aamc.org/download/358384/data/holisticreviewbrochure.pdf. Published 2014. Accessed May 9, 2018.
7. Urban Universities for Health. Holistic Admissions in the Health Professions. 2014. Washington, DC: Urban University of Health; http://urbanuniversitiesforhealth.org/media/documents/Holistic_Admissions_in_the_Health_Professions.pdf. Accessed November 13, 2016. [No longer available.]
8. Day CS, Lage DE, Ahn CS. Diversity based on race, ethnicity, and sex between academic orthopaedic surgery and other specialties. J Bone Jt Surg Am Vol. 2010;92(13):23282335.
9. Schwartz J, Young M. The evolution of racial, ethnic, and gender diversity in U.S. otolaryngology residency programs. Otolaryngology. 2013;149(1):7176.
10. Xierali IM, Fair MA, Nivet MA. Faculty diversity in U.S. medical schools: Progress and gaps coexist. AAMC Analysis in Brief. 2016;16(6). https://www.aamc.org/download/474172/data/december2016facultydiversityinu.s.medicalschoolsprogressandgaps.pdf. Accessed May 14, 2018.
11. Association of American Medical Colleges. Underrepresented in medicine definition. https://www.aamc.org/initiatives/urm. Accessed May 14, 2018.
12. Green M, Jones P, Thomas JX Jr.. Selection criteria for residency: Results of a national program directors survey. Acad Med. 2009;84:362367.
13. McGaghie WC, Cohen ER, Wayne DB. Are United States Medical Licensing Exam Step 1 and 2 scores valid measures for postgraduate medical residency selection decisions? Acad Med. 2011;86:4852.
14. Andriole DA, Jeffe DB, Hageman HL, et al. Variables associated with full-time faculty appointment among contemporary U.S. medical school graduates: Implications for academic medicine workforce diversity. Acad Med. 2010;85:12501257.
15. Miller C, Stassun K. A test that fails. Nature. 2014;510(7504):303304.
16. Davis D, Dorsey JK, Franks RD, Sackett PR, Searcy CA, Zhao X. Do racial and ethnic group differences in performance on the MCAT exam reflect test bias? Acad Med. 2013;88:593602.
17. Kleshinski J, Khuder SA, Shapiro JI, Gold JP. Impact of preadmission variables on USMLE Step 1 and Step 2 performance. Adv Health Sci Educ Theory Pract. 2009;14:6978.
18. Quinn DM, Cooc N. Science achievement gaps by gender and race/ethnicity in elementary and middle school: Trends and predictors. Educ Res. 2015;44(6):336346.
19. Boatright D, Ross D, O’Connor P, Moore E, Nunez-Smith M. Racial disparities in medical student membership in the Alpha Omega Alpha Honor Society. JAMA Intern Med. 2017;177:659665.
20. U.S. Department of Veterans Affairs. VA National Center for Patient Safety: Root cause analysis (RCA). https://www.patientsafety.va.gov/professionals/onthejob/rca.asp. Accessed May 14, 2018.
21. Ishikawa K. Introduction to Quality Control. 1991.Tokyo, Japan: Asian Productivity Association Publications.
22. Association of American Medical Colleges. Medical Minority Applicant Registry (Med-MAR). https://students-residents.aamc.org/choosing-medical-career/article/medical-minority-applicant-registry-med-mar. Accessed May 14, 2018.
23. Grimm L, Yarnold P. Reading and Understanding Multivariate Statistics. 2000.Washington, DC: American Psychological Association Press.
24. Brancati FL, Mead LA, Levine DM, Martin D, Margolis S, Klag MJ. Early predictors of career achievement in academic medicine. JAMA. 1992;267:13721376.
25. Woolf K, McManus IC, Potts HW, Dacre J. The mediators of minority ethnic underperformance in final medical school examinations. Br J Educ Psychol. 2013;83(pt 1):135159.
26. Brosnan C, Southgate E, Outram S, et al. Experiences of medical students who are first in family to attend university. Med Educ. 2016;50:842851.
27. Rubin M, Denson N, Kilpatrick S, Matthews KE, Stehlik T, Zyngier D. “I am working-class”: Subjective self-definition as a missing measure of social class and socioeconomic status in higher education research. Educ Res. 2014;43(4):196200.
28. Woolf K, Potts HW, McManus IC. Ethnicity and academic performance in UK trained doctors and medical students: Systematic review and meta-analysis. BMJ. 2011;342:d901.
29. Lumb AB, Vail A. Comparison of academic, application form and social factors in predicting early performance on the medical course. Med Educ. 2004;38:10021005.
30. Regan de Bere S, Nunn S, Nasser M. Understanding differential attainment across medical training pathways: A rapid review of the literature. Final report prepared for the General Medical Council. https://www.gmc-uk.org/-/media/documents/GMC_Understanding_Differential_Attainment.pdf_63533431.pdf. Published 2015. Accessed May 14, 2018.
31. Steele CM, Aronson J. Stereotype threat and the intellectual test performance of African Americans. J Pers Soc Psychol. 1995;69:797811.
32. White AA 3rd, Stubblefield-Tave B. Some advice for physicians and other clinicians treating minorities, women, and other patients at risk of receiving health care disparities. J Racial Ethn Health Disparities. 2017;4:472479.
33. Olayiwola JN. Racism in medicine: Shifting the power. Ann Fam Med. 2016;14:267269.
34. Paul-Emile K, Smith AK, Lo B, Fernández A. Dealing with racist patients. N Engl J Med. 2016;374:708711.
35. Powers BW, White AA, Oriol NE, Jain SH. Race-conscious professionalism and African American representation in academic medicine. Acad Med. 2016;91:913915.
36. Mensah MO. Making all lives matter in medicine from the inside out. JAMA Intern Med. 2017;48(6):12671278.
37. Burgess DJ, Warren J, Phelan S, Dovidio J, van Ryn M. Stereotype threat and health disparities: What medical educators and future physicians need to know. J Gen Intern Med. 2010;25(suppl 2):S169S177.
38. Rosenstein JE. Individual threat, group threat, and racial policy: Exploring the relationship between threat and racial attitudes. Soc Sci Res. 2008;37:11301146.
39. Slopen N, Williams DR. Discrimination, other psychosocial stressors, and self-reported sleep duration and difficulties. Sleep. 2014;37:147156.
40. Lucey CR, Navarro R, King TE Jr.. Lessons from an educational never event. JAMA Intern Med. 2017;177:14151416.
41. Schmader T, Johns M. Converging evidence that stereotype threat reduces working memory capacity. J Pers Soc Psychol. 2003;85:440452.
42. Woolf K, Rich A, Viney R, Needleman S, Griffin A. Perceived causes of differential attainment in UK postgraduate medical training: A national qualitative study. BMJ Open. 2016;6:e013429.
43. Girod S, Fassiotto M, Grewal D, et al. Reducing implicit gender leadership bias in academic medicine with an educational intervention. Acad Med. 2016;91:11431150.
44. Williams DR, Rucker TD. Understanding and addressing racial disparities in health care. Health Care Financ Rev. 2000;21:7590.
45. Montenegro R. Microaggressions during medical training—Reply. JAMA. 2016;316:1114.
46. Purdie-Vaughns V, Steele CM, Davies PG, Ditlmann R, Crosby JR. Social identity contingencies: How diversity cues signal threat or safety for African Americans in mainstream institutions. J Pers Soc Psychol. 2008;94:615630.
47. Woolf K, Potts HW, Patel S, McManus IC. The hidden medical school: A longitudinal study of how social networks form, and how they relate to academic performance. Med Teach. 2012;34:577586.
48. Vaughan S, Sanders T, Crossley N, O’Neill P, Wass V. Bridging the gap: The roles of social capital and ethnicity in medical student achievement. Med Educ. 2015;49:114123.
49. Fernandez A, Wang F, Braveman M, Finkas LK, Hauer KE. Impact of student ethnicity and primary childhood language on communication skill assessment in a clinical performance examination. J Gen Intern Med. 2007;22:11551160.
50. Kuo LE, Hoffman RL, Morris JB, et al. A milestone-based evaluation system—The cure for grade inflation? J Surg Educ. 2015;72:e218e225.
51. University of California, San Francisco, School of Medicine. Medial education: Bridges curriculum. http://meded.ucsf.edu/bridges. Accessed May 14, 2018.
52. Hauer KE, Boscardin C, Fulton TB, Lucey C, Oza S, Teherani A. Using a curricular vision to define entrustable professional activities for medical student assessment. J Gen Intern Med. 2015;30:13441348.
53. University of California, San Francisco, School of Medicine. Differences Matter initiative. http://differencesmatter.ucsf.edu. Accessed May 14, 2018.
54. Smith DG. Building institutional capacity for diversity and inclusion in academic medicine. Acad Med. 2012;87:15111515.
55. Frenk J, Chen L, Bhutta ZA, et al. Health professionals for a new century: Transforming education to strengthen health systems in an interdependent world. Lancet. 2010;376:19231958.

Supplemental Digital Content

Copyright © 2018 by the Association of American Medical Colleges