Because this is an Olympic year, with numerous stories of remarkable individuals overcoming adversity through perseverance, talent, and luck, I can’t help but view medical student and resident selection partly through the lens of the Olympics. I suspect I am not alone in desiring the more clear-cut, diverse, and equitable process symbolized by an Olympic contest. In some Olympic events, such as the metric mile run, the “selection process” is visible to all: The winner is the one who crosses the finish line first. It does not matter what the nationality, ethnicity, or race of the competitor is. For me, part of the fascination of Olympic events like the metric mile run is the mix of nationalities, races, and social classes and the fact that the outcome usually cannot be predicted by the backgrounds or physical attributes of the competitors. Each Olympic race has its own dynamic, its own set of challenges, and its own personal influences, and even the experts are often fooled. On the day of the race there will be one winner, and the criterion for deciding who wins is clear.
This straightforward approach cannot be used for decisions about the selection of medical students or residents, which requires the combined judgment of a carefully chosen group of committee members. Such decisions are more like those used in Olympic events such as gymnastics, in which judges score the contestants based on a set of criteria. However, unlike gymnastics, in medical student and resident selection, determining the quality of performance is based on information of variable reliability and validity such as letters of recommendation, personal statements, interviews, medical student performance evaluations (MSPEs)—often referred to as deans’ letters—and performance on standardized tests such as the Medical College Admissions Test (MCAT) and the United States Medical Licensing Examination (USMLE) Step 1. How can we improve the information and processes that we use for medical student and resident selection to yield future physicians who will meet the needs of our population?
Razack et al1 have framed the problem of medical school selection as “seeking inclusion in an exclusive process” and have defined the key elements of a successful selection process as excellence, equity, and diversity. Equity and diversity are included as part of the selection process because of the belief that a diverse group of students will best meet the needs of the community and will, to some degree, make up for historical inequities. The Institute of Medicine (IOM)2 has suggested that a diverse workforce would be one part of a solution to the disparities in health outcomes of racial and ethnic minorities because of the greater tendency for underrepresented minority physicians to practice in minority neighborhoods, where there are shortages of physicians, and to care for minority patients. In their monograph the IOM also noted that a physician’s ability to speak the patient’s language is an important consideration when Hispanic patients choose a physician.
Both Razack et al and the IOM describe the difficulties in defining and measuring excellence and argue for a broadening of our definitions in recognition of the many cognitive and noncognitive skills that are present in the best doctors. Bandiera et al3 have suggested that the selection framework should include cognitive and interpersonal attributes, diversity representative of the practice population, reliability and validity of the criteria used, and accountability for the process based on agreed-upon outcomes, such as societal contributions. In the rest of this editorial I will discuss how we might broaden our thinking about selection criteria to improve our selection process.
First, we need information about our applicants that is accurate and valid.
In this issue of Academic Medicine, Hom et al4 investigated key words used in MSPEs to describe student performance (outstanding, excellent, good, etc.) and clerkship grades. The MSPEs were from 117 of 131 medical schools accredited by the Liaison Committee on Medical Education and were for students applying to the internal medicine residency program at Stanford University School of Medicine in the 2013–2014 selection cycle. Only 61% of schools provided key words with complete explanatory and distribution data. Complete information on clerkship grades was present for students at 82% of schools.
Also in this issue, Boysen Osborn et al5 report a similar study to evaluate the use of ranking information in MSPE letters from 134 of 136 U.S. MD-granting medical schools from applications to the programs in emergency medicine or internal medicine at the University of California, Irvine, School of Medicine during the 2012–2013 and 2014–2015 selection cycles. They found “marked variability” in the awarding of top ranking and grades; 75% of schools ranked their students using either key words, percentiles, or specific numerical ranks. Like Hom et al, they found great variability in the use of key words. Among schools using ranking categories, “outstanding” was used to describe students from the 33rd to 99th percentile, and “excellent” was used to describe students from the 1st to the 95th percentile. They also noted great variability in the placement of the ranking information in the MSPE. In both these reports,4,5 the authors noted the risks to students of misclassification, and Hom et al noted that uncertainty about the meaning of the comparative data included in the MSPE could increase the weight given by residency program directors to other information such as the USMLE Step 1 score. Andolsek6 in a Commentary in this issue reviews the current state of the MSPE and makes recommendations for improving it, which include greater accountability, transparency, and standardization.
Letters of recommendation from faculty can also provide valuable information about an applicant. In this issue, Love et al7 describe a modification of a letter of recommendation developed for emergency medicine residency applicants known as a standardized letter of evaluation (SLOE). This letter includes a work-based assessment by faculty who have observed the student in the emergency department with patients and provides comparative information about cognitive and noncognitive skills of the student. Unlike some letters of recommendation that emphasize only positive attributes of a candidate, the SLOE provides a balanced assessment of the student compared with other students and an assessment of fitness for the specialty of emergency medicine from faculty who have worked with the student and understand the clinical demands of the specialty. These features of the SLOE make it a very useful component of the portfolio of the residency applicant
Katsufrakis et al8 present in this issue an overview of the residency application process and the information used in that process besides the MSPE and letters of recommendation. Such information includes the USMLE Step 1 score, membership in the Alpha Omega Alpha Medical Honor Society and the Gold Humanism Honor Society, medical school class rank, interview performance, extracurricular activities, leadership roles, and personal statements. They note that the USMLE Step 1 score has the advantages of being a national standardized objective measure correlated with medical school performance, in-training examination scores, and board certification pass rates. They note that board certification pass rates of residents affect a residency program’s accreditation. While Katsufrakis et al focused on residency applications, many of the same elements and issues exist for applicants to medical school, including the importance of academic performance, letters of recommendation, interviews, personal statements, and performance on the MCAT. Donnon et al9 and Dunleavy et al10 have demonstrated the association with performance on the MCAT with future performance in medical school and on licensing examinations.
The lack of reliable standardized information outside of the MCAT and the USMLE has made these tests a default standard for comparing applicants. Gliatto et al11 in this issue decry the overweighting of the MCAT and the USMLE as measures of excellence and future academic success and suggest the need to reduce their impact on curriculum and selection. While I believe this suggestion has merit, it will be difficult to implement without finding ways to improve the accuracy and utility of the other types of applicant information noted above.
Better Selection Process
Although better information will improve the selection process, it is not a complete solution. We also need the right people with the right training who know what to do with the information that is collected. How will the various fragments of information be assembled to create a picture of the candidate? Who will do that assembly? What are their values? How do those values align with the needs of the population? The selection process requires having the right people on the selection committee and training them how to weigh the information that has been collected with the goals of excellence, diversity, and equity in mind. Witzburg and Sondheimer12 have described an example of a holistic review process in which certain desirable attributes were described. These included intellectual ability, commitment to service, cultural sensitivity, empathy, capacity for growth, emotional resilience, strength of character, interpersonal characteristics, curiosity, and engagement. These attributes were then linked to data elements that would be reviewed as part of the selection process.
One could debate whether the attributes chosen for this holistic review are the most important attributes of a good physician. For example, Lamb and Bristow13 asked expert practicing clinicians from a range of specialties to select the most important noncognitive attributes for a physician. These clinicians chose integrity, the primacy of patient care, communication skills, empathy, lifelong learning, tolerance of ambiguity, and recognition of one’s limits. The specifics of the list may not be as important as the recognition of the broad range of capabilities that can lead to excellence in a physician. Many different types of “best doctor” candidates might come to mind. Is that candidate the one who can sew up a hole in the heart most effectively? The one who can make an obscure diagnosis after a careful history and physical? The one who can listen patiently and calm the suffering of a dying patient? The one who works the longest hours? The one dedicated to the poor and willing to work for the lowest pay? The one who discovers a new treatment for a previously deadly disease? The one who manages other health professionals in an efficient system? While there can be disagreement about which characteristics are most important in describing the best doctors, there is no doubt that numerous features unrelated to cognitive capacity would be included, and a holistic process that values the life experience and diverse talents of candidates might help identify such characteristics in applicants.
Better Transitions and Onboarding
Finally, we need to begin to look at the selection process as part of the continuum of medical education. In other words, although formal selection takes place before students enter medical school or residency, confirmatory selection—the ability of students to succeed as they meet the challenges of medical education—takes place throughout the continuum. And the transitions into medical school and into residency are among those challenges. Just as most search firms help candidates transition into their new positions, we need to anticipate the transitions into medical school and into residency as parts of the selection process. For example, students who enter a program with identifiable risks in test taking should be provided support to reduce those risks as early as possible so they will be better able to succeed in medical school and beyond.
In the transition from medical school to residency there are often opportunities to enhance the confidence of the newly selected intern. Many medical schools are developing boot camps at the end of the fourth year to give specific training to prepare students to enter their residencies. Teo et al14 identified 16 boot camp programs in 2011, most ranging from one to four weeks, and described their own three-week program. And more could be done. Petrilli et al15 recently proposed a model of enhanced faculty and resident patient coverage in the month of July—when new interns arrive—to improve both the educational experience of the interns and to address risks to patient safety. Many institutions also have orientation programs for new residents that include introduction to the culture and resources of their new institution, and simulation training to address the immediate need for new skills and knowledge. All of these efforts can be enhanced and diffused throughout our medical education system to provide strong transitions for medical students and residents after selection.
Recent efforts in Canada to explore a collaboration on admission policies among all the Canadian medical schools are described by Hanson et al16 in this issue. The authors identified four relevant core admissions practice and policy domains: social accountability strategies, standardized admissions testing, interviewing procedures, and application procedures. While there are a number of possible facilitators to bring about collaboration on these key areas, the authors noted that one barrier common across multiple domains was the pursuit of prestige by the medical schools; such competition got in the way of cooperation. While this should not be surprising, it should caution us to remember that the common goal of medical school admission is to provide the best physicians possible for our population and to do so equitably and efficiently. Although competition can improve performance, as in the Olympics, it can also lead to pursuit of winning for its own sake.
Kirch and Prescott,17 in a Commentary a few years ago about the limitations of systems that rank medical schools, reminded us that medical schools have different missions. Thus, a winning admission strategy at one medical school may not look the same or yield the same results as winning strategies at other medical schools. Let us incorporate that concept into an admission approach in which we take pride in our differences as institutions and as individual faculty, residents, and students, and find the best ways to recognize the broad spectrum of excellence that our applicants can offer in helping us meet the health care needs of the communities we serve.
David P. Sklar, MD
1. Razack S, Hodges B, Steinert Y, Maguire M. Seeking inclusion in an exclusive process: Discourses of medical school student selection. Med Educ. 2015;49:36–47.
2. Institute of Medicine. Committee on the Institutional and Policy-Level Strategies for Increasing the Diversity of the U.S. Healthcare Workforce. In the Nation’s Compelling Interest: Ensuring Diversity in the Health-Care Workforce. 2004.Washington, DC: National Academies Press.
3. Bandiera G, Maniate J, Hanson MD, Woods N, Hodges B. Access and selection: Canadian perspectives on who will be good doctors and how to identify them. Acad Med. 2015;90:946–952.
4. Hom J, Richman I, Hall P, et al. The state of medical student performance evaluations: Improved transparency or continued obfuscation? Acad Med. 2016;91:1534–1539.
5. Boysen Osborn M, Mattson J, Yanuck J, et al. Ranking practice variability in the medical student performance evaluation: So bad, it’s “good.” Acad Med. 2016;91:1540–1545.
6. Andolsek KM. Improving the medical student performance evaluation to facilitate resident selection. Acad Med. 2016;91:1475–1479.
7. Love JN, Ronan-Bentle SE, Lane DR, Hegarty CB. The standardized letter of evaluation for postgraduate training: A concept whose time has come? Acad Med. 2016;91:1480–1482.
8. Katsufrakis PJ, Uhler TA, Jones LD. The residency application process: Pursuing improved outcomes through better understanding of the issues. Acad Med. 2016;91:1483–1487.
9. Donnon T, Paolucci EO, Violato C. The predictive validity of the MCAT for medical school performance and medical board licensing examinations: A meta-analysis of the published research. Acad Med. 2007;82:100–106.
10. Dunleavy DM, Kroopnick MH, Dowd KW, Searcy CA, Zhao X. The predictive validity of the MCAT exam in relation to academic performance through medical school: A national cohort study of 2001–2004 matriculants. Acad Med. 2013;88:666–671.
11. Gliatto P, Leitman IM, Muller D. Scylla and Charybdis: The MCAT, USMLE, and degrees of freedom in undergraduate medical education. Acad Med. 2016:91:1498–1500.
12. Witzburg RA, Sondheimer HM. Holistic review—shaping the medical profession one applicant at a time. N Engl J Med. 2013;368:1565–1567.
13. Lambe P, Bristow D. What are the most important non-academic attributes of good doctors? A Delphi survey of clinicians. Med Teach. 2010;32:e347–e354.
14. Teo AR, Harleman E, O’Sullivan PS, Maa J. The key role of a transition course in preparing medical students for internship. Acad Med. 2011;86:860–865.
15. Petrilli CM, Del Valle J, Chopra V. Why July matters. Acad Med. 2016;91:910–912.
16. Hanson MD, Moineau G, Kulasegaram K, Hammond R. Is Canada ready for nationwide collaboration on medical school admissions practices and policies? Acad Med. 2016;91:1501–1508.
17. Kirch DG, Prescott JE. From rankings to mission. Acad Med. 2013;88:1064–1066.