Faculties of medicine have a societal mandate to produce a diverse pool of physicians.1–3 Research on selection to undergraduate and postgraduate medical education (UGME and PGME) programs has largely focused on the reliability and validity of specific selection tools rather than systemic issues and outcomes.4–6 Equity, diversity, privacy, and feasibility have assumed greater import in selection processes over the last decade as institutions respond to changes in societal needs, legislation, and the social environment.7,8
In alignment with the stewardship mandate of PGME portfolios, we sought to develop an approach to selecting residents with integrity and fidelity in which all stakeholders would have confidence. In this Perspective, we report on the University of Toronto’s centralized approach to defining system-level best practices for residency application and selection. We first present the rationale for change and the key issues identified in an environmental scan. We then describe the process used to inform and develop recommended best practices. Finally, we describe the implementation strategy and outline recommendations for next steps.
The Need for Better Approaches to Residency Application and Selection
Systemic challenges and calls to action
The landmark Future of Medical Education in Canada reports for undergraduate (FMEC MD) and postgraduate (FMEC PG) medical education laid out explicit mandates to improve selection processes and link them to societal responsibility.9,10 In the United States and Canada, the applicant pool for residency positions has grown larger and more diverse over the last decade, with more applicants from international medical schools and greater variation in applicants’ life and professional experiences.11 In Canada, this greater applicant pool breadth has arisen during a period of expansion in the number of government-funded undergraduate and postgraduate positions; however, the current fiscal climate and health human resources projections may lead to a plateau or reduction in position numbers. As applicant diversity increases amidst a keen focus on equity of access and human rights, UGME and PGME programs are becoming increasingly attentive to the fidelity and integrity of their selection processes. As programs progress through periods of expansion and contraction, they may be well served by adopting evidence-informed and principle-based selection approaches that will provide enhanced accuracy, confidence, and defensibility to admission decisions. Progress in developing such approaches has been hampered by a lack of consensus about what makes for a good future physician and what elements of past performance predict future behavior.12–18 The advent of competency frameworks such as the CanMEDS model19 and competency-based approaches to assessment20 may aid in addressing some of these challenges.
Recent broad-based initiatives, such as the FMEC MD and FMEC PG projects and the Lancet Commission on Education of Health Professionals for the 21st Century, have created a sense of urgency around the issue of who gets accepted into medical school and have explicitly framed admission as a key determinant of the contributions graduating cohorts collectively will make to society.9,10,21 A recent review scrutinized the processes by which PGME programs in the Canadian province of Ontario select international medical graduates (IMGs) and recommended changes to improve equity, reliability, and standardization within and across programs.22 These key reports and their associated recommendations have major implications for how selection committees should go about their work.
Stewardship and the need for broad-based recommendations
In Canada, PGME programs are based in universities, where they fall under the auspices of a senior decanal officer and are typically run within academic departments and divisions.23 Accordingly, departmental program committees establish their own specialty-specific selection criteria and processes. As PGME stewards, senior education leaders must ensure that selection is carried out in accordance with available best practices, within feasibility constraints, in alignment with institutional priorities, and in compliance with relevant regulations. The autonomy of PGME programs in selecting residents is an accreditation standard, which emphasizes the challenge of supporting and setting parameters for program selection committees while ensuring their ability to retain their authority over selection decisions.24
Developing and Advancing Best Practices in Residency Application and Selection at the University of Toronto
The University of Toronto’s complex academic health network hosts 79 specialty and subspecialty residency programs involving the equivalent of approximately 1,800 full-time residents at any time. Clinical training occurs in 27 affiliated health care institutions and numerous outpatient settings. The university has a broad range of medical educators and education scientists, many of whom contributed to the aforementioned FMEC reports. These features provided the university with the opportunity to convene a diverse group of educators and stakeholders to develop recommendations for residency application and selection that would be applicable to a broad range of residency programs. To this end, the University of Toronto PGME office established the time-limited Best Practices in Application and Selection (BPAS) working group to carry out a comprehensive literature review and environmental scan and recommend best practices to inform PGME selection processes. The BPAS working group met six times between September 2012 and April 2013, with intermittent dialog via e-mail between face-to-face meetings. The working group reported to the PGME advisory committee, which in turn is advisory to the vice dean of PGME (S.S.). The goal was to effect good stewardship by enabling programs to deploy the best evidence-based practices as they design and/or revise application and selection processes.
Working group constitution
The PGME advisory committee developed and approved the terms of reference for the BPAS working group. Working group members were selected to achieve a breadth of perspectives and included the associate dean for PGME admissions and evaluation (chair; G.B.), six residents, one clinical fellow, three residency program directors, five content experts external to PGME (director of the physician assistant program, associate dean for UGME admissions and student finances [M.H.], associate dean of equity and professionalism, associate dean for undergraduate health professions student affairs [L.N.], past executive director of the HealthForceOntario Marketing & Recruitment Agency/executive director at IMG Ontario), and three PGME office staff (director of policy and analysis [C.A.], research officer [M.R.], director of resident wellness). The residents brought dual perspectives both as recent applicants and as residents now serving on admission committees. Program directors represented large and small programs, as well as procedural and nonprocedural disciplines. The external content experts represented related portfolios in other education jurisdictions or partner organizations. The PGME office staff were selected for their knowledge of workforce planning, research methods, and policy and procedures around resident quota allocation and university operations.
Working group mandate
The BPAS working group refined the initial mandate from the PGME advisory committee as follows:
- Receive and review a literature scan on selection in the health professions.
- Identify and review local exemplary practices in selection relevant to PGME.
- Develop principles to guide the development of best practices and inform individual program selection activities.
- Develop a set of best practices for selection processes, selection criteria, and instruments.
- Establish (minimum) criteria for assessing applicants.
- Identify links and/or potential overlaps with UGME and with independent practice.
- Recommend a systems-level implementation strategy for PGME.
Working group process
The FMEC MD project9 involved an extensive literature review and produced a detailed analysis of the literature, which identified “Access and Selection” as a key theme.25 The BPAS working group elected to rely heavily on that distillation of the literature and to enhance it with an updated literature review. Thus, the working group commissioned a search of MEDLINE (Ovid) and PubMed for English-language articles published between January 2010 and December 2013, using the following search terms: medical school, postgraduate medical education, admission, selection, social responsibility, accountability, and diversity. Each BPAS working group member reviewed the papers identified in the review as well as the contents and bibliographies of the FMEC MD, FMEC PG, Lancet Commission, and Ontario reports described above.9,10,21,22 Each member independently summarized new and emerging considerations. The group then agreed on key issues to pursue further. The resulting updated review on the Canadian perspective on admission and diversity has recently been published.26
Through this review and discussion of key issues, the BPAS working group identified several knowledge gaps related to residency application and selection processes and invited content experts to present on topics including health human resources forecasting and the history of the Canadian residency Match process. Then, to understand the current state of selection processes across the University of Toronto’s PGME programs and establish a baseline of practices, the working group conducted a local environmental scan. In addition, one working group member (M.R.) reviewed the narrative self-study questionnaires recently completed by programs for accreditation purposes. Specifically, M.R. extracted, coded, and categorized responses to the question “Standard B1.5: Describe how residents are selected into the program” and presented the results to the group for consideration.
BPAS working group members independently reflected on the totality of the information presented, and each produced recommendations for discussion. A draft report, with 13 principles and 21 recommendations, was presented to the PGME advisory committee in May 2013. At that committee’s request, feedback was sought on the draft from all 79 University of Toronto residency program directors and all senior departmental education leads, as well as representatives of provincial and national resident organizations and the Canadian Resident Matching Service. The BPAS working group convened a final time to revise several recommendations and add 3 further best practices, based on the feedback received. The PGME advisory committee approved the final report, with 13 principles and 24 recommendations for best practices, in September 2013.
Key Outcomes: Observations on the Current State and Plans for the Future
Observations about the state of residency application and selection processes
The BPAS working group’s local environmental scan reaffirmed the need for this initiative: The review of PGME program self-study questionnaires revealed significant variability in selection processes. The working group identified the following important current issues facing residency programs:
- Significant tension exists between selecting the best candidates from the applicant pool and selecting to achieve greater diversity in the physician pool to better reflect the general population to be served.
- Selection processes must address inequities and underrepresented populations in residency programs and must avoid unintended and/or inappropriate bias.
- Selection processes used for IMGs should be improved.
- Canadian medical faculties are engaged in isolated innovations to improve admission processes, but widespread implementation is a challenge.
- The rigor of assessment methods, particularly assessment of noncognitive characteristics and skills-based testing, must improve.
- Improved communication is required between the UGME and PGME communities.
These topic areas guided the BPAS working group’s development of key principles and best practices. Members felt there was a strong need to increase diversity in residency programs (items 1, 2, and 3, above) but realized that selection committees are somewhat constrained by the breadth of diversity in the UGME pool. If meaningful change is to occur, diversity initiatives should transcend the medical education spectrum through the development of pipeline programs and streams for individuals from targeted populations. PGME programs should focus on removing inappropriate bias in their selection processes, educating selection committee members about assessing a heterogeneous applicant pool, and correcting existing inequities in some specialties and programs.
The BPAS working group also identified several weaknesses in current systems that limit widespread change (item 4), calling for sharing of best practices, implementation of minimum standards, and attention to diversity initiatives. Further, multiple examples in the literature and local practice highlighted a focus on objective assessments. Although objectivity is desirable, indiscriminant focus on that which is seen as objectively assessable can cause heightened emphasis on objective measures (such as academic standing and marks) over those measures perceived to be less well defined and more subjective (such as leadership skills, teamwork abilities, and empathy). However, many of the latter are key competencies and attributes of future physicians, such as compassion and leadership; efforts need to be made to develop and incorporate reliable and valid measures of these less well-defined but no less important competencies (item 5). Finally, there is widespread opinion in the literature that better communication between the UGME and PGME communities would serve the system, programs, and trainees well in terms of clarity of selection processes and disclosure of learner needs (item 6).
Accordingly, the BPAS working group developed the view that a sound and high-quality resident selection process would be based on the following assertions that emerged from the group’s deliberations:
- Selection for residency programs must be viewed fundamentally as an assessment activity that starts well before the actual application process.
- Increasing accountability is required within the system in terms of transparency, equity and diversity, reliability, validity, and oversight.
- Successful results are most likely when selection is aligned with socially responsive institutional and programmatic goals.
Setting a course for the future
The BPAS working group produced 37 final recommendations that collectively respond to the key assertions described above and were supported by a wide range of interested stakeholder groups. Thirteen of these recommendations were felt to be sufficiently generalizable and fundamental to be identified as guiding principles in selection process design (List 1). Programs can look to these principles when prioritizing activities and in pursuing change within their institution. The remaining recommendations were viewed as more concrete, practical actions that aligned with the aforementioned principles and for which there was either empirical support or face validity. These 24 recommendations were framed as best practices and categorized into eight functional groupings to assist in implementation: transparency, fairness, selection criteria, process, assessors, assessment instruments, knowledge translation, and ranking (Appendix 1). These best practices were felt to align well with directions identified in the medical selection literature, despite the literature’s focus on UGME admission.26–29 Further more, the recommendations may collectively help to address significant challenges facing residency education, such as the mandate to observe human rights in the face of increasing diversity, the need to demonstrate social accountability by choosing future practitioners who will deliver care where and when needed, and the incorporation of the growing evidence around good instrument and process design.10,25,30 Each of the recommendations serves to advance one or more of the key assertions, as described further below.
List 1 Recommended Principles to Guide Process Design for Resident Application and Selection, BPAS Working Group, University of Toronto Cited Here...
- Selection criteria and processes should reflect the residency program’s clearly articulated goals.
- Selection criteria and processes should reflect a balance of emphasis on all CanMEDS competencies.
- Selection criteria used for initial filtering, file review, interviews, and ranking should be as objective as possible.
- Selection criteria and processes should be fair and transparent for all applicant streams.
- Selection criteria and processes should promote diversity of the resident body (e.g., race, gender, sexual orientation, religion, family status), be free of inappropriate bias, and respect the obligation to provide for reasonable accommodation needs where appropriate.
- Programs should choose candidates who best meet the above criteria and are most able to complete the specific residency curriculum and enter independent practice.
- Multiple independent objective assessments result in the most reliable and consistent applicant rankings.
- Undergraduate and postgraduate leaders and communities must engage in collaborative planning and innovation to optimize the transition between undergraduate and postgraduate as well as between specialty and subspecialty postgraduate programs for all learners.
- Postgraduate programs must be well informed of educational needs of individual candidates to allow effective and efficient educational programming.
- Recognizing that past behavior and achievements are the best predictors of future performance, efforts should be made to include all relevant information (full disclosure) about applicants’ past performance in application files.
- Applicants should be well informed about specialties of interest to them, including health human resources considerations.
- Programs must consider and value applicants with broad clinical experiences and not expect or overemphasize numerous electives in one discipline or at a local site.
- Diversity of residents across postgraduate medical education programs must be pursued and measured.
Abbreviations: BPAS indicates Best Practices in Application and Selection; CanMEDS, Canadian Medical Education Directives for Specialists.19
Building on the key assertion that selection is fundamentally an exercise in assessment.
Much of the assessment literature focuses on evaluation of validity and reliability.12,13,31–33 Multiple sampling using validated, standardized tools informed by explicit criteria and expectations will lead to the most reliable results. Predictive validity research converges around the ideas that (a) past performance predicts future performance and (b) results of assessments using a specific modality—for example, examination results for knowledge—are only likely to predict results of similar assessments of the same attributes using the same methodology in the future—for instance, knowledge-based certification examinations.6,15–17,34–37 Newer methods, such as performance-based testing, narrative analysis, and portfolio review, support assessment of a broader set of attributes using modalities that venture beyond quantitative knowledge-based testing.4,5,14,26,38 Even the Medical College Admission Test (MCAT) has recently undergone comprehensive review and revision with the goal of helping identify those applicants who are most likely to become well-rounded physicians. The new version of the MCAT, which was launched in spring 2015, is expected to have greater predictive validity.39,40
New applicant assessment modalities focus more on behavior-based assessments, external narrative perspectives, and in-person encounters (e.g., interviews), and they specifically seek to assess noncognitive attributes. A comprehensive multimodality approach to application and selection that involves some performance-based testing may allow for better detection of exemplary or concerning behaviors and attributes, resulting in different admission decisions than an approach that relies on only quantitative assessments and student-driven narratives.18,38,41,42
Building on the key assertion that increased accountability is required.
The BPAS working group heard varying accounts of impediments to selection that were related to lack of information exchange and transparency between the UGME and PGME communities. Similar findings were reported by the FMEC MD and FMEC PG projects and in other reports.7,9,10,26,43 This realization precipitated a significant early shift in focus for the working group, resulting in a scope change from “Applicant Selection” to “Application and Selection.” Career counseling was identified as a significant issue; learners, program directors, and counselors expressed concerns about mismatches between trainees’ specialty aspirations and both societal need and labor market fluctuations. UGME representatives noted that it’s often not clear how various residency programs rank candidates and what criteria are important. PGME representatives felt that not all relevant information that was known and might inform their selection decisions is shared by UGME leaders. The need for better collaboration and communication between the UGME and PGME communities—including clarification of PGME program priorities and sharing of accurate, relevant UGME assessments—underpinned several of the BPAS working group’s recommendations.
Building on the key assertion that alignment with social accountability is a high priority.
Defining and demonstrating the social accountability of medical schools and, by extension, residency programs is not a new issue but is a high priority.1–3,44,45 The importance of programs and institutions identifying their social mandates and establishing selection criteria based on these mandates was seen as an important first step for many programs. Admission decisions can play a significant role in effecting the social mandate to provide a comprehensive physician workforce. Programs that purposely design their educational models around a specific type of practice and then select individuals who are aligned to these models can help reduce health inequities, address human resources maldistribution, and increase class diversity.46–52 Most studies have focused on UGME admission, but PGME programs can adapt these models to help them produce graduates who practice in specific specialties, serve specific populations, or practice in specific communities. Some studies have looked at candidate selection with the aim of increasing representation from traditionally underrepresented groups, such as racial minorities or those with rural backgrounds.47–50 The literature suggests that a diverse physician population that is representative of the population it serves is beneficial to patient care.25
Although PGME programs are somewhat constrained by the availability of candidates—and thus by the diversity of UGME cohorts—it is time for residency programs to implement innovations that align with already-established UGME initiatives to comprehensively address issues of diversity throughout all selection phases of medical education. Options to advance incorporation of PGME diversity goals within selection processes include dialogue across the PGME and UGME communities regarding institutional diversity goals, streaming programs, removal of inappropriate and unintended biases, use of objective criteria and assessment methods that are not subject to familiarity bias (i.e., selecting those who resemble the self), and PGME pipeline and mentorship programs for underrepresented groups.26
Implementation Strategy and Next Steps
The centralized PGME oversight role may manifest differently in different institutions. As an exemplar, this Perspective describes the University of Toronto’s centralized approach to defining system-level best practices for residency application and selection. The resulting 13 principles and 24 best practices have been approved by the university’s PGME advisory committee and circulated across the institution’s PGME system. They are now considered to be the institution’s minimum standards for developing specialty-specific processes that allow for local and content specificity and adherence to the Canadian PGME accreditation requirement specifying that programs develop and oversee their own selection processes. Questions around implementation of these principles and best practices have been added to program internal review self-study questions. Compliance with the minimum standards will thus be reported to the vice dean of PGME through the institution’s internal review process.
An extensive faculty development program has been developed to support local program committees in implementation of the principles and best practices. A toolbox of resources is in development and will include documents and templates to assist with selection practices (e.g., guidelines for creating written program descriptions, file review tools and interview questions, and scoring protocols and training materials for interviewers and file reviewers). Workshops on selected recommendations have been presented nationally and internationally53 to introduce tools and strategies to improve residency application and selection processes.
At the University of Toronto, the PGME office plays an active role in assisting programs in addressing sensitive issues that arise during selection and which may be mitigated by adherence to the BPAS recommendations. These issues are used as learning opportunities and stimulus for change in all programs. The office provides advice on messaging for potential candidates pre and post interview, provides advice on unusual issues (e.g., when to make exceptions, how to interpret exam results and other documents, what to do with information obtained external to the defined application process), and deals with complaints and information requests. The PGME office also responds on behalf of the programs and institution to external queries about and challenges to the selection process.
To assess the overall impact of selection to our PGME programs, the PGME office monitors Match results by program and in aggregate, reviewing school of origin and gender patterns annually. A diversity survey was recently designed and deployed to assess aspects of diversity within the current resident cohort. It will be repeated at regular intervals to assess the impact of diversity initiatives.
At a national level, a Learner Education Handover committee has been convened by the FMEC implementation committee to develop a national protocol for post-Match communication between UGME and PGME programs to identify the attributes and needs of incoming trainees. Applicants cannot and should not be excluded from selection because of special needs; however, programs need to be fully informed before the arrival of new trainees in order to balance trainee, program, and patient care needs. The introduction of a learner education handover protocol will help PGME programs be better prepared, with the goal of improving resident performance, ease of transition from UGME to PGME, and quality of patient care. This protocol will, in part, be informed by the BPAS working group recommendations.
It is hoped that sharing this work will stimulate ongoing dialog, with the overarching goal of ensuring that future physicians are selected into residency programs through a process with integrity and fidelity and that has the confidence of stakeholders. Translating the BPAS recommendations across jurisdictions and taking a systems-level approach to implementation hold promise for helping rectify long-standing deficiencies in both the process of selection to PGME programs and service provision to relevant populations by a comprehensive physician workforce.
Acknowledgments: The authors wish to thank all members of the Best Practices in Application and Selection (BPAS) working group. The BPAS working group members include Glen Bandiera (chair), Caroline Abrahams, Amanda Cipolla, Naheed Dosani, Susan Edwards, Joel Fish, Jeannette Goguen, Maureen Gottesman, Mark Hanson, Karl Iglar, Roaa Jamjoom, Aaron Lo, David McKnight, Leslie Nickell, Mariela Ruetalo, Kevin Shore, Brad Sinclair, Derek Tsang, and Zoe Unger. In addition, the authors wish to thank the Canadian Resident Matching Service, the Resident Doctors of Canada, and the Professional Association of Residents of Ontario for their thoughtful review and contributions to the BPAS report.
1. Wollard RF. Caring for a common future: Medical schools’ social accountability. Med Educ. 2006;40:301–313
2. Rourke J. Social accountability in theory and practice. Ann Fam Med. 2006;4(suppl 1):545–548
3. Dopelt K, Davidovitch N, Yahav Z, Urkin J, Bachner YG. Reducing health disparities: The social role of medical schools. Med Teach. 2014;36:511–517
4. Albanese MA, Snow MH, Skochelak SE, Huggett KN, Farrell PM. Assessing personal qualities in medical school admissions. Acad Med. 2003;78:313–321
5. Bore M, Munro D, Powis D. A comprehensive model for the selection of medical students. Med Teach. 2009;31:1066–1072
6. Mercer A, Puddey IB. Admission selection criteria as predictors of outcomes in an undergraduate medical course: A prospective study. Med Teach. 2011;33:997–1004
7. Razack S, Lessard D, Hodges BD, Maguire MH, Steinert Y. The more it changes; the more it remains the same: A Foucauldian analysis of Canadian policy documents relevant to student selection for medical school. Adv Health Sci Educ Theory Pract. 2014;19:161–181
8. Young ME, Razack S, Hanson MD, et al. Calling for a broader conceptualization of diversity: Surface and deep diversity in four Canadian medical schools. Acad Med. 2012;87:1501–1510
12. Bandiera G, Regehr G. Reliability of a structured interview scoring instrument for a Canadian postgraduate emergency medicine training program. Acad Emerg Med. 2004;11:27–32
13. Bandiera G, Regehr G. Evaluation of a structured application assessment instrument for assessing applications to Canadian postgraduate training programs in emergency medicine. Acad Emerg Med. 2003;10:594–598
14. Eva KW, Rosenfeld J, Reiter HI, Norman GR. An admissions OSCE: The multiple mini-interview. Med Educ. 2004;38:314–326
15. Eva KW, Reiter HI, Trinh K, Wasi P, Rosenfeld J, Norman GR. Predictive validity of the multiple mini-interview for selecting medical trainees. Med Educ. 2009;43:767–775
16. Harfmann KL, Zirwas MJ. Can performance in medical school predict performance in residency? A compilation and review of correlative studies. J Am Acad Dermatol. 2011;65:1010–1022.e2
17. Eva KW, Reiter HI, Rosenfeld J, Trinh K, Wood TJ, Norman GR. Association between a medical school admission process using the multiple mini-interview and national licensing examination scores. JAMA. 2012;308:2233–2240
18. Ginsburg S, Eva K, Regehr G. Do in-training evaluation reports deserve their bad reputations? A study of the reliability and predictive ability of ITER scores and narrative comments. Acad Med. 2013;88:1539–1544
19. Frank JR, Danoff D. The CanMEDS initiative: Implementing an outcomes-based framework of physician competencies. Med Teach. 2007;29:642–647
20. Frank JR, Snell LS, Cate OT, et al. Competency-based medical education: Theory to practice. Med Teach. 2010;32:638–645
21. Frenk J, Chen L, Bhutta ZA, et al. Health professionals for a new century: Transforming education to strengthen health systems in an interdependent world. Lancet. 2010;376:1923–1958
25. Hodges BD, Albert M, Arweiler D, et al. The future of medical education: A Canadian environmental scan. Med Educ. 2011;45:95–106
26. Bandiera G, Maniate J, Hanson M, Woods N, Hodges B. Access and selection: Canadian perspectives on who will be a good doctor and how to find them. Acad Med. 2015;90:946–952
27. Koczwara A, Patterson F, Zibarras L, Kerrin M, Irish B, Wilkinson M. Evaluating cognitive ability, knowledge tests and situational judgement tests for postgraduate selection. Med Educ. 2012;46:399–408
28. Plint S, Patterson F. Identifying critical success factors for designing selection processes into postgraduate specialty training: The case of UK general practice. Postgrad Med J. 2010;86:323–327
29. Nallasamy S, Uhler T, Nallasamy N, Tapino PJ, Volpe NJ. Ophthalmology resident selection: Current trends in selection criteria and improving the process. Ophthalmology. 2010;117:1041–1047
30. Frank JR, Harris KA Competence by Design: Reshaping Canadian Medical Education. 2014 Ottawa, Ontario, Canada Royal College of Physicians and Surgeons
31. Blouin D. Reliability of a structured interview for admission to an emergency medicine residency program. Teach Learn Med. 2010;22:246–250
32. Axelson R, Kreiter C, Ferguson K, Solow C, Huebner K. Medical school preadmission interviews: Are structured interviews more reliable than unstructured interviews? Teach Learn Med. 2010;22:241–245
33. Hanson MD, Kulasegaram KM, Coombs DL, Herold J. Admissions file review: Applying the multiple independent sampling (MIS) methodology. Acad Med. 2012;87:1335–1340
34. Thundiyil JG, Modica RF, Silvestri S, Papa L. Do United States Medical Licensing Examination (USMLE) scores predict in-training test performance for emergency medicine residents? J Emerg Med. 2010;38:65–69
35. Spurlock DR Jr, Holden C, Hartranft T. Using United States Medical Licensing Examination(®) (USMLE) examination results to predict later in-training examination performance among general surgery residents. J Surg Educ. 2010;67:452–456
36. Zahn CM, Saguil A, Artino AR Jr, et al. Correlation of National Board of Medical Examiners scores with United States Medical Licensing Examination Step 1 and Step 2 scores. Acad Med. 2012;87:1348–1354
37. Andriole DA, Jeffe DB. Prematriculation variables associated with suboptimal outcomes for the 1994–1999 cohort of US medical school matriculants. JAMA. 2010;304:1212–1219
38. Carlson ML, Archibald DJ, Sorom AJ, Moore EJ. Under the microscope: Assessing surgical aptitude of otolaryngology residency applicants. Laryngoscope. 2010;120:1109–1113
39. Monette M, Hemond C. MCAT revisions finalized. CMAJ. 2012;184:E473–E474
40. Collier R. New Medical College Admissions Test in the works. CMAJ. 2011;183:E801–E802
41. Christakis PG, Christakis TJ, Dziura J, Christakis JT. Role of the interview in admissions at the University of Toronto ophthalmology program. Can J Ophthalmol. 2010;45:527–530
42. Max BA, Gelfand B, Brooks MR, Beckerly R, Segal S. Have personal statements become impersonal? An evaluation of personal statements in anesthesiology residency applications. J Clin Anesth. 2010;22:346–351
43. Razack S, Maguire M, Hodges B, Steinert Y. What might we be saying to potential applicants to medical school? Discourses of excellence, equity, and diversity on the Web sites of Canada’s 17 medical schools. Acad Med. 2012;87:1323–1329
44. Butler WT. Academic medicine’s season of accountability and social responsibility. Acad Med. 1992;67:68–73
45. Foreman S. Social responsibility and the academic medical center: Building community-based systems for the nation’s health. Acad Med. 1994;69:97–102
46. Raghavan M, Martin BD, Roberts D, Aoki F, MacKalski BA, Sandham JD. Increasing the enrolment of rural applicants to the faculty of medicine and addressing diversity by using a priority matrix approach to assign values to rural attributes. Rural Remote Health. 2011;11:1646
47. Thomas B, Manusov EG, Wang A, Livingston H. Contributors of black men’s success in admission to and graduation from medical school. Acad Med. 2011;86:892–900
48. Puddey IB, Mercer A, Carr SE, Louden W. Potential influence of selection criteria on the demographic composition of students in an Australian medical school. BMC Med Educ. 2011;11:97
49. Scott I, Gowans M, Wright B, Brenneis F, Banner S, Boone J. Determinants of choosing a career in family medicine. CMAJ. 2011;183:E1–E8
50. Wayne S, Timm C, Serna L, Solan B, Kalishman S. Medical students’ attitudes toward underserved populations: Changing associations with choice of primary care versus non-primary care residency. J Health Care Poor Underserved. 2010;21:438–447
51. Garrud P. Who applies and who gets admitted to UK graduate entry medicine? An analysis of UK admission statistics. BMC Med Educ. 2011;11:71
52. Strasser R, Hogenbirk JC, Minore B, et al. Transforming health professional education through social accountability: Canada’s Northern Ontario School of Medicine. Med Teach. 2013;35:490–496