The medical education community has been calling for changes to the residency selection process for decades. These calls have grown in recent years, with many focusing on the need for holistic review and alternatives to academic metrics such as United States Medical Licensing Examination Step exam scores.1–3 One aspect of applicant performance to consider in holistic review is proficiency in behavioral competencies, such as professionalism and interpersonal and communication skills. While applicants’ strengths and weaknesses in these behavioral competencies can sometimes be inferred from other components of the application, such as clerkship assessments and membership in the Gold Humanism Honor Society, the medical education community has lacked a standardized tool and process by which to rigorously and consistently assess their proficiency.
The Association of American Medical Colleges (AAMC) Standardized Video Interview (SVI) is an online, asynchronous video interview that assesses applicants’ knowledge of professional behavior and their interpersonal and communication skills. The AAMC SVI was administered for research purposes only during the Electronic Residency Application Service (ERAS) 2017 cycle and then piloted operationally in residency selection by the emergency medicine (EM) community during the ERAS 2018–2020 cycles. The field of EM was chosen because EM leaders had been engaged in AAMC efforts to find assessment solutions and because the EM community had shown itself to be receptive to adaptation and change. The SVI was administered for free during the pilot.4
At the conclusion of the ERAS 2020 cycle, the AAMC reviewed data collected from all 4 years of SVI administration. The data support the AAMC position that the SVI is a reliable, valid assessment of behavioral competencies. The SVI provides information to program directors that is not available in the ERAS application packet, and it does not disadvantage individuals or groups.4 Yet despite these solid psychometric properties, the AAMC elected not to renew or expand the SVI pilot in residency selection due to the EM community’s lack of interest in continuing to use and research the SVI, as well as the operational challenges of scaling the SVI to the full applicant pool across multiple specialties.
In this Invited Commentary, we describe lessons the Emergency Medicine Standardized Video Interview (EMSVI) working group learned about introducing a new tool for use in residency selection. Our hope is that sharing these lessons will benefit others who introduce new tools as the medical education community endeavors to find ways to support holistic review of applicants. We encourage the academic medicine community to consider this experience in discussions about advancing the future of residency selection.
Lessons Learned From the SVI Project
Engage all stakeholders from the start
Because the AAMC identified a possible solution to a problem that has been plaguing graduate medical education for decades—holistic application review—it acted boldly and quickly to launch the SVI pilot for the ERAS 2018 cycle. The AAMC engaged residency program directors in the initial efforts because they are the ultimate end users of SVI scores. However, the broader EM community—including chairs of departments, vice chairs, and other education and academic leaders—was not included from the start. While the AAMC adjusted by establishing the EMSVI working group, representing all stakeholders affected by the SVI, the initial limited participation led to early concerns about transparency and reduced credibility, which proved difficult to overcome. Future projects should consult or inform all stakeholder groups, even those that may not be direct users of the new tool, at the onset and then on a regular basis during the life cycle of the project.
Communicate the value of the new tool early and often
The AAMC missed an opportunity to communicate early and well about the SVI’s value to the EM community. The early messaging about the SVI focused on logistics and research rather than the potential value added to the selection process for both programs and applicants. This communications strategy was informed by the desire to collect early evidence of effectiveness, which could then be used in messaging around value. However, focusing the early messaging on the inadequacy of the current selection process could have highlighted the need and generated desire for a solution—and emphasized the value added by the SVI. During the SVI project, the AAMC simultaneously provided information about logistics for users, results of evaluation studies, and communication about the value of the tool. Future projects should structure their communications so discussions about potential value occur before logistics. To the extent possible, providing separate presentations and materials for specific audiences may be more effective than trying to include everything in one presentation.
Make direct comparisons with existing tools
If we want the future physician workforce to reflect the demographics of the United States, we need to develop and use tools that have small or no group differences. The SVI is the only tool available for use at the residency preinterview screening stage that does not have performance differences by race or ethnicity.5 In developing and piloting the SVI, the AAMC took several steps to reduce the risk of group differences, including conducting bias and sensitivity reviews of questions and the scoring rubric; providing rigorous, mandatory training to raters on a standardized rating process and on unconscious bias; offering optional unconscious bias training to EM program directors and staff; and conducting and publishing research on group differences in SVI performance. While this information about lack of group differences was included in the AAMC’s outreach efforts, it was often presented with other validity evidence that may have diluted its impact. This important finding should have been messaged clearly and repeatedly as it has potential workforce implications. Moving forward, the medical education community should demand that organizations take steps to proactively minimize the risk of group differences, evaluate group differences, and share findings about group differences for currently used and future residency selection tools.
Give new tools time and space to succeed
The AAMC expected program directors would welcome the SVI as an innovative disruption given the ongoing conversation about the overreliance on academic metrics in the residency selection process. The implementation of any new selection method, particularly one reliant on technology, deserves careful introduction and justification. Many in the EM community approached the SVI with healthy skepticism. Others were reluctant to give the SVI a chance. Often, they held the SVI to a higher standard than existing tools and appraised its value more critically. When comparing tools in the future, both the AAMC and the medical education community must be open to following the data and judging each tool on its merits.
In addition, the timeline for launching and evaluating new tools should align with their intended use and availability of outcome data. The SVI was launched quickly, which created a barrier for program directors to be able to train reviewers in use of the new tool. Launching the SVI with an annual review timeline also contributed to confusion and anxiety in the community. While intended to communicate that the AAMC was committed to evidence-based evaluation, the annual review may have unintentionally communicated that the SVI was not ready for operational use. Future tools should be launched with a longer runway. In the case of the SVI, doing so might have reduced anxiety in the community and communicated that the AAMC stood behind the new tool.
Strike a balance between early adopters and broad participation
The AAMC offered the SVI to the entire EM community to minimize the risk of a decline in applications to participating programs due to the addition of a new requirement. An unintended consequence was that the SVI was perceived to be mandatory even for programs that may not have wanted to participate. This contributed to negative feelings about the SVI and an “us versus them” mentality among some groups. In future projects, it may be more effective to begin by working with individual program directors—early adopters—who want a new tool because they believe it aligns with their goals and is likely to improve their selection process. Early adopters may be more invested in the benefits of using the new tool, less likely to be swayed by the larger community, and more likely to be champions of the tool if it meets their needs.
Help stakeholders understand the limitations of what a tool can do
Some contexts are not appropriate for criterion studies, so test developers and test users must rely on other evidence of validity. Residency selection is not well suited for examining correlations between test performance and resident performance as measured by the Accreditation Council for Graduate Medical Education milestones. In addition, reliable measures of first-year resident performance in nontechnical areas, such as teamwork and cultural competence, are limited. When measures do exist, they lack variance and/or are program-specific, making it difficult to combine data across programs. At the program level, sample sizes are too small to interpret results.
However, many stakeholders may have hoped for criterion evidence to support the SVI. The AAMC should have set realistic expectations about the challenges of studying the correlation between SVI scores and resident performance. In future projects, researchers and educators considering the implementation of new tools should help the medical education community understand that validity evidence based on content and relations with other variables is consistent with best practices in the testing industry and meets the standard for operational use.6
Set clear expectations about stakeholder input
Expecting applicants to have a positive reaction to a selection tool, especially a new one, is not realistic, and the human resources literature supports this.7–9 Program directors and faculty are the key decision makers about whether a new tool adds value to the residency selection process because they understand what their resident positions entail and the competencies required at entry to be successful. Furthermore, part of their role in residency selection is to determine how best to collect high-quality information about applicants that can be used to make effective and efficient decisions about their qualifications. Applicants’ role in the launch of a new tool should be to provide feedback about the testing experience and clarity of the policies, instructions, and test preparation products. Establishing who the “customer” is in advance of launching a new tool will help set appropriate expectations and roles for applicants and other non-decision-making stakeholders at the onset of the project.
Set clear expectations about pricing
From the onset of the SVI project, the EM community had questions about the future pricing and payment model. The value they perceived stemmed from the cost as well as the quality and performance of the SVI—these could not be disentangled. The AAMC spent 4 years exploring pricing but was unable to reach a decision. This created a dilemma: The AAMC’s focus was on developing a tool that would add value to the community, with pricing as a secondary consideration, whereas the EM community could not answer questions about the SVI’s value without knowing the price. As time went on, the EM community grew more distrustful about future pricing, and the AAMC’s credibility on this project diminished. To be successful, future projects should have pricing decisions from the onset (or very near to it) whenever possible.
There were many lessons learned from the SVI project, but there were also successes. The AAMC and the EM community took a chance on an innovation. The AAMC invested over 4 years of research and development in the SVI, and the EM community devoted the time of their program directors, faculty, clerkship directors, department chairs, and applicants. Working together, the AAMC and these stakeholders found a way to introduce a new selection tool using a new platform. Over 10,000 applicants completed the SVI over the 3-year pilot period (ERAS 2018–2020 cycles). The AAMC produced free SVI preparation materials, including 2 practice interviews. Applicants reported positive experiences with taking the SVI and with the procedures used to support it.10
The AAMC and the EMSVI working group also prioritized reducing the risk of bias. They developed a rigorous process to minimize group differences in ratings of applicants and in use of the SVI and offered free unconscious bias training to program directors.4 A tremendous amount of SVI research was conducted and shared, including validity evidence and group differences,4 applicant10 and program director11 reactions, and effects of test preparation12 and testing location13 on SVI scores. These studies demonstrated that the SVI has strong psychometric properties, is valid, and does not have racial/ethnic group differences.
The AAMC and the EM community recognize that one way to reduce the overreliance on academic metrics in residency selection is to develop and use new tools that assess other competencies. This need remains today. We encourage the medical education community to consider future partnerships with the AAMC or other specialty organizations to develop new tools and approaches that prioritize the community’s needs. Failure to seize such opportunities will open the door for a for-profit company to develop tools that prioritize profit. We also encourage the community to learn from the SVI project, as finding solutions to the challenges facing residency selection should be a priority for all stakeholders.
The authors thank the emergency medicine community for their courage to try something new; the Emergency Medicine Standardized Video Interview (EMSVI) working group for their continued efforts to improve the SVI; and John E. Prescott, MD, for his guidance and feedback along the way.
1. Prober CG, Kolars JC, First LR, Melnick DE. A plea to reassess the role of United States Medical Licensing Examination Step 1 scores in residency selection. Acad Med. 2016;91:12–15
2. Radabaugh CL, Hawkins RE, Welcher CM, et al. Beyond the United States Medical Licensing Examination score: Assessing competence for entering residency. Acad Med. 2019;94:983–989
3. Gardner AK, Cavanaugh KJ, Willis RE, Dunkin BJ. Can better selection tools help us achieve our diversity goals in postgraduate medical education? Comparing use of USMLE Step 1 scores and situational judgment tests at 7 surgical residencies. Acad Med. 2020;95:751–757
4. Bird SB, Hern HG, Blomkalns A, et al. Innovation in residency selection: The AAMC Standardized Video Interview. Acad Med. 2019;94:1489–1497
5. Hopson LR, Regan L, Bond MC, et al. The AAMC Standardized Video Interview and the electronic Standardized Letter of Evaluation in emergency medicine: A comparison of performance characteristics. Acad Med. 2019;94:1513–1521
6. American Educational Research Association; American Psychological Association; National Council on Measurement in Education. Standards for Educational and Psychological Testing. 2014.Washington, DC: American Educational Research Association;
7. Blacksmith N, Willford JC, Behrend TS. Technology in the employment interview: A meta-analysis and future research agenda. Pers Asses Decis. 2016;2:12–20
8. Campion MA, Palmer DK, Campion JE. A review of structure in the selection interview. Pers Psychol. 1997;50:655–702
9. Levashina J, Hartwell CJ, Morgeson FP, Campion MA. The structured employment interview: Narrative and quantitative review of the research literature. Pers Psychol. 2014;67:241–293
10. Deiorio NM, Jarou ZJ, Alker A, et al. Applicant reactions to the AAMC Standardized Video Interview during the 2018 application cycle. Acad Med. 2019;94:1498–1505
11. Gallahue FE, Hiller KM, Bird SB, et al. The AAMC Standardized Video Interview: Reactions and use by residency programs during the 2018 application cycle. Acad Med. 2019;94:1506–1512
12. Deiorio NM, Hopson LR, Fletcher LDo Use of Standardized Video Interview practice tests correlate with applicant score?. Presented at: SAEM19; May 14–17, 2019; Las Vegas, Nevada
13. Jarou Z, Karl E, Alker A, et al. Factors affecting Standardized Video Interview performance: Preparation elements and the testing environment. EM Resident. https://www.emra.org/emresident/article/svi-study-results
. April 17, 2018 Accessed June 8, 2020