Multiple-choice question (MCQ) examinations represent a primary mode of assessment used by medical schools. It can be challenging for faculty to produce content-aligned, comprehensive, and psychometrically sound MCQs. Despite best efforts, sometimes there are unexpected issues with examinations. Assessment best practices lack a systematic way to address gaps when actual and expected outcomes do not align.
The authors propose using root cause analysis (RCA) to systematically review unexpected educational outcomes. Using a real-life example of a class’s unexpectedly low reproduction examination scores (University of Michigan Medical School, 2015), the authors describe their RCA process, which included a system flow diagram, a fishbone diagram, and an application of the 5 Whys to understand the contributors and reasons for the lower-than-expected performance. Using this RCA approach, the authors identified multiple contributing factors that potentially led to the low examination scores. These included lack of examination quality improvement (QI) for poorly constructed items, content–question and pedagogy–assessment misalignment, and other issues related to environment and people.
As a result of the RCA, the authors worked with stakeholders to address these issues and develop strategies to prevent similar systematic issues from reoccurring. For example, a more robust examination QI process was developed.
Using an RCA approach in health care is grounded in practice and can be easily adapted for assessment. Because this is a novel use of RCA, there are opportunities to expand beyond the authors’ initial approach for using RCA in assessment.
S.A. Santen is currently senior associate dean of assessment, evaluation, and scholarship and professor, Department of Emergency Medicine, Virginia Commonwealth School of Medicine, Richmond, Virginia, and at the time of this work was assistant dean for educational research and quality improvement and professor, Departments of Emergency Medicine and Learning Health Sciences, University of Michigan Medical School, Ann Arbor, Michigan; ORCID: http://orcid.org/0000-0002-8327-8002.
K.L. Grob is assistant director of preclinical evaluation and assessment, Office of Medical Student Education, University of Michigan Medical School, Ann Arbor, Michigan.
S.U. Monrad is clinical associate professor, Departments of Internal Medicine and Learning Health Sciences, University of Michigan Medical School, Ann Arbor, Michigan.
C.M. Stalburg is division chief and associate professor, Division of Professional Education, Department of Learning Health Sciences, and associate professor, Department of Obstetrics and Gynecology, University of Michigan Medical School, Ann Arbor, Michigan.
G. Smith is professor, Departments of Obstetrics and Gynecology, Molecular & Integrative Physiology, and Urology, University of Michigan Medical School, Ann Arbor, Michigan.
R.R. Hemphill is chief quality and safety officer, Virginia Commonwealth University Health System, and associate dean for quality and safety and clinical professor, Department of Emergency Medicine, Virginia Commonwealth School of Medicine, Richmond, Virginia.
N.L. Bibler Zaidi is associate director of advancing scholarship, Office of Medical Student Education, University of Michigan Medical School, Ann Arbor, Michigan.
Funding/Support: The University of Michigan Medical School and S.A. Santen received funding from the American Medical Association Accelerating Change in Medical Education Grant.
Other disclosures: None reported.
Ethical approval: Reported as not applicable.
Correspondence should be addressed to Sally A. Santen, Department of Emergency Medicine, Virginia Commonwealth School of Medicine, 1201 E. Marshall St., 4th Floor, PO Box 980565, Richmond, VA 23298; telephone: (804) 828-5100; e-mail: Sally.Santen@vcuhealth.org.