Using Contribution Analysis to Evaluate Competency-Based Medical Education Programs: Its All About Rigor in Thinking

Van Melle, Elaine PhD; Gruppen, Larry PhD; Holmboe, Eric S. MD; Flynn, Leslie MD, MEd; Oandasan, Ivy MD, MHSc; Frank, Jason R. MD, MAEd; for the International Competency-Based Medical Education Collaborators

doi: 10.1097/ACM.0000000000001479
Articles

Competency-based medical education (CBME) aims to bring about the sequential acquisition of competencies required for practice. Although it is being adopted in centers of medical education around the globe, there is little evidence concerning whether, in comparison with traditional methods, CBME produces physicians who are better prepared for the practice environment and contributes to improved patient outcomes. Consequently, the authors, an international group of collaborators, wrote this article to provide guidance regarding the evaluation of CBME programs.

CBME is a complex service intervention consisting of multiple activities that contribute to the achievement of a variety of outcomes over time. For this reason, it is difficult to apply traditional methods of program evaluation, which require conditions of control and predictability, to CBME. To address this challenge, the authors describe an approach that makes explicit the multiple potential linkages between program activities and outcomes. Referred to as contribution analysis (CA), this theory-based approach to program evaluation provides a systematic way to make credible causal claims under conditions of complexity. Although CA has yet to be applied to medical education, the authors describe how a six-step model and a postulated theory of change could be used to examine the link between CBME, physicians’ preparation for practice, and patient care outcomes.

The authors argue that adopting the methods of CA, particularly the rigor in thinking required to link program activities, outcomes, and theory, will serve to strengthen understanding of the impact of CBME over time.

E. Van Melle is senior education scientist, Royal College of Physicians and Surgeons of Canada, Ottawa, Ontario, Canada, and clinician–educator, Royal College of Physicians and Surgeons of Canada, Ottawa, Ontario, Canada.

L. Gruppen is professor, Department of Learning Health Sciences, University of Michigan Medical School, Ann Arbor, Michigan.

E.S. Holmboe is senior vice president, Milestones Development and Evaluation, Accreditation Council for Graduate Medical Education, Chicago, Illinois.

L. Flynn is vice dean of education, Faculty of Health Sciences, Queen’s University, Kingston, Ontario, Canada, and clinician–educator, Royal College of Physicians and Surgeons of Canada, Ottawa, Ontario, Canada.

I. Oandasan is director of education, College of Family Physicians of Canada, Mississauga, Ontario, Canada, and professor, Department of Family and Community Medicine, University of Toronto Faculty of Medicine, Toronto, Ontario, Canada.

J.R. Frank is director, Specialty Education, Strategy, and Standards, Office of Specialty Education, Royal College of Physicians and Surgeons, Ottawa, Ontario, Canada.

Funding/Support: None reported.

Other disclosures: None reported.

Ethical approval: Reported as not applicable.

Correspondence should be addressed to Elaine Van Melle, 33 Hill St., Kingston, ON, Canada, K7L 2M4; e-mail: vanmelle@queensu.ca.

© 2017 by the Association of American Medical Colleges