Secondary Logo

Journal Logo

Institutional members access full text with Ovid®

Exploring Validity Evidence Associated With Questionnaire-Based Tools for Assessing the Professional Performance of Physicians: A Systematic Review

van der Meulen, Mirja W. MSc; Smirnova, Alina MD, PhD; Heeneman, Sylvia MD, PhD; oude Egbrink, Mirjam G.A. MD, PhD; van der Vleuten, Cees P.M. PhD; Lombarts, Kiki M.J.M.H. PhD

doi: 10.1097/ACM.0000000000002767
Reviews
Buy
SDC

Purpose To collect and examine—using an argument-based validity approach—validity evidence of questionnaire-based tools used to assess physicians’ clinical, teaching, and research performance.

Method In October 2016, the authors conducted a systematic search of the literature seeking articles about questionnaire-based tools for assessing physicians’ professional performance published from inception to October 2016. They included studies reporting on the validity evidence of tools used to assess physicians’ clinical, teaching, and research performance. Using Kane’s validity framework, they conducted data extraction based on four inferences in the validity argument: scoring, generalization, extrapolation, and implications.

Results They included 46 articles on 15 tools assessing clinical performance and 72 articles on 38 tools assessing teaching performance. They found no studies on research performance tools. Only 12 of the tools (23%) gathered evidence on all four components of Kane’s validity argument. Validity evidence focused mostly on generalization and extrapolation inferences. Scoring evidence showed mixed results. Evidence on implications was generally missing.

Conclusions Based on the argument-based approach to validity, not all questionnaire-based tools seem to support their intended use. Evidence concerning implications of questionnaire-based tools is mostly lacking, thus weakening the argument to use these tools for formative and, especially, for summative assessments of physicians’ clinical and teaching performance. More research on implications is needed to strengthen the argument and to provide support for decisions based on these tools, particularly for high-stakes, summative decisions. To meaningfully assess academic physicians in their tripartite role as doctor, teacher, and researcher, additional assessment tools are needed.

M.W. van der Meulen is PhD candidate, Department of Educational Development and Research, Faculty of Health, Medicine and Life Sciences, School of Health Professions Education, Maastricht University, Maastricht, The Netherlands, and member, Professional Performance Research Group, Medical Psychology, Academic Medical Center, University of Amsterdam, Amsterdam, The Netherlands; ORCID: https://orcid.org/0000-0003-3636-5469.

A. Smirnova is PhD graduate and researcher, Department of Educational Development and Research, Faculty of Health, Medicine and Life Sciences, School of Health Professions Education, Maastricht University, Maastricht, The Netherlands, and member, Professional Performance Research Group, Medical Psychology, Academic Medical Center, University of Amsterdam, Amsterdam, The Netherlands; ORCID: https://orcid.org/0000-0003-4491-3007.

S. Heeneman is professor, Department of Pathology, Faculty of Health, Medicine and Life Sciences, School of Health Professions Education, Maastricht University, Maastricht, The Netherlands; ORCID: https://orcid.org/0000-0002-6103-8075.

M.G.A. oude Egbrink is professor, Department of Physiology, Faculty of Health, Medicine and Life Sciences, School of Health Professions Education, Maastricht University, Maastricht, The Netherlands; ORCID: https://orcid.org/0000-0002-5530-6598.

C.P.M. van der Vleuten is professor, Department of Educational Development and Research, Faculty of Health, Medicine and Life Sciences, School of Health Professions Education, Maastricht University, Maastricht, The Netherlands; ORCID: https://orcid.org/0000-0001-6802-3119.

K.M.J.M.H. Lombarts is professor, Professional Performance Research Group, Medical Psychology, Academic Medical Center, University of Amsterdam, Amsterdam, The Netherlands; ORCID: https://orcid.org/0000-0001-6167-0620.

Funding/Support: None reported.

Other disclosures: None reported.

Ethical approval: Reported as not applicable.

Previous presentations: This review was presented at the 2018 biannual OTTAWA-International Conference on Medical Education Conference in Abu Dhabi, United Arab Emirates, March 14, 2018; at the 2018 annual Association of Medical Educators of Europe Conference in Basel, Switzerland on August 28, 2018 as an oral presentation; and at the 2018 annual Nederlandse Vereniging voor Medisch Onderwijs Congress in Egmond aan Zee, The Netherlands, November 16, 2018.

Supplemental digital content for this article is available at http://links.lww.com/ACADMED/A677.

Correspondence should be addressed to Mirja W. van der Meulen, Professional Performance Research Group, Academic Medical Center, University of Amsterdam, Meibergdreef 9, PO Box 22700, 1100 DD Amsterdam, The Netherlands; telephone +31 (0) 20-566-1274; email: m.w.vandermeulen@amc.uva.nl or m.vandermeulen@maastrichtuniversity.nl; Twitter: @Mirja_vd_Meulen.

Copyright © 2019 by the Association of American Medical Colleges