Measurement of the General Competencies of the Accreditation Council for Graduate Medical Education: A Systematic Review

Lurie, Stephen J. MD, PhD; Mooney, Christopher J. MA; Lyness, Jeffrey M. MD

doi: 10.1097/ACM.0b013e3181971f08
ACGME Issues

Purpose: To evaluate published evidence that the Accreditation Council for Graduate Medical Education's six general competencies can each be measured in a valid and reliable way.

Method: In March 2008, the authors conducted searches of Medline and ERIC using combinations of search terms “ACGME,” “Accreditation Council for Graduate Medical Education,” “core competencies,” “general competencies,” and the specific competencies “systems-based practice” (SBP) and “practice based learning and improvement (PBLI).” Included were all publications presenting new qualitative or quantitative data about specific assessment modalities related to the general competencies since 1999; opinion pieces, review articles, and reports of consensus conferences were excluded. The search yielded 127 articles, of which 56 met inclusion criteria. Articles were subdivided into four categories: (1) quantitative/psychometric evaluations, (2) preliminary studies, (3) studies of SBP and PBLI, and (4) surveys.

Results: Quantitative/psychometric studies of evaluation tools failed to develop measures reflecting the six competencies in a reliable or valid way. Few preliminary studies led to published quantitative data regarding reliability or validity. Only two published surveys met quality criteria. Studies of SBP and PBLI generally operationalized these competencies as properties of systems, not of individual trainees.

Conclusions: The peer-reviewed literature provides no evidence that current measurement tools can assess the competencies independently of one another. Because further efforts are unlikely to be successful, the authors recommend using the competencies to guide and coordinate specific evaluation efforts, rather than attempting to develop instruments to measure the competencies directly.

Dr. Lurie is director of assessment, Office of Curriculum and Assessment, University of Rochester School of Medicine and Dentistry, Rochester, New York.

Mr. Mooney is information analyst, Office of Curriculum and Assessment, University of Rochester School of Medicine and Dentistry, Rochester, New York.

Dr. Lyness is director of curriculum, Office of Curriculum and Assessment, University of Rochester School of Medicine and Dentistry, Rochester, New York.

Correspondence should be addressed to Dr. Lurie, University of Rochester School of Medicine and Dentistry, 601 Elmwood Ave, Box 601, Rochester NY, 14642; telephone: (585) 273-4323; e-mail: (Stephen_Lurie@urmc.rochester.edu).

© 2009 Association of American Medical Colleges