A Multisite, Multistakeholder Validation of the Accreditation Council for Graduate Medical Education Competencies

Smith, C. Scott MD; Morris, Magdalena RN, MSN; Francovich, Chris EdD; Tivis, Rick MPH; Bush, Roger MD; Sanders, Shelley Schoepflin MD; Graham, Jeremy DO; Niven, Alex MD; Kai, Mari MD; Knight, Christopher MD; Hardman, Joseph MD; Caverzagie, Kelly MD; Iobst, William MD; for the Pacific Northwest Consortium for Outcomes in Residency Education

Academic Medicine:
doi: 10.1097/ACM.0b013e3182951efc
Research Reports
Abstract

Purpose: The Accreditation Council for Graduate Medical Education’s (ACGME’s) six-competency framework has not been validated across multiple stakeholders and sites. The objective of this study was to perform a multisite validation with five stakeholder groups.

Method: This was a cross-sectional, observational study carried out from October to December, 2011, in the internal medicine residency continuity clinics of eight internal medicine residency programs in the Pacific Northwest, including a VA, two academic medical centers, a military medical center, and four private hospitals. The authors performed a cultural consensus analysis (CCA) and a convergent-discriminant analysis using previously developed statements based on internal medicine milestones related to the six competencies. Ten participants were included from each of five stakeholder groups: patients, nurses, residents, faculty members, and administrators from each training site (total: 400 participants).

Results: Moderate to high agreement and coherence for all groups were observed (CCA eigenvalue ratios ranging from 2.16 to 3.20); however, high differences in ranking order were seen between groups in four of the CCA statements, which may suggest between-group tension in these areas. Analyses revealed excellent construct validity (Zcontrast score of 5.323, P < .0001) for the six-competency framework. Average Spearman correlation between same-node statements was 0.012, and between different-node statements it was –0.096.

Conclusions: The ACGME’s six-competency framework has reasonable face and construct validity across multiple stakeholders and sites. Stakeholders appear to share a single mental model of competence in this learning environment. Data patterns suggest possible improvements to the competency-milestone framework.

Author Information

Dr. Smith is director, Boise VA Center of Excellence in Primary Care Education, Boise, Idaho, and professor of medicine, University of Washington, Seattle, Washington.

Ms. Morris is nursing faculty, Carrington College, Boise, Idaho.

Dr. Francovich is associate professor of leadership studies, Gonzaga University, Spokane, Washington.

Mr. Tivis is statistician, Boise VA Center of Excellence in Primary Care Education, Boise, Idaho.

Dr. Bush is attending physician, Virginia Mason Medical Center, Seattle, Washington.

Dr. Schoepflin Sanders is clinical faculty instructor, Internal Medicine Residency, Providence St. Vincent Medical Center, Portland, Oregon.

Dr. Graham is faculty, Providence Internal Medicine Residency Spokane, Spokane, Washington.

Dr. Niven is clinical associate professor of medicine, University of Washington, Seattle, Washington, and internal medicine program director, Madigan Healthcare System, Tacoma, Washington.

Dr. Kai is affiliate associate professor of medicine, Oregon Health & Science University, and faculty, General Internal Medicine, Providence Portland Medical Center, Portland, Oregon.

Dr. Knight is associate professor of medicine, University of Washington, Seattle, Washington.

Dr. Hardman is assistant professor of medicine, Oregon Health & Science University, Portland, Oregon.

Dr. Caverzagie is associate vice chair for quality and physician competence, Internal Medicine, University of Nebraska Medical Center, Omaha, Nebraska.

Dr. Iobst is vice president, Academic Affairs, American Board of Internal Medicine, Philadelphia, Pennsylvania.

Correspondence should be addressed to Dr. Smith, Boise VA Center of Excellence in Primary Care Education, VA Medical Center (111), 500 W. Fort St., Boise, ID 83702; telephone: (208) 422-1325; fax: (208) 422-1319; e-mail: scott.smith2@va.gov.

© 2013 by the Association of American Medical Colleges