Academic Medicine

Skip Navigation LinksHome > August 2008 - Volume 83 - Issue 8 > Validity Evidence for an OSCE to Assess Competency in System...
Academic Medicine:
doi: 10.1097/ACM.0b013e31817ec873
Assessment of Competencies

Validity Evidence for an OSCE to Assess Competency in Systems-Based Practice and Practice-Based Learning and Improvement: A Preliminary Investigation

Varkey, Prathibha MD, MPH, MHPE; Natt, Neena MD; Lesnick, Timothy MS; Downing, Steven PhD; Yudkowsky, Rachel MD, MHPE

Collapse Box


Purpose: To determine the psychometric properties and validity of an OSCE to assess the competencies of Practice-Based Learning and Improvement (PBLI) and Systems-Based Practice (SBP) in graduate medical education.

Method: An eight-station OSCE was piloted at the end of a three-week Quality Improvement elective for nine preventive medicine and endocrinology fellows at Mayo Clinic. The stations assessed performance in quality measurement, root cause analysis, evidence-based medicine, insurance systems, team collaboration, prescription errors, Nolan’s model, and negotiation. Fellows’ performance in each of the stations was assessed by three faculty experts using checklists and a five-point global competency scale. A modified Angoff procedure was used to set standards. Evidence for the OSCE’s validity, feasibility, and acceptability was gathered.

Results: Evidence for content and response process validity was judged as excellent by institutional content experts. Interrater reliability of scores ranged from 0.85 to 1 for most stations. Interstation correlation coefficients ranged from −0.62 to 0.99, reflecting case specificity. Implementation cost was approximately $255 per fellow. All faculty members agreed that the OSCE was realistic and capable of providing accurate assessments.

Conclusions: The OSCE provides an opportunity to systematically sample the different subdomains of Quality Improvement. Furthermore, the OSCE provides an opportunity for the demonstration of skills rather than the testing of knowledge alone, thus making it a potentially powerful assessment tool for SBP and PBLI. The study OSCE was well suited to assess SBP and PBLI. The evidence gathered through this study lays the foundation for future validation work.

© 2008 Association of American Medical Colleges


Article Tools


Article Level Metrics