AbstractPurpose.To examine the inter-rater reliability of expert and non-expert observers when they used objective structured checklists to evaluate candidates' performances on three simulated medical procedures.Method.Simulations and structured checklists were developed for three medical procedures: endotracheal intubation, application of a forearm cast, and suturing a simple skin laceration. Groups comprised of two expert and two non-expert observers scored the performances of 101 procedures by 38 medical trainees and practitioners of varying skill levels. Inter-rater reliability was assessed using Pearson correlation coefficients.Results.Inter-rater reliability was good for expert/expert, expert/non-expert, and non-expert/non-expert pairings in all three skills simulations.Conclusion.Both expert and non-expert observers demonstrated good inter-rater reliability when using structured checklists to assess procedural skills. Further study is required to determine whether this conclusion may be extrapolated to other study groups or procedures.Acad. Med. 1999;74:76–78.
Purpose.
To examine the inter-rater reliability of expert and non-expert observers when they used objective structured checklists to evaluate candidates' performances on three simulated medical procedures.
Method.
Simulations and structured checklists were developed for three medical procedures: endotracheal intubation, application of a forearm cast, and suturing a simple skin laceration. Groups comprised of two expert and two non-expert observers scored the performances of 101 procedures by 38 medical trainees and practitioners of varying skill levels. Inter-rater reliability was assessed using Pearson correlation coefficients.
Results.
Inter-rater reliability was good for expert/expert, expert/non-expert, and non-expert/non-expert pairings in all three skills simulations.
Conclusion.
Both expert and non-expert observers demonstrated good inter-rater reliability when using structured checklists to assess procedural skills. Further study is required to determine whether this conclusion may be extrapolated to other study groups or procedures.
Acad. Med. 1999;74:76–78.
Dr. Bullockis an assistant professor in the Divisions of Emergency Medicine and Medical Education, Faculty of Medicine, Dalhousie University, Halifax, Nova Scotia, Canada.Dr. Kovacsis an Assistant Professor in the Divisions of Emergency Medicine and Medical Education, Dalhousie University.Ms. MacdonaldandDr. Storyare enrolled in the undergraduate MD program of Dalhousie University.
Correspondence and requests for reprints should be addressed to Dr. Bullock, Dalhousie University, Division of Medical Education, c/o Office of the Dean of Medicine, 5849 University Avenue, Lower Halifax, Nova Scotia, B3H 4H7, Canada; e-mail: <[email protected]>.
Acad. Med. 1999;74:76–78.
© 1999 by the Association of American Medical Colleges
