Assessment of Communication Skills and Self-Appraisal in the Simulated Environment: Feasibility of Multirater Feedback with Gap Analysis

Calhoun, Aaron W. MD; Rider, Elizabeth A. MSW, MD; Meyer, Elaine C. RN, PhD; Lamiani, Giulia MS, EdM; Truog, Robert D. MD

Simulation in Healthcare: The Journal of the Society for Simulation in Healthcare:
doi: 10.1097/SIH.0b013e318184377a
Empirical Investigations
Abstract

Introduction: Multirater assessment is a powerful means of measuring communication skills. The use of gap analysis to assess self-appraisal is a strength of this technique. On the basis of Kalamazoo Consensus Statement framework and 360-degree assessment models, we developed a multirater instrument with gap analysis, with the goals of examining both communication skills and situational self-appraisal, and assessing the feasibility of the combined approach.

Methods: The multirater communication skills instrument was used to assess Pediatric and Neonatal Intensive Care fellows after participation in seven simulated family meetings. Instrument reliability was determined using Cronbach’s Alpha and Factorial Analysis. Correlations between rater groups were examined with Spearman’s Rank Coefficient. Gap analyses and rater perceptions of the instruments were analyzed using descriptive statistics.

Results: Seven pediatric intensive care unit and neonatal intensive care fellows were each assessed by 11 to 18 raters (108 total assessments). Correlations were identified between disciplinary groups within each encounter. Among the 7 fellows, 30 communication strengths or areas needing improvement and 24 significant gaps were identified, indicating self under-appraisals, 9 (38%) of which overlapped. The instrument was logistically feasible and well received.

Conclusions: Our multirater communication skills instrument with gap analysis proved useful in identifying areas of strength and areas needing improvement, and in highlighting areas of self over- and under-appraisal that require focused feedback. The use of multirater assessment with gap analysis, in a simulated and “safe” environment, may assist in the delivery of feedback to trainees.

Author Information

From the Division of Pediatric Critical Care Medicine (A.W.C.), University of Louisville, Louisville, KY; Institute for Professionalism and Ethical Practice (A.W.C., E.A.R., E.C.M., G.L.), Children’s Hospital Boston, Harvard Medical School, Boston, MA; Departments of Pediatrics (E.A.R.) and Medical Ethics (R.D.T.), Harvard Medical School, Boston, MA; Stoeckle Center for Primary Care Innovation (E.A.R.), Massachusetts General Hospital, MA; Departments of Anaesthesia (E.C.M., R.D.T.) and Psychiatry (E.C.M.), Children’s Hospital Boston, Boston, MA; Institute of Medical Psychology, San Paolo Hospital, University of Milan (G.L.), Milan, Italy

Reprints: Dr. Aaron William Calhoun, MD, 571 S. Floyd St. STE 332, Louisville, KY 40202 (e-mail: aaron.calhoun@louisville.edu).

The authors have indicated that they have no conflict of interest to disclose.

© 2009 Lippincott Williams & Wilkins, Inc.