Share this article on:

Developing a Peer Assessment of Lecturing Instrument: Lessons Learned

Newman, Lori R. MEd; Lown, Beth A. MD; Jones, Richard N. ScD; Johansson, Anna PhD; Schwartzstein, Richard M. MD

doi: 10.1097/ACM.0b013e3181ad18f9
Faculty Issues

Peer assessment of teaching can improve the quality of instruction and contribute to summative evaluation of teaching effectiveness integral to high-stakes decision making. There is, however, a paucity of validated, criterion-based peer assessment instruments. The authors describe development and pilot testing of one such instrument and share lessons learned. The report provides a description of how a task force of the Shapiro Institute for Education and Research at Harvard Medical School and Beth Israel Deaconess Medical Center used the Delphi method to engage academic faculty leaders to develop a new instrument for peer assessment of medical lecturing. The authors describe how they used consensus building to determine the criteria, scoring rubric, and behavioral anchors for the rating scale. To pilot test the instrument, participants assessed a series of medical school lectures. Statistical analysis revealed high internal consistency of the instrument’s scores (alpha = 0.87, 95% bootstrap confidence interval [BCI] = 0.80 to 0.91), yet low interrater agreement across all criteria and the global measure (intraclass correlation coefficient = 0.27, 95% BCI = −0.08 to 0.44).

The authors describe the importance of faculty involvement in determining a cohesive set of criteria to assess lectures. They discuss how providing evidence that a peer assessment instrument is credible and reliable increases the faculty’s trust in feedback. The authors point to the need for proper peer rater training to obtain high interrater agreement measures, and posit that once such measures are obtained, reliable and accurate peer assessment of teaching could be used to inform the academic promotion process.

Ms. Newman is acting director, Faculty Programs in Medical Education, and codirector, Rabkin Fellowship in Medical Education, Shapiro Institute for Education and Research, Harvard Medical School and Beth Israel Deaconess Medical Center; and associate in medicine, Harvard Medical School, Boston, Massachusetts.

Dr. Lown is director of faculty development, Department of Medicine, Mount Auburn Hospital; codirector, Rabkin Fellowship in Medical Education, Shapiro Institute for Education and Research, Harvard Medical School and Beth Israel Deaconess Medical Center, The Mount Auburn Fellowship in Medical Education, and The Harvard Medical School Academy Fellowship in Medical Education; and assistant professor of medicine, Harvard Medical School, Boston, Massachusetts.

Dr. Jones is associate director, Social and Health Policy Research, Institute for Aging Research, Hebrew SeniorLife, Harvard Medical School; and assistant professor of medicine, Harvard Medical School, Boston, Massachusetts.

Dr. Johansson is assistant director, Office of Educational Research, Shapiro Institute for Education and Research at Harvard Medical School and Beth Israel Deaconess Medical Center; and instructor in medicine, Harvard Medical School, Boston, Massachusetts.

Dr. Schwartzstein is vice president for education, Beth Israel Deaconess Medical Center; faculty associate dean for medical education, Harvard Medical School; executive director, Shapiro Institute for Education and Research at Harvard Medical School and Beth Israel Deaconess Medical Center; associate chief, Division of Pulmonary and Critical Care Medicine, Beth Israel Deaconess Medical Center; and professor of medicine, Harvard Medical School, Boston, Massachusetts.

Please see the end of this article for information about the authors.

Correspondence should be addressed to Ms. Newman, Shapiro Institute for Education and Research at Harvard Medical School and Beth Israel Deaconess Medical Center, 330 Brookline Avenue, E/ES-204, Boston, MA 02215; telephone: (617) 667-4742; fax: (617) 667-9122; e-mail: (lnewman@bidmc.harvard.edu).

Dr. Jones is supported in part by NIH Grant AG008812, “Biostatistics and Evaluation Core, Harvard Older Americans Independence Center.”

© 2009 Association of American Medical Colleges