Nofziger, Anne C. MD; Naumburg, Elizabeth H. MD; Davis, Barbara J. PhD; Mooney, Christopher J. MA; Epstein, Ronald M. MD
Teaching, modeling, and assessing professionalism in medical education are widely recognized as important.1–5 Professionalism in medicine is a complex construct that encompasses behavioral (e.g., honesty, teamwork), attitudinal (e.g., altruism, duty), and social (self-regulation and maintaining competence) factors. Professionalism can be viewed on a spectrum from rule-based adherence to behavioral norms6 to a developmental, formative process7,8; professionalism has to do with “both practice and identity.”7 It is not clear how professionalism should be assessed9 or whether feedback about professional behaviors improves future performance.10
Peer assessment has been recommended as one way to measure and promote professionalism,1,9 as medical trainees know each other well and make close observations of their classmates' work in a variety of contexts. The exercise of providing clear, helpful feedback to peers is itself training in professionalism. Students find peer assessment acceptable,11 and surveys have explored what conditions can encourage or prevent participation in peer assessment.12 One report suggests that peer assessment may enhance future professional behavior, finding that students assigned to participate in peer assessment subsequently received higher marks for professional behaviors from their tutors.13
We have reported, on the basis of eight years of experience with required peer assessments by second- and third-year medical students, that peers provide reliable, stable ratings of both work habits (e.g., preparation, problem solving, initiative) and interpersonal attributes (e.g., truthfulness, respect, integrity, empathy)11,14,15; see Chart 1. Peers' ratings of work habits predict future measures of achievement such as clerkship grades and residency directors' evaluations.16 Peer ratings of interpersonal attributes on this scale were also stable over time; however, they did not correlate with grades and residency directors' evaluations.16 This is not surprising, because interpersonal attributes typically do not form a large component of these assessments. Peer assessments may be one of the few ways these critical interpersonal professional attributes can be measured.
Credible and regular feedback is necessary for developing the insight and self-monitoring needed for reflective professional practice.17 Faculty and resident teachers, observing students mostly in structured settings, over limited times, and when students know they are being observed, may have limited insight into students' professional development.18,19 For these and other reasons, teachers may not address professionalism concerns with students or describe possible problems in formal evaluations.20,21 Students view the reluctance of teachers to confront a student about unprofessional behavior as a significant obstacle to effective evaluation of professional behaviors.22,23
The question remains, what is the impact of peer assessment on future professional development? We undertook this study to answer the following questions.
1. What types of peer feedback do medical students remember months to years later?
2. What kinds of immediate and delayed reactions do students have to peer feedback?
3. What transformations in attitudes and/or behaviors do students make in response to peer feedback?
Participants were all students in the University of Rochester School of Medicine and Dentistry's medical school classes of 2005 (fourth-year students) and 2007 (second-year students) who elected to participate in a voluntary study in May 2005. Students from the class of 2005 had experienced two prior peer assessments, one and two years before the study. Students from the class of 2007 first participated in peer assessment two months before the study.
Peer assessment is one element of a formative, multimodal comprehensive assessment program (CA) at the University of Rochester School of Medicine, required for all second- and third-year students.24 Students reflect on performance reports from multiple elements and create an individual learning plan for the following year. The development and follow-through of learning plans is supported through dedicated group and individual student meetings with advisory deans.* Details of the CA and of the peer assessment process, instrument, reliability, and predictive validity have been reported elsewhere.11,14–16,24
Assignment of peer raters.
In Year 2, all students evaluate 10 of their peers, and each student receives 10 peer evaluations. Assignments are made according to contact in small-group settings such as problem-based learning (PBL), interviewing, and lab groups. In Year 3, because students' contact is limited to smaller groups in multiple clinical locations, each student completes and receives six peer assessments.
Before participating in peer assessment, students are introduced to the process via an interactive workshop on giving high-quality feedback. Confidentiality of peer assessment data is emphasized, and students' questions are addressed.
Peer assessment instrument.
Peer assessments are completed online using a standardized 15-item scale that measures two dimensions: “work habits” and “interpersonal attributes”11,14; see Chart 1. Students also provide narrative comments describing their peers' strengths and weaknesses. Students are encouraged to sign their comments, or they may respond anonymously; typically more than 80% of comments are signed. Comments are reviewed by the CA directors, and rare inappropriate comments are deleted (0%–1% of comments) before final reports are released to students. Students also receive copies of all narrative comments they wrote to their classmates, with any deleted material highlighted for their consideration. In contrast to other CA results, peer assessment results are not automatically available to the students' advisory deans. However, students are required to discuss peer assessment results during an individual meeting with their advisory deans.
Peer assessment impact narratives
Information and consent.
We described the planned study to all students during class meetings in May 2005, then sent an e-mail request to both classes shortly after these presentations. In May 2005, second-year students had finished their peer assessments two months previously, and fourth-year students had completed two peer assessments, 11 months and two years previously. An information sheet provided instructions for responding to the question online, stated that participation was voluntary, and explained that students' choice to participate would indicate consent for their responses to be included in analysis and possible publication of results. An incentive (lottery for a bookstore gift certificate) was offered based on the level of participation. The University of Rochester Medical Center's institutional review board granted “exempt” approval for the study.
Students who agreed to participate responded to the following questions online during the stated one-week study period.
Please describe a comment or rating on the peer assessments that you received during the Comprehensive Assessment that had a meaningful effect on your professional or personal development. This may be a small insight or may have had a more significant impact; it might be something about which you felt positive, negative or mixed. Consider including the following:
* Describe specifically the rating(s) or comment(s) that affected you and how.
* If relevant, describe how the peer assessment process may have affected you generally.
* Did your reaction to this feedback change or evolve over time?
* Describe any deliberate changes you made in response to any aspect of the peer assessment.
We recorded responses in a secure database with numeric identifiers only, for purposes of recording class year and sex.
Qualitative analysis of narratives
We deidentified all of the narrative responses and used Atlas.ti software for qualitative analysis. Two members of the team (E.N., B.D.) used an iterative process to review the narratives, identify themes, and build a list of codes, coming together regularly with the team for discussion. When no further new themes were generated by review of additional data, the code list was considered complete (saturation).26 All narratives were dual-coded, and the two coders again met regularly with a third team member (A.N.). We resolved all discrepancies in coding by consensus. To present the results, we organized codes into larger categories and made frequency counts. Finally, we identified representative quotations to demonstrate essential elements of the emergent themes. No student narrative was quoted more than once.
We developed codes within four general categories: (1) recalled content of the peer assessment, (2) cognitive reactions to the peer assessment, (3) emotional reactions to the peer assessment, and (4) personal transformations related to peer assessment. Another group of codes addressed narrative content about the process of peer assessment. Finally, we identified interpretive codes, used when implied issues in a narrative were not explicitly stated but were felt to be important (see Table 1).
In total, we collected 138 responses, from 68 of the 83 students in the fourth-year class (82% response rate) and 70 of the 101 students in the second-year class (69% response rate). The sex distribution among respondents was similar to that of their respective classes: 58% of the fourth-year participants were women, while their class was 54% female; the numbers for the second-year students were 53% versus 55% women. Table 1 shows the themes that were identified, with numbers representing the frequency with which the codes were applied and a breakdown by class. Because all themes were expressed by both second- and fourth-year students, we aggregated the qualitative results presented below. However, we highlight that in our sample, second-year students generally described more content, more emotional and cognitive reactions, and more personal transformations as a result of the peer feedback, compared with fourth-year students.
Content students remembered from peer assessments
A majority of respondents from each class (51 [73%] second-year students and 44 [63%] fourth-year students) described specific feedback they remembered from peer assessments. Within the theme of content of peer feedback, students remembered positive feedback (e.g., about teamwork, leadership, teaching skills, and others) and negative feedback (e.g., being too quiet, dominating discussions, overconfidence, work ethic concerns, low self-esteem, and personal appearance). Eight students noted that they received specific suggestions for behavior change, 32 mentioned receiving similar feedback from multiple peers, and 6 commented on conflicting statements from different peer evaluators (see Table 1).
How students reacted to peer feedback
Students' reactions to peer feedback were both cognitive and emotional. Cognitive reactions included thoughts about whether feedback was important, helpful, surprising or anticipated, encouraging, interesting, or credible. Emotional reactions ranged from appreciation, encouragement, and gratification to anger, shock, hurt, and guilt. Some (8) students described changes in these feelings over time. Ninety-two students (67%; 77% of second-year students, 56% of fourth-year students) said they found the peer assessment helpful, comforting, and/or confirming of some personal trait that they were aware of. The following quote illustrates several of these themes.
I had nine comments during my first peer assessment which told me that I was too quiet. While this was something I was already aware of, it helped to hear what my classmates thought. With my dean, we sought to change this by coming up with specific changes in behavior to make me participate more in group settings. I made this one of my learning goals for the comprehensive assessment. I felt a dramatic improvement over the next two years but still feel that this is a work in progress. I had less comments about me being too quiet on my second comprehensive assessment.
Transformations students reported in response to peer feedback.
Eighty-nine students (65%; 74% of second-year students, 54% of fourth-year students) described at least one change in awareness, attitude, or behavior (see Table 1). Most of the reported transformations were in the direction of positive growth, with such responses as speaking up more in groups, efforts to be more patient or more of a team player, improved punctuality, and increased motivation. Some students described specific responses facilitating transformations, such as discussing the feedback with a friend or their advisory dean. Negative feedback from peers was more likely to result in a student reporting a transformation. Of the 89 students who reported any transformation, 70 (79%; 41 of 52 [79%] second-year students and 29 of 37 [78%] fourth-year students) also described peer feedback they remembered that contained specific negative or critical comments. Only 10 (20%) of the students reporting specific negative feedback failed to report a personal transformation.
A few people noted that I become impatient with others when they do not pick things up as quickly as I do. This is something I know that I do, but was not really aware that it showed. I have immediately tried to change the way I act when I understand something before others. I will continue to work on this.
Transformations were also described by students who reported positive or mixed feedback that had significance for them. The following example demonstrates the outcome for one student whose self-image was different from her peers' perception, and provides information about the credibility of peers as evaluators for this student.
One comment that had a meaningful effect on both my professional and personal development was a constructive point of criticism that was consistent [through] several of my peers' comments. One rater stated that I do not contribute my opinion in group discussions as boldly as my classmates, but that the contributions that have been made by me were very reflective and interesting to him.... Since reading these comments (and initially shocked because I had thought that I was making a decent amount of contributions and felt that I was even leading the discussions in a few occasions), I have made deliberate moves to shed more of my self-consciousness and take a far more proactive role in not only leading group discussions in medical school, but in the work setting of a hospital-team. I have consciously made an effort to take on more leadership roles in many aspects of my life. Regardless of whether I get along with my classmates on a personal level, I deeply respect the opinions of most of them. This is probably one large reason why I was able to respond to their constructive criticisms so immediately and strongly.
Negative effects of peer assessment.
Some transformations were not desirable, such as negative self-image changes or negative attitudes toward classmates. Of the 12 (9%; 4 second-year, 8 fourth-year) students who reported undesirable transformations, 6 reported negative feelings toward classmates resulting from the peer assessment. However, 5 of these 12 students also reported increased awareness—a positive transformation.
I received a comment that stated that I often jumped too quickly to answer questions and didn't leave other students enough time to process their thoughts and come up with an answer. In response I have been more cognizant of waiting (however long and tedious it may seem) for others to speak up. The comment increased my consciousness of the issue, but I also feel that it makes me more hesitant to speak up in large groups, and now I am more reserved with my interactions.
Of the 49 students (25 second-year, 24 fourth-year) who reported at least one negative cognitive reaction (e.g., feedback was inaccurate, hurtful, or not helpful) or negative emotional reaction (e.g., feeling annoyed, upset, disappointed, or angry), 31 (18 second-year, 13 fourth-year) also reported positive reactions, and 31 (17 second-year, 14 fourth-year) identified a new positive personal transformation.
Two of my third-year comments noted that I fell asleep often during third year. These comments really hurt because I had a very tough third year emotionally and was often just sleepless and drained. But, they did motivate me to try to keep my mind occupied during lectures and grand rounds (even if with crosswords) so I would stay awake. The comment still sticks in my mind and stings.
Twenty-one of the narratives suggested emotional reactions that were not described or acknowledged by the student. Interpretive codes were used in these instances and included “anger,” “defensiveness,” and “narrative suggests reaction different than student describes.” There were a few respondents who felt peer assessment had no value for them; however, some of these also revealed unacknowledged qualities and emotions of the writer.
The peer assessment did not affect me at all. I thought it was pointless to have people tell me what they think of me. In the long run, it won't have an effect of a “life-altering” experience because it is difficult to point out what are some weaknesses in people and then expect them to change those weaknesses. I really forgot what people wrote about me, and really, I don't care... because worrying about what people think of me is the last thing on my mind right now. It is pointless to assess peers in medical school when what really matters is the assessment made by faculty.
Students who reported no transformation.
Nineteen students (14%) stated that they had experienced no personal transformation, and another 31 students (22%) made no mention of transformations. Of these 50 students, only 21 (42%) described specific content of their peer feedback (contrasted with 84% of students who did report transformations). Students' narratives identified a lack of specific peer feedback and lack of new information among reasons for not making changes.
Many of the comments I received were generic, and those were not helpful. Comments that were specific were more helpful. Although I did not receive any life-altering comments from my peers, I think they did verify that I was moving in the right direction.
The comments I received were common criticisms I have already received, so I was not surprised or consider this process as having a significant impact.
Comments about the peer assessment program itself.
Some narratives provided useful insight into the relationship of peer assessment to the informal curriculum and the culture of medical training. For example, one student reported that her peer assessment revealed sex and racial stereotypes of her classmates.
A few of the comments resonated prejudices that are too familiar (and tired!) for minorities and women. More specifically, the issue of being “vocal” or “aggressive” or “intimidating,” “emotional,” rather than the more social acceptable characteristics of being outspoken, assertive, confident, (com)passionate, and culturally sensitive, etc. One small subtle example: It was stated that I appear disrespectful and aggressive when raising my hand and “yelling question” at the same time to get the attention of the prof. in lecture... Surprised that my actions were considered “disrespectful,” I decided to change behavior immediately. However, since the PA (now seven weeks later) I have counted 14 incidents where other colleagues did the same thing, and although I did not find it rude at all, I just became more aware of it. However, I also have experienced six incidents where I had my hand up and was never called on and recused my question until the end of the class (sometimes the prof. was rushing out the door).
Twenty students mentioned the issue of whether narrative comments should be signed or anonymous. All of these students felt that a signature makes the comment more helpful, and some felt that their signed comments were of higher quality. A few specifically said that anonymity should remain an option.
I received another comment stating that the person saw how stressed I was and how he felt I would be better not being in that kind of field. But what was different was that he used his name and so I was able to approach him and actually talk to him and thank him for his openness.
I was told by my former roommate, encoated in loving praise, that maybe I could study a little more. I took that to heart, since I know that of anyone he knows me well. I also already guiltily knew it myself, so that independent confirmation pushed me to make a lot of changes in my work ethic.
I want to say, however, that the most constructive feedback was from people who identified themselves in the evaluation. It seemed that these people, possibly because their names were attached to their statements, did a much better job of carefully articulating their feedback so that it was truly constructive (both positive and negative). Certainly, these people who signed their names were not restricted in their negative criticisms. In fact, some of the most forthcoming and telling criticisms I received were from people who signed their names. Also, I respected these evaluations so much more because I figured that if someone was willing to sign his/her name to the critique, it must actually be important to this person that I find a way to improve... almost like these individuals were personally investing in my benefit.
Finally, one student noted that the work of giving feedback to peers was itself meaningful:
Another aspect of peer assessment that did affect me... involved giving critique to a peer. For the one or two peers I had strong critique for I felt obligated to be as constructive and as professional as possible.
To our knowledge, this is the first published report describing the impact and transformative potential of formative peer assessment for medical students. Students' responses suggest that most took peer assessment seriously, and it had a meaningful impact for many. Sixty-seven percent of the respondents found peer assessment helpful, comforting, or confirming of something they already knew, and 65% of the respondents reported personal transformations as a result of peer feedback, most of which were positive. Many students reporting negative experiences also mentioned positive outcomes, such as increased awareness or insight. Recalled peer feedback often related to student interactions that can go unnoticed or unaddressed by faculty or advisors; accordingly, the aspects of professionalism that are most difficult for faculty to assess were the elements that had the greatest impact and transformative potential through peer assessment.
I would have to say that the peer assessment most helped me understand how others interpret (or misinterpret) some things I may say or do. One classmate thought it was inappropriate for me to make negative remarks about classmates under my breath during lecture. Such remarks WOULD be inappropriate, but I didn't make them. I may make offhand comments to classmates during a lecture, but would never direct them toward a classmate. Having read that person's comments, I can see how my behavior may have appeared negative and am more careful about when I say things and how I say them.
Whether their experience with peer assessment was positive or negative, the students expressed agreement on several points. Feedback was more useful when it was specific, when themes were repeated by more than one rater, when it came from a credible source (signature may help), and when there was an opportunity to discuss the feedback with an advisor or friend. Information that was novel or surprising was deemed helpful, as was (for some) mention of qualities students already recognized about themselves. Very few students found peer assessment to be of no value. We find this response to peer assessment encouraging, and it compares favorably with student ratings of other curricular elements and assessments at our institution.
Student surveys have suggested that peer assessment programs can be acceptable if implemented in a safe environment.9,11,27 Clearly, the underlying institutional culture affects the impact of peer assessment. In our setting, as experience and the “word on the street” confirmed that peer assessment was worthwhile, the level of anxiety and resistance to peer assessment decreased. Other elements that may contribute to a sense of support and safety in our setting include the advisory dean system and the preparatory workshops on giving feedback. Like any subjective or interpersonal evaluation, peer assessment is subject to real or perceived bias; advising is essential in helping students think about their data. Peer assessment in our setting focuses on aspects of professionalism and is formative, and results are provided only to the students. Embedding peer assessment within a larger formative assessment, although not essential, may help to promote reflection by providing context with other concurrent performance data. Probably the most powerful component of a culture that supports effective peer feedback is the informal reporting by students who found peer feedback helpful.
Although students always have the option of anonymity, the percentage of those choosing to sign their comments increased from nearly 0% in the first year to over 80% of the comments. From the beginning, students consistently reported that signed comments are more helpful; in response, we encouraged subsequent groups to sign their comments, and we created a simple prompt so that they could easily do so with a mouse click. We encourage signed feedback because it provides context and accountability for comments and allows for clarification and discussion; all of these can facilitate improvement. Although some students write about professional lapses, we specifically do not promote the reporting of professional lapses as the primary purpose of peer assessment. We believe this would hinder the program's purpose of fostering professional formation, reflection, and growth.
Notably infrequent in the students' recollections of peer feedback were observations of performance in clinical settings. Because of the structure of clinical training, students have limited opportunity to observe peers directly when they are working with patients. However we believe there are important observations made by students in other shared work settings in which colleagues form impressions of others' professional behaviors and integrity, such as ward rounds, checking labs, discussing patients informally, interacting with nurses and other health professionals, and during informal social interactions in work settings. A future challenge, at all levels of training, is to find ways for peers and others to observe each other in action during direct patient contact in order to provide high-quality feedback about actual practice.
Although the effects of peer feedback were similar for second- and fourth-year students, a greater percentage of second-year students recalled specific elements of feedback, found confirmatory and helpful elements, and reported peer feedback as transformative. Several observations may help explain this finding. First, second-year students had just received their feedback six to eight weeks previously, whereas fourth-year students' peer assessments had been given one and two years previously. Second, it would not be surprising that, after an initial peer assessment, subsequent iterations would be less likely to present new insights about personality traits and habits and would be experienced as less emotionally intense. Third, when the fourth-year class matriculated at the medical school, peer assessment was still a relatively new program, whereas second-year students matriculated at a time when all current students had done peer assessment. Finally, students noted that, during the third year, when they were having more intense patient contact, their contact with peers was less intense compared with the first two years of medical school. These observations may invite educational planners to find ways to use peer-to-peer interactions more effectively during the clinical rotations.
This study represents experience in one institution. Most medical schools do not require peer assessments, and those that do may use very different formats and processes. Institutional culture varies significantly among institutions, and even the decision to require formative peer assessment represents a culture shift. However, this study may provide helpful information about potential impact and issues to guide others considering implementation of peer assessment.
We chose to elicit narratives rather than answers to closed-ended survey questions to gain a richer understanding of students' reactions and responses. Further, we were particularly interested in individual students' experiences; although focus groups may have fostered further development of ideas through discussion, we felt they might blur individual students' reflections.
The responses collected do not represent all students from the two classes. Although this may be considered a limitation, our purpose in this initial qualitative study was to gather information about the potential range and scope of peer assessment's impact over time. Although the use of self-reported, retrospective data could misrepresent students' actual peer comments, or even their actual reactions, thoughts, or responses to peer assessment that occurred at an earlier point in time, our interest was in lasting impact. Future studies might adopt a longitudinal approach in which students' reactions could be followed prospectively, beginning with impressions about peer assessment before they have participated in the process.
Students find peer assessment a helpful and transformative source of feedback about professionalism. Just like any medical intervention that has potential side effects, peer assessment is a tool that must be used judiciously and appropriately, in environments that stress safety around reflection and feedback and that facilitate meaningful transformations as a result. Advisors and mentors should be prepared to help students use peer feedback constructively. Future development of peer assessment programs should address how peers might assess each others' clinical work with patients more effectively.
The authors wish to thank Dr. Stephen J. Lurie, University of Rochester SMD, Office of Curriculum and Assessment, for his thoughtful review of the manuscript. The authors also wish to gratefully acknowledge all of the medical students who elected to participate in this study.
This project was supported by a Dean's Teaching Fellowship (A.C.N.) at the University of Rochester School of Medicine and Dentistry during academic years 2005–2007.
The University of Rochester Medical Center's institutional review board granted “exempt” approval for the study.
Preliminary results from some of these analyses were presented as oral abstracts at the Ottawa Conference on Clinical Competence (May 2006; New York, New York), at the Northeastern Group on Educational Affairs (March 2006; Philadelphia, Pennsylvania), and in a poster session at the Association of Medical Education in Europe Annual Meeting (September 2006; Genoa, Italy).