Pohl, Charles A. MD; Hojat, Mohammadreza PhD; Arnold, Louise PhD
Peer assessment is a promising method for addressing the need in medical education and medicine of ensuring competency and improvement of physicians in training and in practice.1 Its use, introduced into the military during World War II and into medical education in the 1950s, rests on the assumption and empirical finding that peers can provide unique information about each other's performance that is different from performance information available to supervisors or faculty.1–4 A 1998 review of peer assessment in medical education concluded that peer assessment can complement other assessments in areas that are hard to measure, such as personal qualities related to humanism and professionalism.5 A 2006 review of peer assessment found evidence that peers do have unique information about each other's behaviors relevant to professionalism.1
Peers can assess each other in several ways: ratings, rankings, voting, comments, and nominations.1 Peer nomination is the naming of a specified number of group members as the best for having certain qualities. Although it is used in medical education less frequently than peer ratings, it was among the first peer assessment methods to appear in the field.1
The value of peer nomination as an instrument for assessing personal qualities is known largely from work in social and developmental psychology. Research in these disciplines indicates that individuals are indeed able to identify an array of personal qualities of their peers. For example, in a meta-analytic study of children, Newcomb et al6 reported that recipients of positive peer nominations exhibited greater cognitive and social competencies. Peer nomination also detected learning difficulties among young children.7,8 In addition, peer nominations identified positive affect and engagement in conversation among preschool children.9 Moreover, Ollendick and colleagues10 found that patterns of peer nomination are reliable and tend to be relatively stable over time among school-age children.
Although infrequent, the research on peer nomination among medical students suggests its effectiveness as an assessment method due to moderate associations with indicators of clinical competence2,11 and to a multifactorial structure related to technical competence, on the one hand, and, on the other, to interpersonal skills, caring, and—in one instance—service.11,12 Because these factors affect medical professionalism, Arnold and Stern1 recommended using peer nomination in medical education to identify physicians-in-training with outstanding qualities. Peer nomination is now frequently used to select students for membership in a humanism honor society.12 Nevertheless, research on the psychometric properties of peer nominations is less mature than for peer ratings; additional data are still needed to demonstrate the validity of peer nomination.1
On the basis of literature that links peer nomination to academic attainment,2,11 personality, and social competencies,6,9,12 we designed this study to test the hypotheses that medical students who have been nominated by classmates for qualities pertaining to professionalism in medicine would (1) obtain higher ratings on their clinical performances, (2) obtain higher empathy scores, (3) display a more positive personality profile, and (4) express more interest in people-oriented (as opposed to procedure-oriented) specialties. Testing these hypotheses can also serve as a much-needed approach to study the validity of peer nomination.
All 255 third-year students at Jefferson Medical College in the 2007–2008 academic year received an invitation to participate in this study. There were 130 (51%) men and 125 (49%) women in the class.
A variety of tools are available to elicit peer nominations. In the context of medical education, McCormack and colleagues12 developed a six-item peer nomination instrument to identify medical students with exemplary clinical and humanistic characteristics. List 1 describes the six items.
The 6 items came from a 12-item peer nomination form originally developed at Case Western Reserve University School of Medicine13 and used at the University of Florida for more than a quarter of a century.12 Factor analytic results of a study in three medical schools showed that the prominent underlying factor of the six-item peer nomination tool was clinical competence.12
Measures of academic attainment.
We used three measures of academic attainment. The first—grades on objective examinations for six core third-year clerkships (family medicine, internal medicine, obstetrics–gynecology, pediatrics, psychiatry, and general surgery)—typically have reliability coefficients of 0.70 or higher. Scores on Steps 1 and 2 of the United States Medical Licensing Examinations were the second measure, and the third—ratings of clinical competence given by faculty in the six core clerkships (original ratings of “high honors,” “above expected competence,” “expected competence,” “and marginal competence” were converted onto a 100-point scale)—have reasonable predictive validity.14
Measure of empathy.
The Jefferson Scale of Empathy comprises 20 items, each answered on a seven-point Likert scale; possible scores range from 20 to 140. According to prior work, this scale yields valid and reliable scores of empathy among medical students, physicians, and other health care professionals.15–18 A recent study found a significant link between physicians' scores on the scale and positive clinical outcomes in diabetic patients.19
The Zuckerman–Kuhlman Personality Questionnaire–Short Form20 measures five personality factors:
1. Impulsive Sensation-Seeking is a tendency to act quickly on impulse without planning, often in response to a need for thrills and excitement, change, and novelty.
2. Neuroticism–Anxiety is a tendency to be tense and worried, overly sensitive to criticism, easily upset, and obsessively indecisive.
3. Aggression–Hostility is a tendency to express verbal aggression and show rudeness, thoughtlessness, vengefulness, spitefulness, a quick temper, and impatient behavior.
4. Sociability is a tendency to interact with others, enjoy being with others, and not tolerate social isolation.
5. Activity is a tendency to be active, prefer challenging work, and be impatient or restless when there is nothing to do.
During a class meeting held after six months of clinical training, third-year medical students were asked to complete the peer nomination instrument. They were told that their participation was part of a peer assessment exercise to help the medical college award a humanism prize, and they were asked to name three classmates, for each of the six items, who best fit the situations described (List 1). The instructions explained that the students' participation was voluntary, asked that they refrain from nominating themselves or signing the survey, and assured them that their nominations would be treated confidentially. They received no incentive for participating. Completion of the instrument was tantamount to informed consent.
The Jefferson Longitudinal Study's database was the data source for students' examination grades, ratings of clinical competence, scores on medical licensing examinations, and empathy and personality scores. More information about the Jefferson Longitudinal Study can be found in a recent publication21 and at http://jdc.jefferson.edu/jlsme.
The total number of times a student was nominated by his or her classmates constituted that student's peer nomination score. The t test and chi-square were the statistical techniques used to compare the academic attainment, empathy, personality factors, and specialty preferences of nominated students and students without any peer nominations. The number of observations in different analyses varied because of missing data.
The university's institutional review board exempted the study.
A total of 166 (65%) students returned peer nomination forms. Of the 255 students in the class, 155 (61%) received at least one peer nomination, leaving 100 (39%) with no nominations at all. After merging the data with the Jefferson Longitudinal Study database, the effective sample for the final statistical analysis included 226 students (150 in the nominated group and 76 in the nonnominated group). The number of nominations per student ranged from 1 (received by 50 students) to 22 (received by 1 student); the median was 3 nominations.
The means and standard deviations of the third-year examination grades, ratings of clinical competence in the third year, and scores on medical licensing examinations for the two groups of students who were or were not nominated appear in Table 1.
As the table shows, the only significant difference between the two groups' academic attainment was the rating of clinical competence. Students who received peer nominations had higher ratings (t(215) = 4.7, P < .01). The effect size of the difference (d = 0.71) indicates that this difference is of practical importance.22 No significant difference on objective examinations emerged between the two groups. These findings confirmed that peer nomination was associated with faculty's ratings of clinical competence, but not with performance on objective examinations.
Empathy and personality factors
Table 2 presents the results of comparisons of empathy and personality scores of the two groups. Students with peer nominations scored significantly higher on the empathy scale (t(218) = 2.2, P < .05) and on the “activity” personality factor (t(218) = 2.44, P < .01) than did students with no nominations. The nominated students were also more sociable, although the difference between the groups was not significant at the conventional level of <0.05 (t(218) = 1.64, P < .10). The effect sizes ranged from 0.25 (for sociability) to 0.32 (for empathy). These findings supported the second research hypothesis concerning the associations between peer nomination, on the one hand, and empathy and selected personality factors on the other. Although the associations were statistically significant, the effect size estimates were not large.
Table 3 shows the comparison of the specialty preferences of students who received peer nominations with those who did not. The classification of students' specialty preferences into two categories rendered the comparison manageable; 145 students preferred people-oriented specialties (e.g., emergency medicine, family medicine, internal medicine, obstetrics–gynecology, pediatrics, psychiatry, medical specialties), whereas 70 indicated a preference for technology/procedure-oriented fields (e.g., pathology, radiology, anesthesiology, surgery and surgical specialties). This comparison did not include 11 students whose preferences could not be classified into these categories.
As Table 3 indicates, students who received peer nominations were more likely to prefer people-oriented specialties than were students who had received no nominations (74% versus 54%). Concomitantly, fewer nominated students than nonnominated students expressed an interest in technology/procedure-oriented specialties (26% versus 46%). The association was statistically significant (χ2(1) = 8.8, P < .01). These findings confirmed the third research hypothesis concerning the association between peer nomination and specialty interest, which in turn lends credence to peer nomination as a way to identify students with an interest in people-oriented specialties.
Discussion and Conclusion
Findings of this study support the hypotheses that favorable nomination by medical student peers is related to higher ratings of clinical competence, greater orientation toward empathic engagement in patient care, greater tendency to be active and enjoy being with others, and a preference to pursue people-oriented as opposed to technology/procedure-oriented specialties.
This supporting evidence has several possible explanations. First, peers are likely to nominate individuals who help in emergency situations, have superior caring skills, are interested in community services, and enjoy professional success—qualities that are all relevant to clinical competence and professionalism. Second, the students' criteria for choosing nominees may have been shaped by their awareness that their nominations would help identify students to receive a humanism award and by the content of the nomination instrument, which provides a comprehensive notion of a humanistic physician—one who has exemplary clinical skills and who cares about and relates to patients as people. Finally, students observe their peers in a variety of situations, often alone with each other and away from faculty evaluators, where they can let down their guard. Peers sharing a call room, for example, hear each other's authentic reactions to, say, a 2:00 am page to assist in admitting a patient in distress.
Our finding that peer nomination is significantly associated with clinical performance ratings underscores the relationship already found in the literature.11 In contrast, peer ratings have not been uniformly found to correlate with faculty ratings.1 The inconsistency may reflect the different assessment forms used by faculty across the research studies or the varying emphasis that faculty accord professionalism in assessing students' clinical competence. At any rate, the inconsistency in the relationship between peer and faculty assessments prompted a key question in a recent review as to whether faculty ratings should be the “gold standard” against which to judge the validity of peer assessment.1
From a psychometrics perspective, our findings provide important new data about the criterion-related validity of peer nominations. After all, empathy, personality factors, and specialty preferences stand apart from other performance measures, such as those derived from self- or faculty ratings that have been previously used in validation studies of peer assessment.1
On the basis of these results, we encourage medical schools to add peer nomination to their assessment tool kit. It is, we acknowledge, just one tool in that kit. Peer nomination identifies individuals at the extremes—in this instance, exemplars. When the purpose is to assess and provide feedback to all medical students in a class, peer nomination may not be as productive; at the very least, additional sources of assessment would be necessary. Further, because peer nomination does not yield information about all students, it is not useful as a summative assessment. In fact, from the students' perspective, peer assessment—regardless of the type—is best for formative use23 and could result in inflated judgments of performance in high-stakes evaluations.24
Limitations of this study include a nonrandomized sampling from a single institution, which may preclude generalization of the findings. In addition, because participation in the study was anonymous, we could not determine whether the respondents were representative of their class on variables that might alter the study results, such as gender16 (although the 65% response rate should yield accurate results, considering the class size25). Replication of this study in other medical schools could offer additional support for the external validity or generalization of the findings. Finally, a concern may arise that peer nominations suffer from inappropriate bias because of their relationship to personal qualities such as sociability or likeability that may seem irrelevant. However, previous work has found peer nominations relatively free of bias stemming from likeability, friendship, and acquaintanceship2 and peer judgment capable of fairly detecting clinical skills and humanistic qualities.1,26 Still other research has noted that medical students believe they are suited to judging peers' professionalism1 and are willing to do so in a supportive learning environment.23 Moreover, because sociability, the desire to be active, people-orientation, and empathy underlie the repertoire of physicians' behaviors in caring for patients and families from all backgrounds, assessment methods such as peer nomination that take these personal qualities into account are right on the mark.
In short, depending on its use, peer nomination can supplement and enrich faculty observations and judgments in evaluating clinical competence, empathy, and people-orientation—three important dimensions of medical students' professionalism.
The authors wish to thank Susanne Friedman and Joyce Muwwakkil for helping us in data collection and management, and Dorissa Bolinski for helping with the editorial polishing of the manuscript.
This study was exempted by the Thomas Jefferson University internal review board.
1Arnold L, Stern D. Content and context of peer assessment. In: David DT, ed. Measuring Medical Professionalism. Oxford, UK: Oxford University Press; 2006:175–194.
2Kubany A. Use of sociometric peer nomination in medical education research. J Appl Psychol. 1957;41:389–394.
3Korman M, Stublefield RL. Medical schools evaluation and internship performance. J Med Educ. 1971;46:670–673.
4Kane JS, Lawler EE. Methods of peer assessment. Psychol Bull. 1978;85:555–586.
5Holmboe ES, Hawkins RE. Methods for evaluating the clinical competence of residents in internal medicine: A review. Ann Intern Med. 1998;129:42–48.
6Newcomb AF, Bukowski WM, Pattee L. Children's peer relations: A meta-analytic review of popular, rejected, neglected, controversial, and average sociometric status. Psychol Bull. 1993;113:99–128.
7Frederickson NL, Furnham AF. The long-term stability of sociometric status classification: A longitudinal study of included pupils who have moderate learning difficulties and their mainstream peers. J Child Psychol Psychiatry. 2001;42:581–592.
8Nabuzoka D, Smith PK. Sociometric status and social behaviour of children with and without learning difficulties. J Child Psychol Psychiatry. 1993;34:1435–1448.
9Walker S. Sociometric stability and the behavioral correlates of peer acceptance in early childhood. J Genet Psychol. 2009;170:339–358.
10Ollendick TH, Weist MD, Borden MC, Greene RW. Sociometric status and academic, behavioral, and psychological adjustment: A five-year longitudinal study. J Consult Clin Psychol. 1992;60:80–87.
13Brozgal JL. Evaluation of the Clinical Performance of Medical Students in an Experimental Program [dissertation]. Cleveland, Ohio: Case Western Reserve University; 1957.
15Hojat M, Mangione S, Nasca TJ, et al. The Jefferson scale of physician empathy: Development and preliminary psychometric data. Educ Psychol Meas. 2001;61:349–365.
16Hojat M, Gonnella JS, Mangione S, et al. Empathy in medical students as related to clinical competence, gender, and academic performance. Med Educ. 2002;36:522–527.
17Hojat M, Gonnella JS, Nasca TJ, Mangione S, Vergare M, Magee M. Physician empathy: Definition, measurement, and relationship to gender and specialty. Am J Psychiatry. 2002;159:1563–1569.
18Hojat M. Empathy in Patient Care: Antecedents, Development, Measurement, and Outcomes. New York, NY: Springer; 2007.
20Zuckerman M. Zuckerman–Kuhlman Personality Questionnaire (ZKPQ): An alternative five-factor model. In: DeRaad B, Perugini M, eds. Big Five Assessment. Seattle, Wash: Hogrefe & Huber; 2002:377–396.
22Hojat M, Xu G. A visitor's guide to effect sizes: statistical significance versus practical (clinical) importance of research findings. Adv Health Sci Educ Theory Pract. 2004;9:241–249.
24Hay JA. Tutorial reports and ratings. In: Shannon S, Nocterm G, eds. Evaluation Methods: A Resource Handbook. Hamilton, Ontario, Canada: McMaster University; 1995. Reprinted in: Norcini JJ. Peer assessment of competence. Med Educ. 2003;37:539–543.
26Ramsey PG, Wenrich MD, Carline JD, Inui TS, Larson EB, LoGerfo JP. Use of peer ratings to evaluate physician performance. JAMA. 1993;269:1655–1660.