Feedback is a critical component of medical education and residency training.1–4 5–8 9–12 13,14 13
We conducted a national survey of medical students, interns, residents, chief residents, and fellows to learn the extent to which certain critical behaviors were observed, to examine trainees' beliefs about accurate knowledge of each other's performance, and to assess whether trainees gave and received feedback regularly. The study reported here investigated the extent to which direct observation of relevant activities was associated with the perception of having accurate knowledge of a trainee's performance, and whether the perception of accurate knowledge of performance was associated with the provision of feedback.
Methods
A random sample of 67,500 United States Medical Licensing Examination (USMLE) registrants were invited to participate in a national survey during August and September 2008. To reach trainees several years into residency training or in fellowship, this sample included individuals who had registered up to four years before the survey. E-mail invitations sent from the National Board of Medical Examiners explained that the survey was not connected with the USMLE, was for research purposes only, was voluntary, and was confidential. The invitation included a link to an Internet-based questionnaire. Nonrespondents were sent two reminder e-mails. No incentive was offered.
Questionnaire items were developed and refined iteratively, with two rounds of pilot testing. Multiple versions of the questionnaire were created to provide broad coverage on a wide variety of relationships while minimizing response burden. The versions differed in three ways: (1) whether respondents were asked about their experiences as observers and feedback providers (active version), or about their experiences as trainees and feedback recipients (passive version), (2) whether respondents were asked about their experiences with medical students, interns, residents, chief residents, fellows, attendings, or nurses (colleague version), and (3) whether items about experiences with patients were included. Each respondent was randomly assigned to either a passive or active version. The colleague version was assigned at random based on the respondent indicating that he or she had experience working with that type of colleague during the current rotation. For example, if a medical student reported working with medical students and interns, but not residents and attendings, she might be asked about her interactions with medical students or interns (but not both), and she would not be asked about her interactions with residents or attendings. Finally, questions about experiences with patients were omitted for the minority of respondents who reported no direct contact with patients in their current rotation.
This paper focuses on responses to three sets of items assessing direct observation, perceived knowledge, and feedback. The first item set (eight items) assessed direct observation of interactions with patients and family members, and activities related to patient care but not involving direct interactions with patients. The next item set (four items) focused on perceptions of accurate knowledge of colleagues' performance during interactions with patients, other patient-related activities, educational activities, and work responsibility. The final item focused on the delivery of feedback on any aspect of the trainee's performance. Exact item wording and response options are included in the Appendix .
Background items collected information on the respondent's current clerkship or residency training program, level of training, and gender.
Prior to the main analyses, the impact of active versus passive questionnaire version was evaluated using chi-square statistics, to assess whether the versions could be combined.
Cross-tabulations of responses and chi-square analyses were used to evaluate the relationship between direct observation and the perception of accurate knowledge, and between the perception of accurate knowledge and the provision of feedback. These analyses were conducted for the sample overall and for each observer–trainee pair (e.g., for medical students observing residents, for residents observing interns, etc.). For the analyses involving direct observations, responses were summarized to create three levels of direct observation: “0,” “1–2,” and “3 or more” (see Table 1 and Appendix ).
Table 1: Percent Reporting Accurate Knowledge of Performance, by Number of Activities Observed, by Role of Observer, and by Role of Observee
This study was reviewed and approved by Essex Institutional Review Board, Inc.
Results
Of the 60,005 invitations which were presumed delivered (i.e., for whom we did not receive notification of a bad e-mail address), 17,822 recipients (30%) responded. Of these respondents, 5,692 were ineligible for the survey (i.e., not currently in training or not working with other trainees, attendings, or nurses). The final response rate for this survey, calculated using the American Association for Public Opinion Research Standard Definitions, equation RR4, was 22.8%.15
Preliminary analyses found minimal effects of the active/passive phrasing of items. For example, we compared responses of residents who asked about observing interns with responses of interns asked about being observed by residents, and we found these to be similar; we therefore aggregated these into a single observer/observee pair for subsequent analyses.
Table 1 summarizes the relationship between observation and the perception that the observer has accurate knowledge of the observee's performance. The first row of data presents results aggregated across all respondents. Overall, when an observer had not seen the observee perform any of the specified activities involving direct interactions with patients and family members during the preceding seven days, 39% of respondents believed that the observer had accurate knowledge of the trainee's performance in this area; this increased to 55% if one or two of the specified activities had been observed, and it increased further to 72% if three or more of the specified activities had been observed. This same pattern and very similar percentages were obtained for other patient-related activities not involving direct interaction. Overall, when an observer had not observed any of the specified patient-related activities in the preceding seven days, 40% of respondents viewed the observer as having accurate knowledge of the observee's performance in this area; this increased to 53% if one to two of the specified behaviors had been observed and to 71% if three behaviors had been observed. Overall, the relationship between observation and the perception of accurate knowledge was highly statistically significant, with P values of chi-square statistics less than .001 for both direct patient interactions and other patient-related activities.
Subsequent rows of Table 1 present results for different observer/observee dyads (e.g., medical students observing medical students, interns observing medical students, residents observing medical students, etc.). An example may aid interpretation: The second row of data in Table 1 shows results for medical students observing other medical students. Medical students who had not observed other medical students engaged in any of the specific activities involving direct interactions with patients during the preceding seven days were still perceived to have accurate knowledge of these peers by 49% of respondents; this percentage increased to 59% if one or two relevant activities had been observed and to 68% if three or more relevant activities had been observed. The separate analyses of the relationship between direct observation of specific activities and the perception of accurate knowledge for each dyad showed similar patterns in general. The relationship between direct observation and the perception of accurate knowledge was positive and statistically significant for the vast majority of observer/observee pairs for both patient interactions and other patient-related activities. Observers who had not seen the observee engaged in relevant activities during the past seven days were less likely to be viewed as having accurate knowledge of the observee.
The second set of analyses examined the relationship between feedback and the perception of accurate knowledge of the recipient's performance in four critical areas (Table 2 ). Again, the first row of data in this table presents results aggregated across all respondents, and subsequent rows present results for each feedback provider/recipient dyad. Overall, the perception that the feedback provider had more accurate knowledge of the trainee's performance was associated with a greater likelihood of feedback occurring (P < .001). Of those peers, colleagues, or supervisors who did not have accurate knowledge of the recipient in any of the four areas specified (patient interactions, other patient-related activities, educational activities, or work activities), 27% had provided feedback during the prior seven days; this increased to 44% when the provider had accurate knowledge of the recipient in two areas and to 60% when the provider had accurate knowledge of the recipient in three or four areas.
Table 2: Percent Providing Feedback by Accurate Knowledge of Performance, by Role of Feedback Provider, and by Role of Feedback Recipient
Chi-square analyses of the association between accurate knowledge and provision of feedback for each distinct feedback provider/feedback recipient pair (e.g., for medical students providing feedback to medical students, for interns providing feedback to medical students, etc.) suggested positive and statistically significant associations for the vast majority of pairs. Again, an example may aid interpretation of the table. When a medical student (provider) had no accurate knowledge of the medical student (recipient) in any of the four domains specified, the medical student gave feedback 22% of the time. This rose to 44% if the medical student had accurate knowledge in three or four of the domains specified.
Discussion
These findings suggest that the perception of accurate knowledge about a trainee peer, colleague, or supervisor is a function of having directly observed the person in question as he or she performed relevant activities. This is as expected—more direct observation should increase knowledge of an observee's ability and performance. However, it is noteworthy that even when none of the specified activities had been observed recently, a substantial percentage of respondents (approximately 40%) felt that the observer had accurate knowledge of the observee's performance. Further, even when the observer had seen the observee engaged in at least three relevant activities within the past week, over a quarter of respondents did not perceive the observer as knowledgeable with respect to the observee's performance in that area. Clearly, it is not only direct observation that influences perceptions of whether one is knowledgeable about another's performance.
We also found that feedback providers who are perceived as more knowledgeable about an observee are more likely to provide feedback. This is an important positive finding because feedback recipients are more likely to consider feedback credible, and to consider taking action, if the feedback provider is knowledgeable.13,14
The findings reported here, though unsurprising, have important implications for medical educators and residency directors, and for the design and implementation of evaluation and feedback systems in medical education and training, including but not limited to multisource feedback systems. First, although direct observation alone may not be a sufficient condition for credible feedback, direct observation is clearly important and strongly associated with the perception that one has accurate knowledge of another's performance. This suggests that medical educators and administrators may want to provide explicit expectations for direct observations by feedback providers in order to ensure that a sufficient number of observations have occurred and to allow potential respondents to decline to give feedback if they have not had sufficient opportunities for direct observation. Finally, many trainees believe that in addition to their attendings, peers, those they supervise, and nurses have accurate knowledge of their performance, especially if those others have directly observed them performing relevant activities. Medical educators and residency directors could use these others as feedback providers in multisource feedback systems.
The limitations of this study include reliance on self-report and the potential for recall and social desirability bias. However, these results were generally consistent regardless of whether a respondent was asked about observing or being observed and across observer/observee pairs, and they are based on a large number of respondents at various training levels, suggesting that the main findings are robust and valid. Our decision to sample from several years of USMLE registrants, the absence of an incentive, and the fact that potential respondents likely had intensive time pressures all may have negatively affected the response rate. Nonresponse affects generalizability if respondents differ systematically from nonresponders on experiences with direct observation and feedback. Unfortunately, we are not able to assess whether such differences are present. Additional research using direct measurement of trainee contact (e.g., by providing radio-frequency identification chips to students and housestaff, or using cell phones to collect time and location data) should be conducted to confirm and extend these findings. Also worthy of further investigation are studies of the content, quality, or impact of feedback as a function of direct observation and perceived knowledge.
In conclusion, the findings from this study suggest that increased observation is associated with the perception of more accurate knowledge, which in turn is associated with increased feedback delivery; these findings are generally as expected and confirm the value of direct observation. We have also identified a concern—specifically, a substantial minority of evaluators are likely to provide feedback in the absence of observational data, a finding that has been reported previously.16
Funding/Support:
None.
Other disclosures:
None.
Ethical approval:
This study was reviewed and approved by Essex Institutional Review Board, Inc.
References
1Ende J. Feedback in clinical medical education. JAMA. 1983;250:777–781.
2Ende J, Pomerantz A, Erickson F. Preceptors' strategies for correcting residents in an ambulatory care medicine setting: A qualitative analysis. Acad Med. 1995;70:224–229.
3Kilminster S, Jolly B, van der Vleuten CP. A framework for effective training for supervisors. Med Teach. 2002;24:385–389.
4Shute VJ. Focus on formative feedback. Rev Educ Res. 2008;78:153–189.
5Hawkins RE, Katsufrakis PJ, Holtman MC, Clauser BE. Assessment of medical professionalism: Who, what, when, where, how, and… why? Med Teach. 2009;31:348–361.
6Ramsey PG, Curtis JR, Paauw DS, Carline JD, Wenrich MD. History-taking and preventive medicine skills among primary care physicians: An assessment using standardized patients. Am J Med. 1998;104:152–158.
7Wood L, Wall D, Bullock A, Hassell A, Whitehouse A, Campbell I. ‘Team observation’: A six-year study of the development and use of multi-source feedback (360-degree assessment) in obstetrics and gynaecology training in the UK. Med Teach. 2006;28:e177–e184.
8Violato C, Marini A, Toews J, Lockyer J, Fidler H. Feasibility and psychometric properties of using peers, consulting physicians, co-workers, and patients to assess physicians. Acad Med. 1997;72(10 suppl 1):S82–S84.
9Burdick WP, Schoffstall J. Observation of emergency medicine residents at the bedside: How often does it happen? Acad Emerg Med. 1995;2:909–913.
10Holmboe ES. Faculty and the observation of trainees' clinical skills: Problems and opportunities. Acad Med. 2004;79:16–22.
11Kassebaum DG, Eaglen RH. Shortcomings in the evaluation of students' clinical skills and behaviors in medical school. Acad Med. 1999;74:842–849.
12Pulito AR, Donnelly MB, Plymale M, Mentzer RM Jr. What do faculty observe of medical students' clinical performance? Teach Learn Med. 2006;18:99–104.
13Dolmans DH, Wolfhagen IH, Heineman E, Scherpbier AJ. Factors adversely affecting student learning in the clinical learning environment: A student perspective. Educ Health (Abingdon). December 2008;21:32.
14Watling CJ, Kenyon CF, Zibrowski EM, et al. Rules of engagement: Residents' perceptions of the in-training evaluation process. Acad Med. 2008;83(10 suppl):S97–S100.
15Standard Definitions: Final Dispositions of Case Codes and Outcome Rates for Surveys. 5th ed. Lenexa, Kan: American Association for Public Opinion Research; 2008.
16Mazor KM, Canavan C, Farrell M, Margolis MJ, Clauser BE. Collecting validity evidence for an assessment of professionalism: Findings from think-aloud interviews. Acad Med. 2008;83(10 suppl):S9–S12.
Appendix 1: Questionnaire Items and Variable Definitions
