Direct Observation of Students during Clerkship Rotations: A Multiyear Descriptive Study

Abstract

Purpose 

To determine how often students report that they are observed while performing physical examinations and taking histories during clerkship rotations.

Method 

From 1999–2001, 397 students at the University of Virginia School of Medicine were asked at the end of their third year to report the number of times they had been observed by a resident or faculty member while taking histories and performing physical examinations on six rotations.

Results 

Three hundred and forty-five students (87%) returned the survey instrument; of these, 322 (81%) returned instruments with complete information. On average, the majority reported that they had never been observed by a faculty member while taking a history (51%), performing a focused physical examination (54%), or a complete physical examination (81%). The majority (60%) reported that they had never been observed by a resident while performing a complete physical examination. Faculty observations occurred most frequently during the four-week family medicine rotation and least frequently during the 12-week surgery rotation. The length of the clerkship rotation was inversely related to the number of reported observations, χ² (5, n = 295) = 127.85, p < .000.

Conclusions 

Although alternative assessments of clinical skills are becoming more common in medical education, faculty ratings based on direct observation are still prominent. The data in this study reflect that these observations may actually be occurring quite infrequently, if at all. Decreasing the evaluative weight of faculty and resident ratings during the clerkship rotation may be necessary. Otherwise, efforts should be made to increase the validity of these ratings.

Direct observation of students by faculty and residents during clerkship rotations is vital to the instruction and valid assessment of clinical skills. Studies have shown that direct observation provides an authentic patient-centered teaching environment and improves history-taking and physical examination skills.^1–3 In addition, these bedside observations are inherent in many performance ratings of students during clerkship rotations and, therefore, play an important role in the evaluation of clinical skills. Medical educators consider direct observation to be an important method for ensuring clinical competency. This is evidenced by the Liaison Committee on Medical Education's requirement that faculty provide “… ongoing assessment that assures students have acquired and can demonstrate on direct observation the core clinical skills, behaviors, and attitudes that have been specified in the school's educational objectives.”⁴

Virtually all medical schools report the use of performance ratings based on direct observations of students for the assessment of clinical skills.⁵ Despite their wide use, the validity and reliability of these ratings have been challenged.^6,7 One problem with performance ratings may relate to the low incidence of direct observations by faculty. A recent survey by the National Board of Medical Examiners (NBME)⁸ reported that students were observed more often by a resident than by a faculty member and approximately 20% of students indicated that they had been observed by faculty zero to two times while performing a history or physical examination. Another survey of graduates by the Association of American Medical Colleges (AAMC)⁹ reported even lower rates of observation. Specifically, they found that 27% of students reported that they had never been evaluated by a faculty member while taking a complete history and performing a complete physical examination. This survey also found that these reports varied greatly, depending on the graduate's medical school, and ranged from zero to 77%. Twenty-one schools had reports of “no observation” by 10% or fewer students and 20 schools had reports by 40% or more students.

The NBME survey did not clarify how many students had never been observed or when during the clinical curriculum the observations were occurring. The AAMC survey focused on whether students had ever had their clinical skills evaluated by faculty observation; it also did not delineate the number of observations or when they occurred. Additionally, students were not asked to report whether they had been observed by a resident. Neither survey differentiated between complete (head-to-toe) and focused (one-system) physical examinations. We undertook our descriptive study to investigate student estimates of the number of direct observations that occurred during clerkship rotations, type of skill observed, when these observations occurred, and who conducted them.

Method

To determine how often third-year students at the University of Virginia School of Medicine estimated they were observed during the clerkship rotations, from 1999–2001 we administered a survey instrument to 397 students at the end of their sixth rotation. Because of the difficulty of gaining access to all students during their third year, the survey instrument was administered at the end of the academic year when the entire class was required to attend a comprehensive performance assessment. Before conducting our study, we sought approval from the university's Institutional Review Board.

We asked students to estimate, for each of the six rotations, the number of times they had been observed by a resident or faculty member while taking a history, conducting a focused physical examination, and performing a complete physical examination. Each of these three clinical skills were presented in separate grids that included the following possible responses: “0,” “1–3,” “4–6,” “7–9,” “10–12,” and “13+” times observed. The clerkship rotations varied in length: internal medicine and surgery lasted 12 weeks, pediatrics eight weeks, obstetrics–gynecology (Ob/Gyn) and psychiatry six weeks each; and family medicine four weeks. Survey completion was optional and anonymous.

Results

Three hundred and forty-five students (87%) returned the survey instrument; of these, 322 (81%) returned instruments with complete information. The percentages of students reporting that they had been observed by a resident and faculty member 0, 1–3, 4–6, 7–9, 10–12, or 13+ times while performing each of the three clinical skills for combined rotations are presented in Table 1. On average, the majority of students reported having never been observed by a faculty member while they interviewed a patient (51%) or conducted a focused (54%) or a complete (81%) physical examination. Students reported that they had been observed more frequently by a resident while performing each of the three skills; however, on average 60% of students reported that they had never been observed by a resident while they conducted a complete physical examination.

Table 1: Percentages ± 95% Confidence Intervals of 322 Third-Year Medical Students in Combined Rotations Who Reported That They Had Been Observed by a Resident or Faculty Member While Performing Three Clinical Skills, University of Virginia School of Medicine, 1999–2001

The percentages of students reporting that they had never been observed by a faculty member for each rotation and each clinical skill are shown in Table 2. Although history-taking was the skill most often observed by a faculty member, overall the majority of students reported that they had never been observed taking a patient's history during the internal medicine (59%), surgery (74%), and ob–gyn (68%) rotations. Despite the shorter rotation length, psychiatry (27%) and family medicine (26%) clerkships had relatively fewer students reporting that they had never been observed by a faculty member while taking histories. The majority of students reported that they had never been observed by a faculty member while performing a focused physical examination on the internal medicine (51%), surgery (71%), psychiatry (71%), and ob–gyn (61%) rotations. Family medicine (25%) and pediatrics (47%) had relatively fewer students reporting that they had never been observed by a faculty member while performing a focused physical examination. The data reflect that faculty observed students conducting a complete physical examination fewer times than they observed the other two skills. This finding was consistent across all rotations and ranged from 70% on the family medicine rotation to 91% on the surgery rotation.

Table 2: Percentages of 322 Third-Year Medical Students in Six Rotations Who Reported Never Having Been Observed by a Faculty Member While Performing Three Clinical Skills, University of Virginia School of Medicine, 1999–2001

Figure 1 displays the overall percentage of students who reported having been observed, 1–6, 7–12, and 13+ times by a faculty member for each rotation and each skill. The skill and rotation that had the largest percentage of students who reported having been observed “13+ times” was history-taking during psychiatry (31%). With the exception of the family medicine rotation, fewer than 5% of students reported having been observed “13+ times” during their rotations for each skill component. Overall, students reported having been observed by a faculty member least frequently during the 12-week surgery and six-week Ob/Gyn rotations. We calculated a chi-square goodness-of-fit test comparing the frequency of reported observation with the length of the rotation. A highly significant deviation from the expected values was found, χ² (5, n = 295) = 127.85, p < .000. In other words, the number of reported observations did not increase as the students spent more days on the rotation.

Figure 1.: Overall percentages of 322 third-year medical students in internal medicine, surgery, pediatrics, psychiatry, family medicine, and obstetrics–gynecology rotations who reported having been observed 1–6 times, 7–12 times, or 13 or more times by a faculty member while taking a history and performing a focused physical examination or a complete physical examination. University of Virginia School of Medicine, 1999–2001.

Discussion

The descriptive data in our study reinforce and further delineate the findings from previous studies. Previous research did not differentiate what skills were being observed, when the observation was occurring, or who was conducting the observation. According to our findings, on average 81% of students reported never having been observed by a faculty member while performing a complete physical examination during a required clinical rotation. Although observations by residents appear to have occurred more frequently, on average 60% of students reported never having been observed by a resident while performing a complete physical examination. With the exception of family medicine and psychiatry, we found no great disparities across clerkship rotations. Students estimated that they were observed taking a patient's history more frequently on the psychiatry rotation than they were on any other. This finding is consistent with the expectations of the psychiatry clerkship. Specifically, during the course of the study, psychiatry was the only rotation during the third year that required faculty to conduct formal and structured observations of their students’ interviewing skills.

The length of the rotation did not relate to the number of observations. In other words, the longer rotations (12 weeks) were not associated with a larger number of estimated observations. Surprisingly, the surgery rotation, one of the longest clerkships at 12 weeks, received the largest percentage of students reporting that no observations had occurred. Conversely, the family medicine rotation was the shortest (four weeks) and received the smallest percentage of students who reported that they had not been observed by faculty or residents. We do not understand the reasons for this discrepancy. However, we speculate that the family medicine faculty, who predominantly practice in an outpatient setting, may have more time to observe their students than do those faculty who practice in inpatient settings. Additional research is needed to further investigate this disparity.

Our study had several limitations. First, students completed the survey instrument at the end of the sixth rotation. They may have had difficulty recalling the number of times they were observed during rotations they completed at the beginning of the third year, approximately 12 months earlier. However, the rotation schedule is assigned equally and randomly among all students. This inherent randomization of the clerkship rotation schedule should greatly reduce any possible recency effect. Second, our study did not include data regarding the required neurology clerkship, which is scheduled throughout the fourth year of the curriculum and would have occurred after we administered the survey instrument. We felt that waiting until the end of the fourth year to include the neurology rotation might make it even more difficult for the students to recall the observations. Finally, response bias is always a consideration when conducting a survey. Although we had a high response rate (87%), the anonymous data did not allow us to analyze the 13% of students who chose not to respond, nor did it allow us to follow-up on the approximately 6% of respondents who returned incomplete survey instruments.

Although the benefits of direct observation and bedside teaching of medical students have been long understood, these encounters have become increasingly infeasible. As early as 1964, Reischman and colleagues³ observed that bedside teaching was declining on hospital rounds. Soon after, Morgan² postulated four reasons for the apparent change: faculty subspecialization lending to insecurities with teaching generalist skills, third-party funding requirements, increasingly complex patient challenges, and technological advances. Unfortunately, these barriers are even more pronounced today. The current acute and managed care environment has left faculty, preceptors, and house-staff with higher productivity standards, increasing administrative demands, increasing expectations for direct patient contact and documentation by faculty, and subsequently, less time for direct observation of students.¹⁰

Several authors have suggested methods to increase the time faculty spend directly observing students, but none have empirically documented the outcomes of these methods.^11,12 One study found that “structured clinical observations” that were limited to five minutes, focused on components of the clinical process, and included a limited number of feedback points, were qualitatively effective for teaching clinical skills.¹¹ Since the completion of our study, a similar method, referred to as the “Clinical Skills Passport Program,” has been initiated at the University of Virginia for each of the required clinical clerkships. One of the goals of this new program is to encourage direct observation of specific clinical skills by faculty and residents. It is hoped that some of the problems identified by our study will be obviated by this initiative. Future research is planned to investigate the educational impact of this program, namely whether direct observations by faculty and residents have increased.

Given the current and related reports that students are rarely observed during the clinical rotations, the increasing demands on faculty and residents, and the changing face of medical education, consideration should be given to alternate forms of assessment to replace or augment ratings based on direct observation. Alternatively, efforts should be made to increase the occurrence and validity of direct observations. These efforts to produce more valid assessments and better instruction will ultimately better equip medical students with the clinical skills required for residency training.