Elnicki, D. Michael MD; Kolarik, Russell MD; Bardella, Inis MD
A growing body of literature indicates that the quality of teaching during clinical clerkships is an important determinant of medical students' learning. Students exposed to good teachers score higher on clerkship examinations,1 National Board of Medical Examiners' Step 2 examinations,2 and perform better overall in clerkships.3 Excellent teachers serve as role models for students and influence their career choices.4
Medical schools are increasingly using nontraditional learning environments and teachers. Most descriptions of teaching quality have involved traditional inpatient experiences,1–5 but a few studies have examined factors affecting the quality of ambulatory teaching.6–8 Irrespective of the clinical setting, studies have usually described teaching by regular university faculty members. However, outpatient, community-based experiences are becoming common, and they often involve volunteer faculty preceptors. Most reports have dealt with a single discipline, but ambulatory clerkships are now commonly multidisciplinary.9 Previous studies have shown that case content seen by students10 and educational needs perceived by preceptors11 differ across disciplines in primary care ambulatory settings. Because those elements of students' experiences differ, students' perceptions of what constitutes effective teaching may vary in these diverse ambulatory settings as well.
We had the opportunity to assess third-year medical students' evaluations of full-time and volunteer faculty members in the three primary care disciplines of family medicine, internal medicine, and pediatrics. We collected students' evaluations of their ambulatory preceptors over a complete academic year (July 2001 to June 2002) and examined them with the following questions in mind: Which teaching behaviors are associated with perceived effective teaching in ambulatory settings? Do these behaviors vary with preceptor characteristics, such as academic discipline or faculty status?
Our 12-week, ambulatory clerkship is mandatory for all third-year medical students at the University of Pittsburgh School of Medicine. Each student completes a three-week rotation in family medicine, internal medicine, pediatrics, and an elective chosen from one of these disciplines, women's health or emergency medicine. Students have the option of extending their rotation in family medicine, internal medicine, or pediatrics as their elective experience. Each week, students spend nine half-day sessions with a clinical preceptor, and they return to the medical school every Monday morning for didactic sessions that cover a variety of outpatient topics in pediatric and adult medicine.
During a single rotation, students may spend their clinical time with a single attending physician or within a group practice that may include residents. No physician groups include more than a single specialty, and few mix full-time with volunteer faculty members. Several groups include both male and female physicians. The majority of preceptors are generalists and practice in the Pittsburgh, Pennsylvania, area. Preceptors may hold either full-time or part-time volunteer faculty appointments. Many have participated in faculty development programs, and all are instructed about the course goals and objectives, which are for students to learn about and develop clinical skills in the following areas: (1) health systems finance and delivery, (2) the practice of evidence-based medicine with a population perspective, (3) patient care and health system ethics, (4) developing patient relationships and communication skills, (5) promoting effective working relationships between health care professionals, (6) quality management and improvement, (7) systems-based care, (8) medical informatics, and (9) wellness and prevention.
At the end of the clerkship and just before examinations are given, students complete a series of anonymous surveys on aspects of the clerkship. Among these is a 15-item evaluation on which the students rate the quality of the preceptors. Using a five-point, Likert scale (1 = “hardly at all” and 5 = “to a great degree”), the student answers eight questions on aspects of preceptors' teaching styles and ability, six questions on attainment of clerkship goals, and a question assessing overall teaching effectiveness. The evaluation form is similar to previously validated surveys.5,12 The student is asked to complete only one evaluation per rotation. If during a rotation the student interacted with a group of physicians, rather than a single physician, the student is told that the evaluation should reflect the overall experience. If the student selects a six-week rotation, he or she still completes a single evaluation for that rotation.
For the purposes of our study, students recorded the practice site and specialty on the evaluations forms. We reviewed the forms to add the preceptor's faculty status (full-time versus volunteer) and the preceptor's gender to the data set. Individual preceptor names were deleted before data entry.
We performed the analysis using a standard statistical software. Comparisons of means were performed with t-test, analysis of variance, or Wilcoxon rank sum test, when appropriate. Correlations between ratings of individual evaluation items were determined by using Pearson's r or Spearman's p. Multivariate analysis was performed with forward, stepwise linear regression.
We performed the research with the permission of the University of Pittsburgh's Institutional Review Board.
In 2001–02, 128 students completed our ambulatory clerkship. Because 37 students chose six-week rotations in one of the disciplines, they were each asked to complete only three preceptor evaluations. We received 276 evaluations from a possible 475, for a 58% response rate. The distribution of responses by preceptors' discipline, gender, and faculty status are shown in Table 1.
The students' ratings of the preceptors' teaching effectiveness were generally high, with a mean of 4.4 (SD .9) and a range of 1–5. These ratings did not differ significantly by preceptor gender, discipline, or faculty status (p > .2 for each) (Table 1).
For the remaining 14 items on the evaluation, students' mean ratings are shown in Table 2. The item on whether the preceptor provided useful feedback received the lowest rating (mean = 3.7), and the item on whether ethical medicine was practiced during the clerkship received the highest rating (mean = 4.7). The feedback and practicing ethical medicine ratings did not differ significantly among faculty disciplines or appointment types (p > .1 for each). The ratings of all 14 items significantly correlated with each other (r = .25 to r = .60, p < .01), and all were significantly associated with the ratings of teaching effectiveness (each p < .001).
To determine which of the 14 items were independently associated with the overall teaching effectiveness rating, we performed a multivariate analysis of each group of items. We found that four of the items in the preceptor teaching behavior group and three items in the clerkship goal attainment group remained independently associated with overall teaching effectiveness (Table 3).
Our study addressed the teaching effectiveness of a wide variety of ambulatory preceptors: men and women from the three primary care disciplines who were full-time and volunteer faculty members. In our analysis of students' perceptions of their experiences, we found that teaching effectiveness was not associated with the specialties, gender, and faculty status of the preceptors. These results may be reassuring to those directing clerkships who must balance the needs of students with the time constraints of full-time faculty members.
Medical education seminars and other skill-building exercises are more accessible to full-time faculty members, who are often considered more skilled as preceptors. However, increasing demands for clinical productivity have limited their time to precept medical students. Although the volunteer faculty is viewed as an important preceptor pool for students' clinical experiences, some educators express concern that the volunteer faculty may not possess the skills to teach medical students. One of us (IB) has devoted extensive efforts to faculty development programs among the volunteer faculty members at our institution, and these efforts may have improved their teaching effectiveness and had some effect on our results.
That 14 items on the evaluation were associated in univariate analysis with teaching effectiveness, the 15th item was not surprising. The items included were chosen to represent important aspects of clinical teaching and to reflect clerkship goals, of which both preceptors and students were made aware. Although studies of clinical teaching have repeatedly found feedback to be an important aspect of students' learning,13 clinicians find this to be a difficult aspect of medical education.14 Despite efforts to improve preceptor feedback to students, this aspect remains the lowest rated by the students and will continue to be an area on which to focus future faculty development efforts. Our students rated their preceptors highly on enthusiasm and their treatment of students, indicating that preceptors remain committed to teaching at a time when the stresses of practice could have compromised their teaching efforts. The high ratings for students' ability to see patients as first contact and their exposure to the ethical practice of medicine reassured us that these critical clerkship goals were being met.
Our multivariate analysis demonstrated that several preceptor behaviors are independently associated with perceived teaching effectiveness. These behaviors were inspiring confidence in medical skills, explaining decisions, treating students with respect, and providing a role model. In another study, students from three other medical schools expressed similar sentiments about their ambulatory internal medicine experiences in focus group interviews. Among the items they listed as important were explaining choices and several examples of treating students respectfully.6 A study of residents' evaluations of their continuity clinic preceptors noted that perceived clinical excellence was an important component of teaching effectiveness, as it was with our students.15 Providing a role model has often been cited as an important aspect of being an excellent clinical teacher.16 The ambulatory clinic may be more conducive than other settings to meeting the expectations of students. In two recent studies, ambulatory preceptors were rated better than inpatient preceptors in several areas, including improving clinical skills, showing respect for the students, availability, and giving feedback.17,18
We were pleased to see that three of the clerkship goals remained independently associated with the students' perceptions of effective teaching. These goals are to learn and practice clinical skills, to practice ethical medicine, and to encourage evidence-based medicine. Evidence-based medicine and ethics are emphasized during the didactic portion of our curriculum, and both students and preceptors are told that a primary purpose of the clerkship is for students to acquire clinical skills in outpatient settings. In the report of focus-group interviews mentioned earlier, those students also highly valued seeing patients first and approaching patient care with academic rigor.6 For medical students to mature as learners, being able to see patients independently within a structured, supervised environment seems to be very important. The value they place on evidence-based medicine, ethical practices, and the explanation of decision processes may reflect how students learn to organize and build on their experiences.
Our study had several limitations. First, we only obtained the perceptions from a single class at one medical school, so the generalizability of our findings may be limited. Other schools may have more, fewer, or different programs for faculty development. Second, our response rate was less than ideal. Part of the reason for the reduced response rate was that students who took their final examinations late, because of illnesses or other leaves, did not receive the preceptor evaluation forms. These forms were part of an extensive batch of clerkship exit surveys, and many students may have been fatigued by the end of the process. Third, although our overall sample was fairly large, the number of evaluations of each preceptor was small. A previous study of clerkship evaluations indicated that multiple raters are required for adequate generalizability.19 Finally, we had only determined the students' perception of effective teaching. Because of the anonymity of the surveys, we were unable to correlate perceived effectiveness with any objective outcomes, such as test scores.
Our study shows that students' perceptions of teaching effectiveness are influenced by a variety of preceptor behaviors but not by preceptor demographic characteristics. Preceptors should actively involve students in ambulatory medicine in such a way that they can build their clinical skills. Being the type of physician the student wants to emulate by demonstrating superior clinical skills, practicing evidence-based medicine, and practicing ethical medicine seems particularly important. Faculty development efforts should emphasize that these aspects of clinical medicine are important to medical students.
1. Stern DT, Williams BC, Gill A, et al. Is there a relationship between attending physicians' and residents' teaching skills and students' examination scores? Acad Med. 2000;75:1144–6.
2. Griffith CH, Georgesen JC, Wilson JF. Six-year documentation of the association between excellent clinical teaching and improved students' examination performances. Acad Med. 2000;75:S62–S64.
3. Roop SA, Pangaro L. Effect of clinical teaching on student performance during a medicine clerkship. Am J Med. 2001;110:205–9.
4. Griffith CH, Georgesen JC, Wilson JF. Specialty choices of students who actually have choices: the influence of excellent clinical teachers. Acad Med. 2000;75:278–82.
5. Litzelman DK, Stratos GA, Marriott DJ, Skeff KM. Factorial validation of a widely disseminated educational framework for evaluating clinical teachers. Acad Med. 1998;73:688–95.
6. Kernan WN, Lee MY, Stone SL, Freudigman KA, O'Connor PG. Effective teaching for preceptors of ambulatory care: a survey of medical students. Am J Med. 2000;108:499–502.
7. Kernan WN, O'Connor PG. Site accommodations and preceptor behaviors valued by third-year students in ambulatory internal medicine clerkships. Teach Learn Med. 1997;9:96–102.
8. O'Malley PG, Kroenke K, Ritter J, Dy N, Pangaro L. What learners and teachers value most in ambulatory educational encounters: a prospective, qualitative study. Acad Med. 1999;74:186–91.
9. Speer AJ, Stagnaro-Green A, Elnicki DM. Interdisciplinary clerkships: educational models of the future? Am J Med. 1995;99:451–3.
10. Carney PA, Pipas CF, Eliassen S, et al. An analysis of students' clinical experiences in an integrated primary care clerkship. Acad Med. 2002;77:681–7.
11. Quirk M, Stone S, Chuman A, et al. Using differences between perceptions of importance and competence to identify teaching needs of primary care preceptors. Teach Learn Med. 2002;14:157–63.
12. Copeland HL, Hewson MG. Developing and testing an instrument to measure the effectiveness of clinical teaching in an academic medical center. Acad Med. 2000;75:161–6.
13. Irby D. Teaching and learning in ambulatory care settings: a thematic review of the literature. Acad Med. 1995;70:898–932.
14. Hewson MG, Little ML. Giving feedback in medical education: verification of recommended techniques. J Gen Intern Med. 1998;13:111–6.
15. Hayward RA, Williams BC, Gruppen LD, Rosenbaum D. Measuring attending physician performance in a general medicine outpatient clinic. J Gen Intern Med. 1995;10:504–10.
16. Wright SM, Ken DE, Colander K, Howard DM, Barnacle FL. Attributes of excellent attending-physician role models. N Engl J Med. 1998;339:1986–93.
17. Mazor KM, Stone SL, Carlin M, Alper E. What do medicine clerkship preceptors do best? Acad Med. 2002;77:837–40.
18. Myers KA. Evaluating clinical teachers: does the learning environment matter? Acad Med. 2001;76:286.
19. Mazor K, Clauser B, Cohen A, Alper E, Pugnaire M. Student and resident evaluations of faculty: how dependable are they? Acad Med. 1999;74(10 suppl):S19–S21.