Secondary Logo

Journal Logo

Research Report

The Influence of Teaching Setting on Medical Students’ Clinical Skills Development: Is the Academic Medical Center the “Gold Standard”?

Carney, Patricia A. PhD; Ogrinc, Greg MD, MS; Harwood, Beth G. MEd; Schiffman, Jennifer S. MPH; Cochran, Nancy MD

Author Information
  • Free

Abstract

Becoming a competent physician is a complex process, and much needs to be learned about the development of clinical skills and how they evolve over time. In the 1990s, several reports indicated that clinical skills need to be taught earlier in medical education.1–3 Following these reports, several initiatives were launched to begin earlier clinical training and mentoring by primary care providers and to encourage primary care as a career choice.4–7 U.S. medical schools followed suit despite the challenges involved in ensuring students’ placements, such as increased patient volume in outpatient settings, increasing pressures on teaching faculty, and competition for patients by multiple levels of learners.8 By 2001, 94% of U.S. medical schools were using community preceptors as clinical teachers, especially in ambulatory settings,9 despite the fact that teaching associated with busy ambulatory settings can vary in quality and content.10,11

The majority of published research on early clinical education has focused on satisfaction with the learning experience12,13 or has described untested methods to assess consistency.14 Our early work assessed the types of early clinical learning encounters students were exposed to in their ambulatory preceptor sites and how these encounters evolved between the first and second year of medical school.15 For students to excel in the clerkship and elective years, they need a solid clinical foundation, so it is vital to assess the clinical teaching settings in the first two years of medical school. Assessing clinical performance in medical school has become increasingly important, especially as the National Board of Medical Examiners (NBME) has implemented the Clinical Skills Exam (CSE) portion of the United States Medical Licensing Examination (USMLE) Step II.16 Current educational research must focus on what factors influence the development of core clinical skills so we can create the best learning environments for our students. In this study, we tested the hypothesis that second-year medical students’ overall clinical performances would be highest if they received training in academic medical center (AMC) clinics versus either office-based AMC-affiliated clinics or community-based primary care offices. We hypothesized that the “gold standard” for the clinical teaching mission would be the AMC where course directors developing curricular content were located.

Method

The educational setting

Dartmouth Medical School (DMS) in Hanover, New Hampshire, is affiliated with the Dartmouth-Hitchcock Medical Center in Lebanon, New Hampshire, which supports a range of residency and fellowship programs. Between 78 and 90 students enter DMS each fall, with approximately 18 transferring to Brown Medical School in Providence, Rhode Island, for their clinical training in the third year, and 60–70 graduating from DMS after four years. During the first and second year the students take the course “On Doctoring—a Longitudinal Clinical Experience.” The mission of this required course is to provide the fundamentals of interviewing and physical examination, and introduce concepts related to the doctor–patient relationship. Each year, the course includes large- and small-group exercises alternating weekly with half-day or longer visits to a clinical preceptor. Approximately eight medical students are in each small group.

Table 1 outlines the educational content of this clinical course by program year. Each large-group session lasts approximately one hour, each small-group session lasts three hours, and each preceptor visit lasts an average of four hours. In general, the clinical topic is introduced and described in the large-group session. Initial practice sessions and discussion occurs in small groups, and then students apply the skills they’ve learned in the classroom settings at their preceptor’s offices.

Table 1
Table 1:
Content of the Longitudinal Clinical Skills Course “On Doctoring I & II” for First- and Second-year Medical Students, Dartmouth Medical School, Hanover, New Hampshire

Medical students’ clinical placements are made by the course co-director (JSS), who, along with a community-based faculty liaison (GO), identifies and recruits the number of preceptors needed in each year of the course. Preceptor placements are done systematically in the first year prior to the start of the course, and no specific criteria are used to make the placements beyond matching an alphabetized list of students to the clinical placement database. Because students have yet to arrive on campus when these placements are made, we believe very little bias enters the decision making process. Because our study design was observational rather than randomized, this approach represents what occurs in the majority of U.S. medical schools. We included students from two sequential cohorts in our analysis. The first cohort was 70 second-year students in 2002–03 and the second cohort was 85 second-year students in 2003–04.

Students generally stay with the same preceptor for both years of the “On Doctoring” course. Occasionally students switch preceptors between their first and second years. Students who switched in the 2002–03 cohort did so within the stratum assessed in this study (e.g., switches occurred within the group of community-based preceptors, AMC preceptors, or office-based AMC-affiliated preceptors), and did not affect study group assignment. However, three students in the 2003–04 cohort switched from the AMC in their first year to community-based settings in their second year, and five students switched from community-based settings to the AMC in their second year. Because more intensive skills are taught in the second year of the course, we used the second-year site to classify students into their analytic groups. We believe this small number of switches (5.1%) had very little, if any, effect on the results.

Design and development of the OSCE

As part of the “On Doctoring” course, in the second-year OSCE, each student evaluates two standardized patients (SPs) consecutively. Every one of the five SPs portraying each case presents with symptoms of either a cardiopulmonary or an endocrine condition. Students have 20 minutes to conduct a focused history and physical examination. In the endocrine case, students are also expected to educate the patient about possible causes of their condition.

The SP cases and measurement instruments were designed and tested jointly by the course director (NC) and DMS’s clinical skills evaluator (PAC). This examination is part of the standard evaluation process in the second year at DMS, and the school has an evaluation policy to notify students that publications may result from students’ standard evaluations. Thus, this project was exempt from review by the Medical Education Institutional Review Committee (the committee that reviews educational research projects). Table 2 shows the SP cases and the assessment variables collected. The cardiopulmonary case involves a 55-year-old male patient presenting with shortness of breath, which is associated with not taking previously prescribed beta-blockers because of the resulting erectile dysfunction. The endocrine case is a 50-year-old female patient presenting with fatigue, weight loss, and anxiety. SPs are chosen from the pool of eligible individuals used for evaluation at DMS based on the patient’s age and sex in the case to be portrayed. Training was conducted jointly by the SP trainer (BGH) and the clinical skills evaluator (PAC). The clinical skills evaluator (PAC) also trained faculty observers.

Table 2
Table 2:
Case Characteristics of the Objective Structured Clinical Examination and Assessment Variables in the “On Doctoring” Course, Dartmouth Medical School, Hanover, New Hampshire

We measured students’ skills using a multiitem checklist. A faculty observer scored the cardiopulmonary case (due to its complexity) and the SP scored the endocrine case. All cases and instruments were tested prior to implementation using techniques similar to those used in research studies that used unannounced SPs.17–19 All encounters were videotaped, and an extensive data quality assessment program is in place to monitor case portrayal and scoring.

We assessed data quality using several approaches. Once data from the checklists were entered, the data file was cleaned of any data entry errors. Data were then analyzed using indicator variables for each SP playing the case and the faculty observers scoring each case to determine possible outliers in how examinations were scored. Faculty observers and SPs were assigned to students systematically; however, to reduce scoring bias, students were not paired with a faculty observer who was either their preceptor or small-group facilitator. We noted no outliers in either the SP portrayal or case scoring. In our quality assurance assessment, we found no difference in the SP portraying or scoring the case, and we found no differences in scoring among faculty observers.

Analysis

Each medical student’s preceptor was classified into one of the three different settings under study (community-based offices, AMC clinics, or AMC-affiliated office-based clinics). Community-based physicians have no direct relationship with DMS, although they may have adjunct faculty status. They are paid $750 for each student they precept per year in the On Doctoring course. Preceptors with formal relationships to the medical school have faculty status at the instructor or assistant professor level and are not paid to precept medical students.

We assessed for heterogeneity in the scores of the 70 second-year students in 2002–03 and the 85 second-year students in 2003–04, and after finding no significant differences, the data from both years’ examinations were merged into one analytic file. Then individual checklist variables were classified into three skill areas: overall communication (introduction, questioning, listening, eye contact, and responding skills); history taking (symptom analysis, assessment of associated symptoms, past medical, family, social, and other related history); and physical examination skills (vital signs, heart and lung for cardiopulmonary case and neck, abdomen, heart, and lungs for the endocrine case—see Table 2). The endocrine case additionally assessed the student’s ability to educate the patient about her condition. Each variable was scored with a “0” if the skill was not done, a “1” if the student could use feedback on how to improve, and a “2” if the skill was done well. Individual items were summed to reflect a score for each skill area, then expressed as percentages by dividing the mean score by the total possible score. We made comparisons across study groups using descriptive statistics and analysis of variance (analyses of variance [repeated-measures analysis of variance {ANOVA}] - mixed model). We conducted subgroup analyses on preceptor discipline (family medicine, internal medicine, pediatrics, and specialties) and students’ and preceptors’ gender to identify possible covariates in our study. We additionally used general linear regression techniques to assess the relationship between overall scores on the two cases and teaching setting. All tests were two-sided with alpha levels set at .05.

Results

One hundred and fifty-five students participated in the OSCE in the On Doctoring course. The characteristics of the medical students and preceptors are shown in Table 3. As indicated, the students were fairly evenly distributed by gender, but 105 (67.7%) preceptors were male. Ninety-five (61%) preceptors were community-based, while 31 (20%) were AMC-based and 29 (19%) were in AMC-affiliated office-based clinics. Sixty-two preceptors were family physicians (40%), 55 (36%) were in internal medicine, 22 (14%) were pediatricians, and the 16 remaining preceptors were specialists, including five cardiologists, two obstetrician/gynecologists, five general surgeons, two in emergency medicine, and two infectious disease specialists.

Table 3
Table 3:
Characteristics of Preceptors and 155 Second-year Medical Students Who Participated in the Objective Structured Clinical Examination, Dartmouth Medical School, Hanover, New Hampshire, 2002–03 and 2003–04

Table 4 shows the students’ mean scores (SD) on the OSCE, the findings for the study’s main hypothesis. We found that students’ performances did not differ among clinical learning sites with overall scores in the cardiopulmonary case of 61.2% in AMC clinics, 63.3% in office-based AMC-affiliated clinics, and 64.9% in community-based offices (p = .20). Scores in the endocrine case similarly did not differ, with overall scores of 65.5% in AMC clinics, 68.5% in office-based AMC-affiliated clinics, and 66.4% in community-based offices (p = .59). Summary communication skill scores ranged from 77% to 93% across the two cases and did not differ statistically across teaching settings. Summary history-taking scores ranged from 44% to 77% across the two cases, and again, and we noted no statistical differences by teaching setting. Summary physical examination scores ranged from 48% to 76%, and noted no statistical differences by clinical learning setting. The only skill area in which we noted differences by clinical teaching setting was for patient education in the endocrine case, where the mean score was 71.7% in AMC clinics compared to 90% in AMC-affiliated office-based clinics and 85% in community-based settings (p = .01). Figure 1 is a scatter plot with general linear regression findings for overall scores for the two cases. We assessed performance on one case compared to the other case to identify possible differences (none were found) with r squares ranged from 0.001 to 0.12 with p values > .20.

Table 4
Table 4:
Mean Scores (SD) of 155 Second-year Medical Students on the Objective Structured Clinical Examination (OSCE), by Learning Setting, Dartmouth Medical School, Hanover, New Hampshire, 2002–03 and 2003–04
Figure 1
Figure 1:
General linear regression of total case scores by preceptor type. F for corrected model of between-subjects effects by preceptor type = 0.53 for cardiopulmonary case (p = .59) and 1.63 for cardiopulmonary case (p = .20). From a study of 155 second-year medical students who took an objective clinical skills examination in the course “On Doctoring –a Longitudinal Clinical Experience” at three preceptor sites, Dartmouth Medical School, Hanover, New Hampshire, 2002–03 and 2003–04.

In our subgroup analysis of clinical skills scores by preceptor discipline (family medicine, internal medicine, pediatrics, and specialties), we found no statistical differences for any exam component in either the cardiopulmonary or endocrine case by preceptor discipline. Nor did we find any statistical differences in students’ scores by small group for either case.

Discussion

In this study of early clinical learners, we rejected our hypothesis that the AMC is the “gold standard” of clinical teaching. We found in our analysis of the OSCE scores of 155 second-year medical students that clinical teaching setting does not appear to influence the development of core clinical skills. Our only statistically significant finding was in one specific skill area: Patient education skills among students who trained at the AMC were lower than were scores of students who trained in the other two settings. Perhaps this difference can be explained by less available time on the part of the preceptor or by different roles within a practice setting. Patient education may be done more often by a nurse or other ancillary health care professional in an AMC compared to the other two settings.

Preparation for the clerkship years is vitally important, which is why we chose to conduct this analysis at the end of the second year before students enter the intense clinical clerkship years. In fact, conducting such an analysis in the third year would be exceedingly difficult because the complex sequence of clinical rotations, variations in teaching setting and individual site would produce more confounding variables than is possible to study in one medical school.

Although our findings did not support our hypothesis, we were pleased to discover that community-based teaching sites can offer teaching quality that is at least as high as teaching quality in the AMC. Community-based preceptors, those affiliated most distantly with our medical school, appear to be committed to the teaching and learning responsibilities associated with medical students who are early in their training. This phase in learner development often requires additional time, since these students are simultaneously learning about the basic science foundation of medicine and associated clinical conditions. As stated earlier, we do pay community preceptors for every medical student they precept in the longitudinal course, although we do not know if this payment makes a difference in the teaching preceptors undertake. This is a possible area for future research.

As we have learned from other reports of clinical teaching,4–10 community preceptors often make up the bulk of clinical educators, which is certainly the case at DMS with more than 60% of them precepting in the first and second years. We are more confident now that this group of clinical preceptors makes fine teachers, and we will continue our efforts to support them.20

One strength of our study is that we used two cohorts of students, whose scores on these patient cases did not differ significantly between 2002–03 and 2003–04, indicating the stability of our findings. Some may question whether two patient cases were a valid measure of our outcome variable; however, we were not assessing individual student or individual preceptor performance, the former of which has been reported to require between six and eight SPs.21 Our goal was to assess overall performance by clinical training site. We had three study groups with 95, 31, and 29 individuals in each respectively. Using the smallest group-wise sample size (n = 29), and an alpha level of .05, a u (degrees of freedom) of 2 (for a 3-way comparison), and an f (effect index) of 0.34, these provided 80% power for the F test to detect differences among the three study groups.22 While assessing clinical training sites in the clerkship year is also important, the complexities of assessing clinical settings in the clerkship years are much more challenging due to the number of settings, variation in the sequences of rotations, and differences in clinical skills assessments. To address this research question adequately, many medical schools would need to take part.

In conclusion, we found that second-year medical students’ clinical performance skills did not differ overall by the preceptors’ teaching settings. OSCE scores for students precepted in community-based and academic medical center affiliated clinics were similar. Since students’ performances were similar across sites and disciplines, we can now focus on improving early clinical training across all sites.

This work was supported by the Office of Community-based Education and Research at Dartmouth Medical School and with resources and facilities at White River Junction VA.

References

1General Medical Council. Tomorrow’s Doctors. Recommendations on Undergraduate Medical Education. London: General Medical Council, 1993.
2Nelson MS, Traub S. Clinical skills training of U.S. medical students. Acad Med. 1993;68:926–28.
3Marcus E, White R, Rubin RH. Early clinical skills training. Acad Med. 1994;69:415.
4Matson CC, Ullian JA, Boisaubin EV. Integrating early clinical experience curricula at two medical schools: lessons learned from the Robert Wood Johnson Foundation’s Generalist Physician Initiative. Acad Med. 1999;74 (1 suppl):S53–S58.
5Carney PA, Bar-on ME, Grayson M, Klein M, Cochran N, Eliassen MS, Gambert SR, Gupta KL, LaBrecque MC, Munson PJ, Nierenberg DW, O’Donnell JF, White-hurst-Cook M, Willett RM. Assessing the Impact of Early Clinical Training in Medical Education: a Multi-Institutional Perspective. Acad Med. 1999;74(1 supp):S59–S66.
6O’Brian-Gonzales A, Blavo C, Barley G, Steinkohl DC, Loeser H. What did we learn about early clinical experience? Acad Med. 2001;76 (4 suppl):S49–S54.
7Steele D, Susman J, McCurdy F, O’Dell D, Paulman P, Stott J. The Interdisciplinary Generalist Project at the University of Nebraska Medical Center. Acad Med. 2001;76 (4 suppl):S121–S26.
8Grum CM, Richards PJ, Wooliscroft JO. Shifting medical student education to the outpatient setting: what are we losing? Acad Med. 1995;70:447
9Walling AD, Sutton LD, Gold J. Administrative relationships between medical schools and community preceptors. Acad Med. 2001;76:184-87.
10Irby D. Teaching and learning in ambulatory care settings: a thematic review of the literature. Acad Med. 1995;70:898–931.
11Carney PA, Pipas CF, Eliassen MS, et al. An analysis of students’ clinical experiences in an integrated primary care clerkship. Acad Med. 2002;77:681–87.
12Lam TP, Irwin M, Chow LW, Chan P. Early introduction of clinical skills teaching in a medical curriculum–factors affecting students’ learning. Med Educ. 2002;36:233–40.
13Makoul G, Altman M. Early assessment of medical students’ clinical skills. Acad Med. 2002;77:1156.
14Matson C, Davis A, Steinkohl DC, Blavo C. How did we make the Interdisciplinary Generalist Curriculum Project Work? School-level efforts to facilitate success. Acad Med. 2001;76 (4 suppl):S31-S42.
15Ogrinc G, Eliassen MS, Schiffman JS, Pipas CF, Cochran N, Nierenberg DW, Carney PA. Early clinical training in medical school: aligning early clinical experiences with the curriculum. (In Review).
16United States Medical Licensing Examination Web Site. Clinical Skills Examination. Available at 〈http://www.usmle.org/step2/step2cs/step2indexes/step2csindex.htm〉. Accessed 8 September 2005.
17Carney PA, Dietrich AJ, Freeman DH, Mott LA. The periodic health examination provided to asymptomatic women: an assessment using standardized patients. Ann Intern Med. 1993;119:129–35.
18Carney PA, Ward DH. Using unannounced standardized patients to assess HIV preventive practices of family nurse practitioners and family physicians. Nurse Pract. 1998;23:56–76.
19Carney PA, Dietrich AJ, Eliassen MS, Owen M, Badger LW. Recognizing and managing depression in primary care: a standardized patient study. J Fam Pract. 1999;48:965–72.
20Hamann C, Volkan K, Fishman MB, Silvestri RC, Simon SR, Fletcher SW. How well do second-year students learn physical diagnosis? Observational study of an Objective Structured Clinical Examination (OSCE). BMC Med Educ. 2002;2:1.
21Margolis MJ, Clauser BE, Swanton DB, Boulet JR. Analysis of the relationship between score components on a standardized patient clinical skills examination. Acad Med. 2003;78:S68-S71.
22Cohen J. Statistical Power Analysis for the Behavioral Sciences. Chapter 8: F Tests on means in the analysis of variance and covariance. New York: Academic Press, 1977:313.
© 2005 Association of American Medical Colleges