Matthews, Julia PhD, MD; Kadish, William MD; Barrett, Susan V. MS; Mazor, Kathleen EdD; Field, Deborah MD; Jonassen, Julie PhD
Since the early 1970s, awareness of the need to enhance training about substance abuse (SA) in medical education has led to initiatives both within the United States and abroad to support curriculum development.1–6 Follow-up surveys have documented substantial improvements in the number of required curriculum hours, availability of relevant electives, number of faculty teaching in SA areas, and perceptions of students of the adequacy of training.2,3,6 By 1992, 125 of the 126 U.S. medical schools had required SA curricula in place, although only eight of these schools had separately identified courses in SA treatment.6 As noted by Walsh,7 however, “Despite the substantial effort invested in achieving these curriculum gains, there is a dearth of research demonstrating impacts on medical behaviors or evaluating the cost-effectiveness of different educational strategies.” Numerous studies also document physicians' continuing failure to diagnose SA disorders or provide appropriate interventions.6–9 Many physicians continue to believe that SA is a social or moral issue rather than a medical disorder.10
As part of our response to the need for curriculum enhancement regarding the diagnosis and treatment of SA disorders, the University of Massachusetts Medical School (UMMS) introduced a two-day required SA interclerkship in 1997–98. This initiative was sponsored by the Department of Psychiatry, but it was specifically developed to involve a broad interdisciplinary faculty from the Departments of Psychiatry, Internal Medicine, Family Medicine, and Pediatrics, as well as community representatives from law enforcement, self-help groups, and community treatment settings. The program was positively received by students and faculty and has been refined over the subsequent three years. The interclerkship integrates a number of teaching formats, with an emphasis on small-group teaching and skill development.
Since 1997–98 all third-year students have participated in the interclerkship, which is scheduled into the academic year between the standard clerkship blocks. In 1997–98 and 1998–99, the interclerkship was a two-day intensive experience. The interclerkship consisted of lectures, precepted small-group (five—six students) interviewing sessions with standardized patients (SPs), facilitated discussion groups with real patients recovering from addictions, site visits to community detoxification and treatment facilities, and elective workshops on more specialized topics such as adolescent SA, addiction and the law, and twelve-step programs. Since 1999–00, the curriculum has been reduced to one day (precluding the site visits), but it has retained the emphasis on small-group teaching. Sessions have several key goals: the acquisition of core knowledge about SA disorders, intoxication and withdrawal syndromes, and acute and long-term treatment approaches; the fostering of an attitude of empathy and realistic optimism in relation to these disorders; and the development of basic skills for interviewing, assessing, and intervening with patients who have SA disorders.
Pre- and post-interclerkship testing has consistently demonstrated that students enhanced their knowledge and had more positive attitudes about diagnosing and treating SA disorders.11 Furthermore, the faculty preceptors in the small-group sessions generally noted students' improved interviewing skills and confidence despite the short time span of the interclerkship. We were interested in whether these changes led to demonstrably better skills in subsequent clinical encounters, i.e., whether students retained and practiced the skills in the months following the interclerkship.
To evaluate students' acquisition and retention of knowledge, positive attitudes, and skills regarding SA disorders, we collected and analyzed pre- and post-interclerkship knowledge and attitude scores, students' performance scores on an objective structured clinical examination (OSCE), and data from the AAMC's Graduate Questionnaire.
Initial Questionnaire and Pre- and Post-interclerkship Tests
The students completed an initial questionnaire, a pre-interclerkship test, a final course evaluation, and a post-interclerkship test. The initial questionnaire gathered demographic information, including self-reported past work or personal experience (“friends or family”) with SA disorders. The pre- and post-interclerkship tests asked identical questions covering attitudes about SA disorders and treatment (eight items), and knowledge about SA disorders and their treatment (18 items). The attitude-assessment instrument was modified from that of Chappel et al.12,13 using items from the factors for “treatment intervention” and “treatment optimism,” the two factors that have been shown to best discriminate clinicians experienced in working with SA disorders from the general clinician population.12 An additional item (“The diagnosis of substance use disorders is a fundamental medical responsibility”) was added to our attitude assessment. The knowledge assessment instrument was developed by one of us (WK) using items from the Brown University Project ADEPT3 materials as well as original questions. At the time of the pre- and post-interclerkship tests, the students were also asked two questions that probed each student's level of confidence in (1) assessing SA problems and (2) providing brief interventions in office visits.
All students at UMMS complete a six-week psychiatry clerkship during their third year. As part of the evaluation of students at the end of the clerkship, each student performs one or more observed clinical interviews with a standardized patient acting a preset clinical vignette (OSCE). Faculty observers rate the students' interviews on 12 measures (including professionalism, rapport, adequacy of the clinical history, mental status examination skills, risk assessment for dangerous behaviors, and diagnostic reasoning) using a four-point scale (1 = below expected, 2 = expected, 3 = above expected, and 4 = outstanding; maximum possible score = 36). The observers are drawn from a cadre of about 20 psychiatry faculty who are generally experienced OSCE observers and who receive individual training and feedback from the two clerkship directors (JM, DF).
Throughout 1998–99, students were required to conduct two OSCE interviews. Although the students were told only that there were two different clinical situations, each student had one standardized case without active SA issues (requiring only an adequate SA screening) and one standardized case in which SA was an active concern. The interview for the active SA case should have included: (1) a detailed characterization of the patient's patterns and amounts of use, (2) an assessment of evidence for SA or dependence and of the patient's readiness for change, and (3) an appropriate brief intervention (communication of concern about SA and an attempt to negotiate an initial plan to address this issue). Both interviews were rated on the same general performance measures, and students were rated separately on their SA “screening” (for the non-active SA case), and “details,” “assessment,” and “intervention” (for the active SA case). Global performance on the active SA case was also calculated as the summed score for these last three measures. Since the SA interclerkship was offered in January 1999, only those students who did the psychiatry clerkship in the second half of the academic year had also done the interclerkship. This allowed a group comparison of the impact of the interclerkship on students' skills as seen in the OSCE interviews.
All students graduating from U.S. medical schools are surveyed by the AAMC following graduation regarding their perceptions of the adequacy of their education in various skill and content areas (“inadequate,” “appropriate,” or “excessive”) The item “Drug and Alcohol Abuse” has been included on the-survey for many years, allowing us to assess changes over time in our students' perceptions of the adequacy of their training as compared with the responses for all U.S. students.
We compared group means using non-directional t-tests and compared the pre- and post-interclerkship test scores for knowledge, attitudes, and confidence using non-directional paired sample t-tests. We calculated Spearman correlations to examine the relationships among attitude, knowledge, confidence, and self-reported past experience within the pre- and post-interclerkship tests, and to evaluate the relationship between students' performances on the OSCE and their responses on the pre- and post-interclerkship assessments. We used chi-square tests (Fisher's exact test, two-tailed) to compare the frequencies of OSCE performance ratings falling below and above the “expected” level for the groups of students completing the OSCE before or after the interclerkship. Finally, we used stepwise multiple regression to examine the contributions of the SA interclerkship and the internal medicine and family medicine clerkships to a student's OSCE performance.
Between 1997 and 2001, 386 students participated in the SA interclerkship. Data are presented for 1998–99 (two-day format, 100 students) and 1999–00 and 2000–01 (one-day format, 97 and 93 students, respectively), years for which we had comparable outcome measures. Students' evaluations of the interclerkship were positive overall, particularly with respect to the interactive small-group sessions and workshops. Almost all of the students agreed (“agree” or “strongly agree”) that “this material is essential to their medical education” (>95%), and that “the teaching formats were appropriate to meet the course objectives” (>90%). The majority of students (85% in 1998–99, 71% in 1999–00, and 79% in 2000-01) also agreed that the interclerkship “provided opportunities to gain new knowledge and skills which are not available elsewhere in my medical education.”
Prior training and experience in SA. The distributions of students' responses about prior professional experience with SA disorders did not differ substantially in the three academic years. At the start of the interclerkship, about 90% of students felt that they had had “some” or “moderate” background knowledge about SA disorders prior to medical school, and a similar percentage of students (85–90%) indicated “some” or “moderate” prior training since entering UMMS. A significant minority of UMMS students (17–37%) reported no personal experience of family or friends with SA disorders, while about 25–35% of students reported “moderate” or “extensive” personal experience.
Changes in knowledge about and attitudes toward SA. A comparison of students' responses on pre- and post-interclerkship testing using paired samples indicated significant increases in the students' levels of knowledge about SA disorders and in their attitudes about working with SA patients (see Table 1). Students' pre-interclerkship attitude scores were not significantly correlated with their knowledge scores (paired samples). The students' scores on attitudes and knowledge tests were not correlated with self-reported prior work experience or past experience with SA disorders in family or friends.
Changes in students' confidence. Following the interclerkship, the students indicated increased confidence in addressing SA issues. Table 2 shows the distributions of responses on the pre- and post-interclerkship test surveys for two key confidence questions. As a result of their interclerkship experience, the number of students who felt uncertain about these skills decreased dramatically and, conversely, the number of students who felt very confident about these skills increased dramatically. To examine the statistical significance of these changes, we combined negative (“strongly disagree” + “disagree”) and affirmative (“agree” + “strongly agree”) responses. Using McNemar's test, these changes were highly statistically significant (p < .001) for both items in all years.
In general, students who reported higher levels of past work or training experience also expressed greater pre-interclerkship test confidence in their skills. However, students' expressed confidence on the pre-interclerkship test about performing an SA assessment or intervention was not correlated with self-reported personal experience with SA in family or friends.
A total of 99 students participated in the OSCE exams during 1998–99. Of these, 96 completed the non-active SA case (rated for appropriate “screening” for SA only) and 97 completed the active SA case (rated for “details,” “assessment,” and “intervention”). Faculty ratings were missing for three students on the measures of assessment and intervention.
Table 3 presents the group mean overall OSCE performance scores, the group mean scores for the four SA rating items (screen, details, assess, and intervene), and the summed scores for the active SA case, divided by half-year, i.e., before and after the interclerkship. As expected, the students' overall performance on the OSCE improved over the year as they gained more clinical experience (pre- and post-interclerkship comparison significant at p < .05). No significant difference was seen in the students' pre- and post-interclerkship performances of basic SA screening for the non-active SA case (screen) or in the elaboration of greater details when the screen was positive for the active SA case (details). However, the students showed significant improvement in the abilities to assess SA issues (assess, p = .005) and to provide an appropriate intervention (intervene, p < .05). The SA summed score (the three items assessed for the active SA case) integrates these improvements, showing a clear improvement from the first half of the year to the second half of the year (p = .02). Students' ratings on “assess” were strongly correlated with their ratings on “intervene” (r = .637, p = < .001), showing a consistent pattern of skill level for each student.
The most striking finding emerged when we examined the frequencies of “below expected” and “above expected” and “outstanding” ratings on OSCE items (see Figure 1). A substantial minority of students (15 of 48 students, or 31%) received below-expected ratings on “assess” and/or “intervene” during the first half of the academic year, whereas only two students (<5%) received this rating on one or both of these items in the second half of the year.
We found no significant correlation between the overall active SA OSCE score and the interclerkship measures of knowledge, attitudes, and confidence (pre- and post-interclerkship tests). Nor was the overall active SA OSCE score correlated with self-reported background knowledge of SA disorders prior to medical school (r = −.061, p = .630), student's perception of the previous amount of SA training at UMMS (r = .013, p = .921), or self-reported degree of past experience with SA disorders in family or friends (r = .092, p = .464). Students' performance ratings for “assessment” on the OSCE were not correlated with their reported confidence about doing assessment of SA issues (r = .054, p = .652), either for all students or for the second half-year. Likewise, students' performance ratings for “intervention” on the OSCE were not correlated with confidence about doing a brief office intervention about a SA issue (r = −.050, p = .677).
No difference was found in students' overall active SA OSCE scores as a function of the primary site to which the student was assigned for the psychiatry clerkship. Of particular note, students assigned to a designated dual-diagnosis unit did no better than did their peers at other sites. The students' OSCE performances on the SA items also did not differ as a function of whether the student had completed the internal medicine or family medicine clerkships. In a stepwise multiple regression analysis, the variables for completion of the internal medicine and family medicine clerkships did not meet the entry criterion for the model (p < .1), whereas the interclerkship did (p = .02), although it accounted for only about 5% of the variance.
Nationally the percentage of graduating medical students who report that their training in SA treatment has been “inadequate” (as opposed to “appropriate” or “excessive”) has declined steadily since the mid-1970s,6 and this same trend has also been evident at UMMS. Figure 2 shows the percentages of students at UMMS and nationally who rated their education as “inadequate” over the graduation years 1993 to 2000. Major curricular changes begun in the mid-1990s at UMMS seem to have had a significant effect by the class of 1998. However, following the introduction of the SA interclerkship for third-year students in 1997–98 (class of 1999), the number of students indicating “inadequate” training in SA disorders decreased dramatically.
Over the past two decades, a number of innovative strategies have been reported to enhance medical school education about the identification and treatment of SA disorders.1,2,4,13–16 Some of these approaches immerse students in a SA treatment setting,14 some are primarily didactic,4 and some combine didactic training with clinical experience.13,15 We offer students a brief intensive training experience (an interclerkship) that integrates didactic sessions, interaction with patients in recovery from SA disorders, and skill development through role play with SPs. The design of the interclerkship is consistent with recommendations that SA training be experiential and specifically focus on development of the necessary skills for assessment and intervention.1,7,17,18 In their evaluations of the interclerkship, our students strongly endorsed the need for SA training as well as the usefulness of our approach to this curriculum. Furthermore, the Graduate Questionnaire survey data show a dramatic improvement in our graduating students' perceptions of the adequacy of their SA training, which is clearly correlated with the introduction of the SA interclerkship into the thirdyear curriculum. Curricular changes begun in the mid-1990s seem to have had a significant effect by the class of 1998. However, the dramatic changes for the classes of 1999 and 2000 are temporally related to the introduction of the SA interclerkship.
Curricular change, however, does not necessarily lead to a change in clinical performance. Clinical performance reflects a complex interaction of knowledge, attitudes, and skills. It is generally recognized that cognitive aspects are more easily altered than are attitudinal aspects. It is also believed that negative attitudes have a detrimental impact on the care of patients with SA disorders, and that a positive change in attitudes will be associated with a change in clinical practice.1,12,13 Our study replicated the findings of others that a brief teaching intervention leads to significant immediate changes in students' knowledge, attitudes, and skills regarding SA.1,13,14,17 These positive changes were comparable in magnitude, whether the interclerkship's format was one or two days in length. Furthermore, we demonstrated significant increases in students' confidence about their skills following the interclerkship. Most important, however, we found that students who participated in the SA interclerkship performed better on a subsequent OSCE that required SA assessment and intervention even when they had not been specifically directed to address this issue. Since this examination was part of the psychiatry clerkship, and therefore occurred up to six months after the interclerkship, we feel that this demonstrates an enduring change in the students' clinical practices. Furthermore, whereas about 30% of the students completing the OSCE before the interclerkship did not provide an appropriate assessment and/or intervention for active SA issues (i.e., were rated “below expected”), fewer than 5% of the students completing the OSCE after the interclerkship were judged to be deficient in these areas. Thus, the interclerkship may have had its greatest effect on those students who most needed training in skills.
During the psychiatry clerkship at UMMS, students see patients with SA disorders in various clinical settings and are given didactic instruction about SA assessment (one two-hour lecture and a five-hour interviewing seminar in which the SA assessment is one of several areas stressed). At the time of the psychiatry OSCE, students were instructed only to do a thorough psychiatric interview (i.e., they were not specifically instructed to do comprehensive SA assessment or specific SA intervention). Performances on these measures might well have been considerably better had they seen this as the identified task.17 Our objective, however, was to evaluate their performances of these skills in a more natural clinical situation. In this situation, about 80% of our students performed a satisfactory screen for SA on a case in which SA issues were not present, and 90% of our students elicited a detailed history of substance use in the case with active SA issues. The screening and details scores did not change between the first and second half of the year, suggesting that most students had already integrated these skills into their routine interviews even at the beginning of third year. The changes we observed following the interclerkship were specifically in the students' abilities to provide assessment (a more complex task including evaluation of substance dependence and readiness for change) and intervention (a counseling and negotiation skill). These skills were a specific focus of the small-group sessions in the interclerkship, building on their general preclinical training about facilitating behavioral change. Nevertheless, the observed changes that can be attributed to the interclerkship were limited. It is likely that repeated reinforcement in multiple settings through a broadly integrated longitudinal SA curriculum is needed for students to maintain and further develop these skills.
An alternative interpretation of our data would be that we simply documented the expectable general improvement in students' clinical skills over the third year. Indeed, third-year students at UMMS are exposed to SA curriculum in multiple settings other than the interclerkship, notably the internal medicine, family medicine, and psychiatry clerkships. Since the OSCE examination was at the end of the psychiatry clerkship, all students had been equally exposed to this intervention. In a multiple regression analysis we were unable to demonstrate that having completed the internal medicine or family medicine clerkship made any contribution to a student's performance on the SA measures of the OSCE. Only the interclerkship remained in the model as a significant predictor of performance. Therefore, while we would agree that general clinical experience is almost certainly an important factor, we nevertheless feel that the interclerkship had specific positive effects on SA skills.
About 25–35% of the UMMS students reported “moderate” or “extensive” experience with family or friends suffering from SA disorders, consistent with the finding of Waller and Casey9 that 31% of medical students in their cohort had family backgrounds with substantial SA problems. However, we found no correlation between past personal experience with SA disorders and either knowledge, attitudes, or confidence regarding SA issues. This is somewhat surprising in light of prior suggestions that physicians' personal exposures to SA disorders may influence their attitudes and willingness to address SA issues.9 It may be that no significant association was found because some students hold more optimistic attitudes based on their personal experiences while others hold more negative attitudes. Our data did not allow us to examine this important question.
In contrast to prior personal experience, prior work experience seemed to predict more positive students' attitudes on the pre-interclerkship test and greater confidence levels at the beginning of the interclerkship. This supports the idea that increasing students' exposure to SA disorders enhances their comfort with these disorders and engenders a more positive attitude toward treatment. Earlier findings19 that students' and residents' attitudes become more negative as they progress through training may be less a function of direct clinical experience and more related to a negative bias within the educational culture. Exposure to positive clinical outcomes may be the most important factor in shaping students' future practices. UMMS students have been especially positive about the opportunities within the interclerkship to interact with patients in recovery from SA disorders.
The students expressed greater confidence following the interclerkship in all years. Greater confidence may increase students' willingness and comfort to address these issues in clinical settings. However, it should be noted that we did not find a correlation either between confidence and knowledge, as measured on our pre- and post-interclerkship testing, or between confidence and skills, as measured on our OSCE assessment. Further work is needed to clarify the relationships between subjective confidence and actual knowledge, attitudes, and skills in this complex aspect of clinical practice.