Purpose: During 2003, 2004, and 2005, the role of 70 tutors was changed from that of facilitator to discussion leader, in a preclinical PBL learning course, Gastrointestinal Pathophysiology, by use of three key business school teaching strategies: questions, summaries, and schematics. The purpose of this study was to learn what difference this new approach made.
Method: During each of the three study years, 171 (2003), 167 (2004), and 170 (2005) students were given Likert-scale attitudinal questionnaires to rate whether their tutors encouraged student direction of the tutorials and whether the summaries and closure schematics benefited their learning. Students’ overall course evaluations and mean USMLE scores were quantitatively analyzed, pre- and postintervention. A variety of statistical tests were used to assess the statistical significance of means at the confidence level of .05.
Results: In the third year of the program, student ratings indicated that their tutors were significantly better at encouraging student direction of the tutorials than in the first year (P < .05). The students reported that the tutorial made a more important contribution to their learning (P < .05), and the course objectives were better stated (P = .038) and better met (P = .007). Overall satisfaction with the course also improved significantly (P = .006). Part I gastrointestinal system mean scores of the USMLE showed a statistically significant increase in 2005 compared with 2001 or 2002.
Conclusions: The tutor as a discussion leader who questions, summarizes, and uses schematics to illustrate concepts had a significant and positive impact on learning in tutorials, achieving course objectives, improving overall course satisfaction, and increasing a standardized national exam’s mean score.
Dr. Shields is associate professor of medicine, Division of Gastroenterology, Beth Israel Deaconess Medical Center, Harvard Medical School, Boston, Massachusetts, and director of the gastrointestinal pathophysiology course for second-year medical students, Harvard Medical School, Boston, Massachusetts.
Dr. Guss is a resident in orthopedic surgery, Massachusetts General Hospital, Boston, Massachusetts. At the time of this research, he was a medical student at Harvard Medical School, Boston, Massachusetts.
Dr. Somers is attending gastroenterologist, Concord Gastroenterology, Concord, New Hampshire. At the time of this research, he was a teaching fellow in gastroenterology at Beth Israel Deaconess Medical Center, Boston, Massachusetts.
Dr. Kerfoot is assistant professor of surgery, urology, Veterans Administration Boston Health Care System, Jamaica Plain, Massachusetts, and Harvard Medical School, Boston, Massachusetts.
Dr. Mandell is a lecturer in public policy, John F. Kennedy School of Government, Harvard University, Cambridge, Massachusetts.
Dr. Travassos is attending gastroenterologist, Digestive Health Specialists, Lowell, Massachusetts. At the time of this research, he was a teaching fellow in gastroenterology at Beth Israel Deaconess Medical Center, Boston, Massachusetts.
Dr. Ullman is attending gastroenterologist, Emerson Hospital, Concord, Massachusetts. At the time of this research, she was a teaching fellow in gastroenterology at Beth Israel Deaconess Medical Center, Boston, Massachusetts.
Dr. Maroo is instructor in medicine, Division of Gastroenterology, Cleveland Clinic Foundation and Cleveland Clinic Lerner College of Medicine of Case Western Reserve University, Cleveland, Ohio. At the time of this research, she was a teaching fellow in gastroenterology at Beth Israel Deaconess Medical Center, Boston, Massachusetts.
Dr. Honan is senior lecturer on education, Harvard Graduate School of Education, Harvard University, Cambridge, Massachusetts.
Dr. Raymond is assistant clinical professor of psychiatry, Harvard Medical School, Boston, Massachusetts.
Dr. Goldberg is assistant professor of medicine, Division of Gastroenterology, University of Maryland Medical School, Baltimore, Maryland. At the time of this research, he was a fellow in gastroenterology at Beth Israel Deaconess Medical Center, Boston, Massachusetts.
Dr. Leffler is a fellow in gastroenterology, Department of Medicine, Beth Israel Deaconess Medical Center and Harvard Medical School, Boston, Massachusetts. At the time of this research, he was a teaching fellow in gastroenterology at Beth Israel Deaconess Medical Center, Boston, Massachusetts.
Ms. Hayward is media specialist, Beth Israel Deaconess Medical Center, Boston, Massachusetts.
Dr. Pelletier is statistician and senior project manager, Program in Medical Education, Harvard Medical School, Boston, Massachusetts.
Dr. Carbo is instructor in Medicine, Division of General Medicine, Beth Israel Deaconess Medical Center and Harvard Medical School, Boston, Massachusetts.
Dr. Fishman is assistant professor of pediatrics, Children’s Hospital of Boston and Harvard Medical School, Boston, Massachusetts.
Dr. Nath is assistant professor of medicine, Division of Gastroenterology, Massachusetts General Hospital and Harvard Medical School, Boston, Massachusetts.
Ms. Cohn is course manager, Program in Medical Education, Harvard Medical School, Boston, Massachusetts.
Dr. Hafler is dean for educational development, Tufts University Medical School, Boston, Massachusetts. When this research was done and the article was written, she was associate professor of pediatrics at the Children’s Hospital of Boston, Boston, Massachusetts, and director of faculty development at Harvard Medical School, Boston, Massachusetts.
Please see the end of this article for information about the authors.
Correspondence should be addressed to Dr. Shields, Beth Israel Deaconess Medical Center, Division of Gastroenterology, Rabb 420, 330 Brookline Avenue, Boston, MA 02215; telephone: (617) 667-2109; fax: (617) 667-5826; e-mail: (email@example.com).
Graduate schools of business, education, and government have focused on the importance of formalized training for faculty in their role as discussion leaders, who are experts in the area under discussion, and who question, listen, respond, summarize, and give out takeaways to a group of approximately 60 students in a classroom.1–5 In contrast, medical school faculty have debated the advantages and disadvantages of the expert tutor and have concluded that optimal student-generated learning occurs with nonexpert facilitators who actively listen but do not direct small tutorial groups of seven to nine students.6,7
PBL tutorials have been used as the cornerstone of Harvard Medical School (HMS)’s second-year PBL learning curriculum. However, informal observations by the course director (H.S.), tutors, and the director of faculty development (J.P.H.) in the weekly tutor meetings throughout the 2002 course indicated that the gastrointestinal (GI) pathophysiology tutorials were not consistently addressing explanations of important pathophysiologic principles or achieving adequate coverage of the course objectives. The course director made the decision to shift the faculty role from a facilitator to an expert discussion leader after visiting Harvard Business School (HBS) in the summer and fall of 2002 and observing expert discussion leaders using case-based learning and conferring with the director of faculty development (J.P.H.) and the gastrointestinal pathophysiology advisory committee, who endorsed the change. To make the change, selected teaching strategies from the HBS case-based teaching method were integrated into a brief course-specific tutor training program.2–5 Once the decision was made to change the role of the tutor, the challenge was to devise a program that would give tutors sufficient skills in a short period of time. Tutors were trained to foster student direction of the tutorials and to be discussion leaders by learning to use questions to encourage discussion and understanding, and to summarize using a closure schematic with takeaways.1,4,5 The majority of tutors were recruited from the field of GI medicine and were considered experts on that topic.
Evaluation of the changes in teaching that occurred as a result of this shift in roles included assessment of the student and faculty perception of the tutorials. The educational impact of the tutorial was also studied by analyzing changes in the means and standard deviations of students’ scores on Part 1 and Part 2 of the USMLE from 2001–2002 (preintervention) to 2003–2005 (postintervention). Approval for this research was obtained from HMS’s institutional review board in 2003, 2004, and 2006.
Design and chronology of changes in faculty development sessions
The GI pathophysiology course is three weeks in length, with the PBL tutorials taking place three times a week for 90 minutes each session. Seven to nine students are in each tutorial with one tutor. Exposure to case-based teaching methods at HBS (H.S.) in the fall of 2002 led the newly formed gastrointestinal advisory committee to select three key teaching strategies: (1) asking questions to guide the discussion,4 (2) summarizing the major points raised during the discussion,5 and (3) using a visual closure schematic as a takeaway concept map.1 In addition, two new structural elements were created, namely, handing out pertinent case pages before the tutorial (2003), and an announcement of daily tutorial objectives (2004).
2003 faculty development sessions
In 2003, 24 tutors were trained using a pilot faculty development program that was designed and implemented with two, two-hour sessions, one devoted to how to ask a variety of types of questions, and the other devoted to how to effectively summarize the tutorial discussion. Weekly faculty meetings were held for all tutors.
2004 faculty development sessions
In 2004, the faculty development program for 24 tutors changed to three two-hour evening sessions before the course, and a weekly faculty meeting during the course. The first of the three sessions dealt with the art of asking questions. Specific goals were to have faculty learn to pose, listen, and respond to questions, as well as to expose them to the use of various types of questions, including open-ended, informational, and diagnostic ones; questions that challenge; and questions that encourage extension of the topic, help establish the priority of the diagnostic tests and therapeutic modalities being discussed, and help predict the possible outcomes of the interventions made.4 The training strategy used was a simulated tutorial of a patient with severe esophageal reflux.
The second of the 2004 sessions focused on managing tutorial dynamics. The session goals were to motivate the quiet student to participate more fully, moderate the participation of the dominant student, and foster an environment that encourages participation. The training strategy used videos (from the Derek Bok Center at Harvard University) illustrating the behaviors of the quiet and dominant student, with tutor commentary.8–9 The tutors viewed each video vignette to broaden the range and repertoire of possible behavior and actions by the tutor in response to challenging situations,8–9 such as nodding and/or sending an encouraging look toward the quiet student to bring him or her into the discussion, and/or speaking to these students after the first or second tutorial regarding their lack of participation and working with them to determine what topic each would prepare to discuss at the next tutorial. For dominant students, tutors suggested averting their gaze and body away from those students and refraining from encouraging them, and/or passing over their raised hands in response to a question. Tutors also recommended speaking to the dominant student outside of class and asking him/her to ration comments to give increased air time to fellow students for a more equitable discussion.
The third and last of the 2004 faculty development two-hour evening sessions focused on developing expert knowledge, learning to summarize, and using closure schematics1,5 to explain the objectives of the session and clarify concepts. The summaries were short (three to five minutes), emphasized important concepts, and clarified points of confusion. Each tutor and student were given an 11 × 17-inch color handout detailing the mechanisms that were expected to be discussed during the tutorial. The schematic could also be projected onto a plasma screen in each tutorial room.10 Tutors were given pertinent chapter and journal articles, and tutorial guides included a review of the essential pathophysiologic concepts in the cases. The training strategy of the session was an interactive discussion of the first week’s tutorial case content.
2005 faculty development sessions
In 2005, 22 tutors were trained. The two faculty sessions on the use of questions and management of the quiet and dominant student were combined into one two-hour session. The second two-hour session focused on summaries and schematics. A third one-hour evening session, primarily for new tutors, was added to go over the mechanics and logistics of the tutorial, including seating arrangements, available written resources, Web resources, visual aids, use of the plasma screen, methods of giving feedback to students, evaluating student contributions, and creating a comfortable atmosphere for discussion.
Essential elements of the methodology to train discussion leaders
Our methodology for training discussion leaders consists of 10 distinct steps. The course director and teaching fellow clearly define daily objectives for each PBL case and choose these objectives carefully to reflect the important concepts and mechanisms that the tutors’ future students are expected to learn during the session for the final examination and for the national board examinations.
1. The course director, teaching fellow, and media specialist design the schematic, which is a visual takeaway for each session, to reiterate the daily objectives. The schematic visual images—including endoscopic, radiologic, and pathologic material, as well as diagrams, figures, and tables—are chosen from textbooks, clinical files, journal articles, or artist’s renditions. Copyright clearance is obtained for each diagram, figure, or table taken from a textbook or journal article, using a commercial copyright clearance company.
2. The course director and teaching fellow give tutors the background reference material for each visual image, including the book chapter or journal article from which it was taken.
3. In addition to the schematics being available online, the course director and course manager give each tutor copies of the schematics (11 × 17 inches, color).
4. Course faculty introduce tutors to the wide variety of types of questions in a mock tutorial, with half of the tutor group discussing part of the pathophysiology of the first tutorial case and the other half of the group discussing afterwards their observations. All tutors, experienced and inexperienced, are encouraged to imagine the possible directions in which the student discussion may go and to write down ahead of time a variety of questions on each tutorial page to steer the discussion, if necessary, toward coverage of the daily objectives, concepts, and mechanisms.
5. At the second faculty development session, the course director gives each tutor the cases, daily objectives, and case guides for the whole course and takes tutors through the entire pathophysiology of the first case, in addition to giving examples of possible model summaries using schematics for each day of the three-day tutorial case. Tutors practice their five-minute summaries before each tutorial to make sure that the summary stays within the five-minute time limit.
6. At the third faculty development session, the course director and teaching fellow take new tutors to a tutorial room to go over the mechanics of the tutorial session. Tutors are reminded to hand out, at the end of each tutorial, the case pages for the subsequent tutorial after their summary using the schematic. The case objectives are sent as an e-mail to each tutor on a daily basis by the course manager. Tutors are expected to do background reading before the tutorial to cover the objectives, using the reading assignment list for that week.
7. At weekly faculty meetings, the course director and teaching fellow review and highlight the pathophysiology content, objectives, and schematics for the next case in a blackboard presentation to properly prepare every tutor for the upcoming sessions.
8. The course faculty encourage tutors to meet with quiet students and dominant students, as well as those who are not participating or not prepared, as soon as possible, preferably at the end of the tutorial in which the problem is noted or at the beginning of the following tutorial.
9. Each tutor meets with each student for 10 to 15 minutes during the second week of the three-week course to give face-to-face feedback on the student’s preparation and participation in tutorial and to hear feedback about his or her own performance as a tutor from the student.
Data collection and analysis
Faculty background and composition.
Tutors were identified in regard to their prior tutoring experience and their medical or surgical academic level. These two factors were compared with each tutor’s student evaluation score.
Students’ views of their tutors as discussion leaders.
From 2003 to 2005, course faculty used HMS’s MyCourses course-evaluation Web site to administer a Likert-type questionnaire to collect student data anonymously from 171 students (2003), 167 students (2004), and 170 students (2005) on their perceptions of the tutor and his or her impact on their learning, examining specifically their perceptions of the use of the tutor’s summaries, the use of schematic concept maps, and whether the tutor encouraged student direction of tutorials. Also, students were asked to rate the tutorials in terms of their contribution to their learning.
2002–2005 tutors’ evaluations of the course.
At the end of the course from 2002 (preintervention) to 2003, 2004, and 2005 (postintervention), tutors anonymously answered evaluation questions that focused on (1) how clear the objectives for the course were (beginning in 2003), (2) how well the course objectives were met, (3) the logistic and organizational administration of the course, and (4) the overall course rating.
2001–2005 USMLE Part I scores.
The means and standard deviations of medical students’ USMLE Part I scores for first-time test takers for the GI system were compared for the years 2001 and 2002 (tutor as facilitator) and for 2003, 2004, and 2005 (tutor as discussion leader) to determine whether the new faculty development program and tutor role had an impact on the standardized test scores and standard deviations. Approval for these studies of the analysis of USMLE Part I data was obtained from HMS’s institutional review board in 2006.
ANOVA analysis was used when appropriate. However, given the unequal variances between groups, statistical procedures that do not require that all population variances be equal were used, namely, the Welch and Brown–Forsythe tests. The results of these tests agreed with the ANOVA results. In addition, the Bonferroni procedure as well as the Temhane T2 statistic and the Games–Howell tests were used to pinpoint differences in the group means. Statistical significance was set at the confidence level of .05. A t test was used for comparison of the means and standard deviations of the USMLE Part I scores.
The tutors’ prior tutoring experience and academic levels indicated that a wide range of experience and professional levels of training were present in each year’s tutor group from 2003 to 2005. Tutors from all academic levels performed well as tutors, according to students’ responses to the final course-evaluation question, “How would you rate your tutor overall?” using a five-point Likert attitudinal scale, where 1 is excellent and 5 is poor. Approximately 75% of the tutors were gastroenterology faculty or gastroenterology fellows for each of the three years from 2003 to 2005, whereas the remaining 25% were attendings or residents in other medical or surgical specialties. Students’ average overall ratings of the 70 tutors from 2003 (24), 2004 (24), and 2005 (22) were 1.46 (n = 26) for gastroenterology attendings, 1.33 (n = 26) for gastroenterology fellows, 1.14 (n = 3) for the medicine attending, 1.28 (n = 13) for medicine residents, and 1.67 (n = 2) for the surgeons. The overall numbers in the groups are too small to test for statistical significance. Half of the tutors in the 2004 and 2005 GI pathophysiology courses had experience as discussion leaders, in contrast to 9 of the 24 tutors in the 2003 academic year, whose prior experience had been as tutors in the traditional HMS PBL model (facilitators).
Students’ evaluations of the tutor as a discussion leader from 2003 to 2005 used a five-point Likert scale, with 1 being excellent and 5 being poor. The data showed a positive trend for the tutor as discussion leader for the three-year period. Significant improvement over time is indicated in the tutors’ ability to encourage the student direction of tutorials (2003: n = 139, score = 1.81; 2004: n = 143, score = 1.62; 2005: n = 129, score = 1.51) (Brown–Forsythe test and Temhane statistic, P < .05). Students consistently rated the summaries and schematics from 2003 to 2005 as particularly helpful in clarifying concepts (no significant difference noted using ANOVA analysis for 2003–2005). The tutor as discussion leader was associated with a significantly increased contribution of the tutorial to the students’ learning in 2005 (n = 129, score = 1.31) compared with 2003 (n = 139, score = 1.53) (Welch and Brown–Forsythe tests and Temhane statistic, P < .05).
Student evaluations of the GI pathophysiology course using the same five-point Likert scale were also turned in by the students at the end of the course.
* Students perceived that the course objectives were more clearly stated (ANOVA .032, Bonferroni P = .038) from 2003 (first year postintervention; n = 145, score = 1.41) to 2005 (third year postintervention; n = 129, score = 1.21).
* In addition, the course objectives were better achieved over time (Bonferroni P = .007, Temhane statistic P = .014) when comparing 2002 (preintervention; n = 52, score = 1.54) to 2005 (third year postintervention; n = 129, score = 1.21) (ANOVA .008) and 2003 (first year postintervention; n = 145, score = 1.45) to 2005 (third year postintervention; n = 129, score = 1.21).
* Overall satisfaction with the course also improved significantly (ANOVA .003) from 2002 (preintervention; n = 52, score = 1.57) to 2005 (third year postintervention; n = 129, score = 1.22).
* The course organization and the students’ perception of their learning of the basic science of gastroenterology were consistently strong and did not significantly change during the four-year period (Data for the course organization are as follows: 2002: n = 52, score = 1.54; 2003: n = 145, score = 1.28; 2004: n = 143, score = 1.24; 2005: n = 129, score = 1.20).
The GI system USMLE Part I mean subscores for our medical school students remained above the calculated United States and Canadian mean throughout the five-year period from 2001 to 2005. This period focused on the two years before (2001–2002) (preintervention) and the three years after (2003–2005) (postintervention) the tutor as discussion leader was introduced. A comparison was made between the mean GI system scores for Part I from the preintervention period, years 2001 and 2002, and those from the postintervention period, years 2003, 2004, and 2005. The mean GI system score was significantly higher (229) in 2005, the third year postintervention, compared with either of the preintervention years of 2001 (225) (P = .047) or 2002 (224) (P = .024; see Figure 1). Although the mean scores for 2003 (226) and 2004 (227) were higher than those in 2001 and 2002, these differences were not statistically significant.
The standard deviation of the aggregate subscores for the GI system revealed a smaller number of students in the postintervention period (2003–2005) with scores below the United States and Canadian means compared with the preintervention period (2001–2002; see Figure 2). A comparison was performed with each of the other 18 preclinical HMS courses’ aggregate subscores on the USMLE Part I to see whether other preclinical courses also had improved scores and smaller standard deviations during the period 2003 to 2005. Only 2 of 18 other courses, one other second-year preclinical course and one first-year preclinical course, were noted to have a definite and consistent improvement in their mean scores and a smaller standard deviation during the 2003–2005 period (data not shown).
To summarize: the faculty development program described here applied HBS’s case-based teaching strategies, generally used for large groups of 60 or more students to small tutorial groups of seven to nine students.1–5 The tutor was taught strategies to be an expert discussion leader who uses questions, summaries, and schematics. The results were positive for the educational contribution of the tutorials, the achievement of the course objectives, and the small but significant improvement of the students’ mean scores and standard deviations on the GI system part of the USMLE Part I.
The tutor as discussion leader
West11 has noted that one deficiency of PBL in teaching science is that the weaker students may not become engaged in the learning process as well as they might in the discussion format. The experimental introduction of the tutor’s guiding questions and use of summary and closure schematics was an attempt to bridge that perceived deficiency and to provide an opportunity for each student to learn the significant scientific concepts or major objectives for that tutorial session.5,11 A recent study comparing PBL with case-based learning for the first-, second-, and third-year clinical doctoring course at University of California School of Medicine at Davis and University of California School of Medicine at Los Angeles found that both students and faculty who filled out a questionnaire overwhelmingly preferred case-based learning over PBL.12 The case-based, guided inquiry approach used questions to steer the discussion of preidentified learning issues and assigned preparatory readings. The authors of this study place an emphasis on using guiding questions to promote discussion, as does our faculty development program for discussion leaders.12
There has been considerable discussion in the literature regarding the optimum level of the expert tutor’s participation.6,7 One concern has been that the expert tutor who has prepared extensively would give mini-lectures and turn the tutorial into a recitation of principles and facts with decreased student-generated exploration, discovery, and participation.6 However, our consistently improving evaluations of the expert tutor do not support this fear. The majority of the GI tutors were already expert in general gastroenterology in that 75% were gastroenterology faculty or fellows in gastroenterology (adult or pediatric). In preparation for conducting the tutorial, it was expected that all tutors would become experts in the specific areas of GI and hepatobiliary pathophysiology under discussion.2–5,11
The faculty development program used the simulated tutorial method to give tutors a live demonstration of a tutor asking a variety of questions to elucidate the important physiology and anatomy concepts.4 Tutors’ discussion of the videos of quiet and dominant students elicited an array of responses to use as a repertoire for their own tutorials should similar challenges arise.8–9 The case-based method at HBS focuses its discussion leadership on expertise, which helps in asking stimulating questions and steering the discussion.1–3 The requirement for an adequate number of expert faculty is clearly an obstacle to the use of small-group PBL tutorials. However, our tutorial evaluation data show that medical residents, internal medical faculty, and surgeons who were tutors had tutorial performance evaluations that were similar in quality to the evaluations of the tutors who were gastroenterologists or fellows in gastroenterology. Further, new tutors were able to become proficient with a limited number of training sessions, indicating that sufficient expertise can be learned by a wide spectrum of interested faculty.
Training sessions for the tutor as discussion leader encouraged each tutor to speak no more or less than any other member of his or her seven- to nine-member tutorial group, and to provide the impetus for learning by asking guiding questions and providing direction toward essential scientific principles. The statistical analyses indicate that the summaries increased the educational impact of the tutorial, consistent with Greenwald’s5 view that in the teaching of technical material, there is an important role for end-of-tutorial summaries. Garvin1 notes that the business school teaching model frequently uses an end-of-class summary with clear takeaways. Similarly, the summary plus the closure schematic in the GI pathophysiology course are used to ground students in the reality of what they should have understood from the discussion. Both also assist in directing students’ reading to have better comprehension of the mechanisms depicted on the schematics.
The new tutor’s role as discussion leader increased the educational impact for the tutorials in the GI pathophysiology course and enhanced the meeting of course objectives. The overall course ratings also improved as tutors became more skilled in serving as discussion leaders. Student evaluations confirmed that students appreciated the tutor as a discussion leader and felt that students were adequately able to direct the tutorial. Student approval ratings for the tutorial as a positive contribution to learning increased significantly from 2003 to 2005. During that three-year period in which the tutor’s role was changed, the pathophysiology course otherwise remained essentially the same, suggesting that the improved outcomes are likely the result of the changed tutor role. However, because we lack a control group of students, we are unable to prove conclusively that our interventions caused the better (lower) Likert attitudinal scale scores in 2005 (postintervention) evaluations compared with those in 2002 (preintervention).
The USMLE Part I GI system subscores at our medical school showed a small but significant increase in the mean test scores in 2005, the third year of the postintervention period, compared with either 2001 or 2002, the two preintervention years. In addition, fewer students fell below the United States and Canadian mean for scores on that test (national reference group of first-time test takers from accredited medical schools) in 2005 compared with 2001 and 2002. In the first year of HMS, the physiology course devotes one week to gastrointestinal physiology. Therefore, the second-year course that was studied here is the second time that students are exposed to gastrointestinal physiologic concepts and mechanisms.
A comparison of our GI system aggregate subscore and standard deviation with 18 other preclinical courses’ subscores and standard deviations at our medical school over the same time period indicated that only two other preclinical courses, one first-year and one second-year course, exhibited the same type of consistent improvement for the three-year period in both the aggregate subscore and the standard deviation.
A recent article by Hoffman et al13 from the University of Missouri–Columbia School of Medicine notes that a PBL curriculum raised their students’ USMLE Part I and Part II scores compared with scores achieved via a traditional curriculum. Prior evaluations of PBL have generally noted no improvement in performance on the USMLE when the PBL method of teaching was used.7,14,15 Because PBL curricula frequently require a larger investment of faculty time and effort, the absence of any improvement in standardized test scores has been a disappointment and has led some educators to question the wisdom of the investment in this labor-intensive methodology.15
Our results may indicate that the methodology that uses the tutor as discussion leader—with its emphasis on a more structured approach to discussing pathophysiologic concepts and mechanisms by means of guiding questions, summaries containing clear takeaways,1 and closure schematics with memorable visual images—is helpful in assisting medical students to improve their mastery of GI system knowledge as measured by a standardized national test, the USMLE Part I. However, our data need to be interpreted in light of the following facts about the USMLE Part I test: (1) the number of items relating to the GI system on the Part I test is relatively small (approximately 10–20 items), (2) these test items are not selected to be a representative sample of knowledge related to the GI system, (3) examinations specifically intended to check overall GI system knowledge may be more accurate in determining the educational benefits of our method, and (4) over the course of four to five years, there may be shifts in the format and content of the USMLE examinations that could affect the scores obtained and therefore our analyses.
Lessons learned, barriers faced
We learned several lessons during the three years of this new faculty development program.
* First, the faculty development sessions and weekly faculty meetings must be designed so that they are informative, lively, and focused on holding the attention of experienced as well as new tutors. We found that a combination of audience participation, instructional videos, excellent visual images, and expert panelists were very helpful in keeping tutors engaged during the faculty development sessions.
* Second, the case objectives are used as anchors to secure the learning in each session and are reiterated and reviewed in the summary and schematic. The case objectives should take into account the examination content expectations of the National Board of Medical Examiners.
* Third, it is important to find a clear and memorable depiction of a concept or mechanism.
* Fourth, sufficient background material must be given to the tutor ahead of time to bring each tutor up to a reasonable level of expertise. This material should include careful discussions of the figures on the schematic takeaway sheets.
* Fifth, tutors should encourage student direction of the tutorial even while they are acting as discussion leaders and guides.
* Sixth, the summaries should not exceed five minutes.
We faced several barriers to achieving our goal of a better GI pathophysiology education. Initially, finding the ideal visual depiction of a particular mechanism or concept is time consuming. After a schematic is designed, the cost for both the copyright clearance for each diagram and for the production of the color schematics is considerable. Although black and white photocopies were used in the first year, 2003, the students felt strongly that the online color version greatly improved their understanding of the concepts depicted. During the tutorial sessions, some tutors found speaking only as much as each student did and sticking to a five-minute summary difficult. Tutors noted that they needed to prepare for several hours before each tutorial session by doing the background reading to familiarize themselves with the area and the details of the mechanisms and concepts. Finally, this method of discussion leadership was sometimes seen at odds with the idea that the students should discover and explore the case during tutorial, in PBL cases. Instead, the emphasis in our method is placed on obtaining the rich and interesting discussion that occurs when students are prepared to participate in the discussion of pathophysiologic mechanisms and concepts during tutorial.
A promising tool
The discussion leader methodology is applicable to all preclinical courses. Our findings suggest that this approach could prove to be an advantageous educational tool for teaching preclinical core content to medical students in a PBL curriculum.
The authors are grateful to all the gastrointestinal pathophysiology tutors who taught during the years 2003 to 2005 and who were the initial discussion leaders; to members of the gastrointestinal pathophysiology advisory committee at HMS who have been instrumental in making valuable suggestions regarding the design of the course and implementation of the faculty development program, specifically Jean McNamara, Sheila Salamone, Douglas Krakower MD, Susan Liu Hoki, MS, MD, and Anne Lyons. In addition, the authors wish to acknowledge the contributions of the following persons: David Garvin, PhD, of HBS, for his keynote address at the HMS’s Experienced Tutor Retreat in June 2002 on the similarities and differences among the case-based methods used at the business, law, and medical schools at Harvard University, for a personal interview regarding discussion-based teaching (H.S.), and for permitting some of the authors (H.S., S.U., S.M., D.L.) to observe his case-based discussion classes at HBS; Lee Warren, PhD, of the Derek Bok Center for Teaching and Learning at Harvard University, for a personal interview on discussion-based teaching (H.S., S.S.), for supplying the teaching videos of the dominant and quiet student, and for teaching the 2003 faculty development session on managing the quiet and dominant student in a small group; Jay Lorsch, PhD, John Gabarro, PhD, and Michael Tushman, PhD, of HBS, for permitting observation of their classes (H.S., S.M., S.U., and D.L.) and for personal interviews with them regarding discussion-based teaching; Jeanne Crowgey, for her advice and encouragement; Joseph Martin, PhD, MD, George Thibault, MD, Jules Dienstag, MD, Nancy Oriol, MD, Daniel Federman, MD, and Mimi Wetzel, PhD, for their encouragement, support, and suggestions; David Cardozo, PhD, of HMS, for important suggestions regarding online course objectives; Terry Galuszka, MS, the registrar at HMS, for giving access to the aggregate USMLE scores; Ed Krupat, PhD, for helpful suggestions regarding data collection and analysis; Elisabeth Peet, MA, for helping in the development of the questionnaires and collection of data; Robert Heroux, for helping in producing the questionnaires and editing the manuscript; and Susan Rakich and Andrea Lenco, for aiding in the manuscript preparation and organization.
Support for the research project reported here was provided by two grants awarded to Dr. Shields by the Academy at HMS’s Competitive Grant funding for Innovative Teaching Projects for the years 2003–2004 and 2004–2005; and the Leo E. and Natalie Wolf Fund, which provided funds for the production and distribution of the schematics to the tutors and students on a yearly basis from 2003 to 2005.
Information from this report was presented earlier at the Association of American Medical Colleges (AAMC) annual meeting’s Research in Medical Education poster session in 2003 in Washington, DC, and in 2004 in Boston, Massachusetts; at Medical Education Day at Harvard Medical School for a plenary session in 2004, and in poster presentations in 2003 and 2005; and at the symposium sponsored by the American Gastroenterological Association’s Education Committee and entitled “Essential Elements of a Successful Gastrointestinal Pathophysiology Course,” in Chicago, Illinois, 2005.