Designing, Conducting, and Evaluating Journal Clubs in Orthopaedic Surgery

Abstract

The first record of a journal club was that founded in 1875 by Sir William Osler at McGill University for the purchase and distribution of periodicals to which he could not afford to subscribe as an individual. Evidence-based medicine is becoming an accepted educational paradigm in medical education at various levels. An analysis of the literature related to journal clubs in residency programs in specialties other than orthopaedic surgery reveals that the three most common goals were to teach critical appraisal skills (67%), to have an impact on clinical practice (59%), and to keep up with the current literature (56%). The implementation of the structured article review checklist has been found to increase resident satisfaction and improves the perceived educational value of the journal club without increasing resident workload or decreasing attendance at the conference. Periodic evaluation of the conference and the institution of appropriate changes ensures that the journal club remains a valuable and successful part of the training program.

As reported by Sidorov, ²⁴ the first record of a journal club was that founded in 1875 by Sir William Osler at McGill University for the purchase and distribution of periodicals to which he could not afford to subscribe as an individual. Moving to John Hopkins University in 1889, Osler established the Book and Journal Club, whose members met to discuss the latest published medical research and recommend new books for the library. Journal clubs in training programs devoted to various medical specialties became more common in the 1920s, but reports concerning their format were sparse. There have been an increasing number of publications discussing the prevalence, format, and (occasionally) effectiveness of journal clubs in various medical specialties. ^{1,8,11,14,20,22,23,24,26} We will examine the purposes for journal clubs in training programs, and provide options to improve their effectiveness. Finally, various examples of journal club formats reported in the literature will be presented. It is our hope that this article will provide the organizers of journal clubs with additional tools that will increase the likelihood of developing successful journal clubs. It remains, however, the task of each residency program to individualize and optimize the journal club format for its particular program.

The Purpose of a Journal Club

Evidence-based medicine is becoming an accepted educational paradigm in medical education at various levels. ^7,15 It focuses on identifying the best evidence for medical decision-making and applying that evidence to patient care. The past several years have seen the widespread inclusion of evidence-based medicine in internal medicine training programs in the United States. More recently, surgical training programs have adopted similar curricula to include aspects of evidence-based surgery during residency training. However, evidence-based surgery often is taught as an independent topic and is integrated poorly into the clinical teaching of surgical trainees. Evidence-based surgery education ideally is suited for a journal club format, focusing on question development, searching, and critical appraisal.

The relative importance of teaching critical appraisal skills to surgical residents has been the focus of several reports. ^5,6,8,22 In a survey of orthopaedic residency program directors, training residents in the critical evaluation of scientific articles was the most important goal of journal clubs in 67% of programs responding. ⁸ Instilling residents with the routine habit of reading scientific journals also was listed as important in 30% of programs, whereas 14% rated learning about current research in orthopaedics as a high priority. In a survey of 278 general surgery program directors (response rate 80%), more than 50% indicated that their journal club was important or very important to their training program, with learning literature review skills and providing training in research education as the top two purposes cited. ⁶ Although some think that journal clubs with high attendance and longevity often are characterized by mandatory attendance and availability of food, ¹ others downplay the role of journal clubs as alternate means for residents to learn orthopaedics, develop camaraderie, and interact socially with faculty. ⁸

It also seems that transmission of clinical information may be a much more important goal of journal clubs in fellowship programs in orthopaedics than it is in residency programs. In a survey of 57 hand surgery fellowship directors, the primary purpose of journal clubs was to familiarize fellows and faculty with the current literature, ²² and almost all fellowship directors thought that the journal club was an important part of the fellowship training.

The literature related to journal clubs in residency programs in specialties other than orthopaedic surgery reveals that the major stated goal of journal clubs is to teach critical appraisal skills to residents. ^1,4,24,26 The only exceptions in the literature are surveys of program directors in physical medicine and rehabilitation and family medicine programs, which revealed that the most common goal of journal clubs was to disseminate information from the current literature. ^11,23

Considerations in Organizing a Journal Club

Although most training programs sponsor journal clubs, they are not equally successful in meeting educational goals. Success for a journal club has been defined as meeting educational objectives (if objectives are stated) and continually promoting and maintaining resident interest in the conference. ²⁴ Designing a format to make journal clubs stimulating, interesting, and educational for its members has proven to be a greater challenge than residency programs might have imagined. The following section discusses some of the key considerations in the design, conductance, and evaluation of a journal club (Table 1).

TABLE 1: Considerations in Preparing a Journal Club

Leadership of the Club

Perhaps the most important step in developing a successful journal club is to have an interested person or small group committed to the organization, conductance, and evaluation of the club. Having one leader for a journal club correlated significantly with effectiveness in one study. ¹¹ Although having a leader who is a faculty member is important, other studies have indicated that having active involvement of the residents in planning and operating a journal club is associated with longevity and success of the club. ^14,20,24

Defining Goals of the Club

Although one goal of almost all journal clubs in training programs is to keep residents up to date on the literature, the importance of this goal varies between programs. In a survey of internal medicine programs in New York City, the three most common goals were to teach critical appraisal skills (67%), to have an impact on clinical practice (59%), and to keep up with the current literature (56%). ²⁰ In a survey of family medicine programs, the most important goal was to keep the residents abreast of the current literature. ¹¹ In pediatric programs, 77% of chief residents responded that learning critical appraisal and keeping abreast of the literature were equally important. ²⁶ In a survey of orthopaedic residency program directors, teaching residents critical evaluation of scientific articles was the most important goal of journal clubs and was listed as the priority for 67% of programs responding. ⁸

Articulating and communicating the goals of any educational conference or experience is an important step. Although each of the studies cited compiled educational goals for journal clubs by surveying program directors or residents, none asked whether the goals had been formalized in writing in each program surveyed. In a survey of emergency medicine program directors, however, this question was asked, with the finding that 42% of programs did not have formal written learning objectives for journal club. ¹⁴ The establishment of formal, written educational objectives for a journal club should be done in each residency program. The content and number of articles discussed during the conference session, and what journal(s) they are drawn from, will be guided by these objectives, as will decisions regarding the format and setting of the journal club. Written educational objectives also will form a basis for assessing the effectiveness of the conference.

Frequency, Setting, Timing, and Attendance

In 78% of orthopaedic training programs, journal clubs meet once per month. ¹ Monthly journal clubs also are conducted in 86% of emergency medicine residency programs, ¹⁴ in 81% of general surgical residency programs in the southeast region, ⁵ in 61% of physical medicine and rehabilitation residency programs, ²³ and in 43% of internal medicine programs. ²⁴ Conducting journal club more than once per month may be difficult, particularly if a program intends to have residents review numerous articles for each journal club or if a detailed, structured, critical appraisal of journal club articles is expected of residents. Journal clubs held as frequently as weekly, however, have been reported to be successful in some internal medicine and pediatric programs. ^1,24,26

Although approximately 50% of journal clubs in orthopaedic residency programs meet in the departmental office or in the hospital, 49% are held outside the department, at a restaurant or the home of a faculty member. ⁸ Sixty-eight percent of orthopaedic journal clubs meet in the evening, with the remainder meeting at the beginning of the workday. ⁸ In contrast, journal clubs in most internal medicine, family medicine, and pediatric residency programs meet in the middle of the day in the departmental office or in the hospital. The daily clinical demands and schedule in a surgical training program lend themselves less well to midday meetings and conferences than do those in nonsurgical training programs.

Several authors reported that consistently high attendance by residents at journal club sessions was an important factor that correlated with the club’s success. ^11,14,24 Scheduling the conference at a convenient time for residents is important, but it may not be enough to ensure consistently high attendance. The residency training program should stress the importance of all educational conferences (including journal club) and should excuse residents from all but emergency patient care responsibilities to attend scheduled educational activities. Some programs have gone further and taken the approach of making attendance by residents at journal club mandatory. Resident attendance at journal club has been reported to be mandatory in 58% of internal medicine programs, ²⁴ in 57% of family practice programs, ¹¹ and in 50% of pediatric programs. ²⁵ Deeper analysis of these data indicates that smaller residency programs are more likely to mandate resident attendance at journal club than are larger residency programs. Other features common to journal clubs that have achieved high resident attendance include: resident involvement in selection of articles for the journal club, serving of food and drink at journal club, consistent faculty attendance at journal club sessions, and the presence of formal teaching of biostatistics and clinical epidemiology as part of the journal club. ²⁴ Eighty-seven percent of journal clubs in orthopaedic training programs provide some sort of food. ⁸ There is an inverse association between high resident attendance and review of only original research articles at journal club sessions. ²⁴ Studies are conflicting on whether a faculty member or resident moderating the conference has a positive impact on attendance. It would seem, from these studies, that having a skilled moderator, whether faculty or resident, is important to the value of and attendance at these conferences.

Generating Resident Participation in Journal Club

Unlike most other medical teaching conferences in a residency program, the journal club requires audience participation to provide the best education to its attendees. It is the constant exchange of ideas, opinions, and interactions among those in attendance that helps optimize the educational value of the conference. Conducting journal club in an environment that facilitates relaxation, conversation, and eye contact among participants is important. Although conducting journal club at a restaurant or pub may be relaxing, conversation may be inhibited if there is music playing, if there is significant distracting ambient noise, or if the space does not allow seating in a circle to maximize eye contact and allow all attendees to participate equally. Seating participants in a circle seems to be the most effective for encouraging participation. Surveys also have shown that strong advocacy for the importance of journal club by the department chairperson or residency program director is instrumental to generating resident participation in the conference. ^8,24,26 It has been stated that groups of 10 to 12 members are optimal for discussion; it may be necessary to form separate groups of eight to 12 members in a large journal club to maximize the ability to use a discussion format. Selection of articles for journal club that are of interest to the residents or, better yet, allowing the residents to participate in the selection of articles, also is an important means to generate resident participation.

There remains considerable variability in the number of articles appraised and the duration of journal clubs. ^4,10,22 Overambitious inclusion of articles risks resident and facilitator fatigue. Moreover, attempts to review a larger number of articles will limit opportunities to discuss important issues often raised during the course of the conference. Each program will need to individualize the volume of articles appraised. A reasonable trade-off between too many articles (limiting valuable discussion) and too few articles (limiting the breadth of information) seems to range from three to 10 articles.

The format of the conference also is an important tool to encourage and generate resident participation. Specific examples of conference formats will be discussed later.

Linking Journal Club to the Curriculum

Increasingly, residency program directors state that an important goal of the journal club is to teach critical appraisal techniques to the residents. Programs aiming to teach critical appraisal skills may include some type of formal teaching of these techniques, either as a series of formal lectures or by devoting some journal club sessions to the analysis of articles showing fundamental principles of study design, biostatistics, and epidemiology. Additionally, program directors may desire to use journal club sessions to help educate and evaluate residents in portions of the American Council for Graduate Medical Education core competencies in the areas of medical knowledge, patient care, or practice-based learning and improvement. If these are goals of program directors, we strongly urge them to formalize the goals, curriculum, and evaluation methods used to ensure these are an integral part of the overall residency curriculum that can be documented at the time of a program site visit by the American Council for Graduate Medical Education.

Several authors have reported on the results of designing and implementing a curriculum to teach the fundamental techniques of critical appraisal of the literature using the journal club as a venue. In one report, two approaches to journal clubs were compared in a randomized trial. ¹⁹ In a self-assessment, residents whose conference emphasized epidemiology and biostatistics thought they read with more attention to study design and methodology than their counterparts who participated in traditional journal clubs. Despite the perceptions of residents, objective testing of knowledge in epidemiology, biostatistics, and critical evaluation revealed no significant differences between groups.

Another study compared the acquisition of knowledge in clinical epidemiology and biostatistics between two pediatric programs with monthly journal clubs; the only difference between the two programs was that one featured two introductory session on epidemiologic principles. ¹⁷ Objective pre-testing and post-testing revealed no difference between study groups in acquisition of knowledge in these areas.

A prospective, case-controlled trial of 32 emergency medicine residents exposed residents to either a standard journal club format or a journal club using an evidence-based medicine approach to critical appraisal. ² Evaluation was conducted by having the residents evaluate a factitious article before and after the 1-year trial. Although both groups improved in performance during the study period, there was no significant difference in the amount of improvement between the two groups.

One study reported the outcome of a very structured journal club in an obstetrics and gynecology residency program designed to teach critical evaluation of the literature in an evidence-based environment. ¹⁸ This very rigorous design, including monthly didactic sessions and reviews of selected journal articles, was well-received by residents and faculty and will be continued without major change.

Structured Review Instruments

The use of a structured review instrument, a checklist or form to guide the resident through the critical appraisal of a journal club article, was assessed in one recent study. ⁴ Before and 6 months after the introduction of a structured checklist format for article review, a 5-point Likert scale (1 = worst, 5 = best) was used to assess residents’ satisfaction with the journal club. The implementation of the structured review instrument was found to increase resident satisfaction and improve the perceived educational value of the journal club without increasing resident workload or decreasing attendance at the conference.

The evidence-based medicine website from Oxford University ¹² provides downloadable guidelines and dataforms for critical appraisal of published studies. In addition, guides for appraising the medical literature also are available in the User’s Guide to Medical and Surgical Literature series of articles published in several journals including the Journal of Bone and Joint Surgery, American Volume. ^3,9

Structured review instruments have been applied in numerous orthopaedic training programs; assessments of the outcomes and effectiveness of this format for journal club are ongoing. An example of one structured review instrument in use in two orthopaedic training programs is shown in Appendix 1.

Taken in whole, the literature cited above shows that a curriculum to teach critical appraisal of the medical literature may be an effective adjunct to the journal club and that a formal evaluation of knowledge may be a useful way to document acquisition of knowledge and to assess the impact of the curriculum on that of the overall residency program. The use of a structured review instrument is a logical tool for teaching critical appraisal and seems to be well-received by residents; however, validation of this instrument for teaching critical appraisal skills has yet to be accomplished.

Evaluating the Journal Club

As with all educational activities, the journal club likely will need periodic refinements to keep it stimulating for and valuable to the participants. Therefore, there is a need for periodic evaluation of the journal club as an educational activity. Perhaps the best way to determine whether and what adjustments in the conference are necessary is to periodically ask the conference participants whether they are satisfied with the educational conference. This sort of evaluation also can ask residents for anonymous feedback about strengths, weaknesses, and potential improvements for the journal club conference. One other method of evaluation is to ask residents to self-assess their clinical reading habits and behaviors as a measure of success; improvements in residents’ self-evaluation with time would indicate a successful journal club program. If improvement of critical assessment skills is a goal of the journal club, then an assessment of the residents’ understanding of the principles of study design, epidemiology, and biostatistics could be an important measure of success of the journal club program. Some programs have reported using a pre-test and post-test format to assess acquisition of critical appraisal skills, ¹⁷ whereas others have used the critical evaluation of a factitious standardized article. ² Whatever the evaluation methodology used, the journal club organizer should do the assessment periodically and use the information obtained to improve the educational value of the journal club conference.

Journal Club Formats

A wide variety of formats for journal clubs have been presented in the medical literature. This section will discuss numerous examples of formats published in the literature. The intent is not to rate, rank, or attach a value to these examples, but to provide the reader with various options and ideas for structuring a journal club. The reader should keep clearly in mind that not all journal club formats will support all goals and objectives for a journal club; an individual residency program may choose to adopt more than one format to achieve more than one goal in the residency program.

Evidence-Based Journal Clubs

One evidence-based journal club format ² already has been discussed. The evidence-based format contained three elements. First, in a case-based presentation format, the presenting resident identified a clinical question derived from an actual patient case and found one journal article that addressed the clinical question. Second, using a structured worksheet, the presenting resident and all conference attendees performed a critical appraisal of the selected article. Third, under close faculty supervision, the presenting resident led a structured discussion with all residents in the residency program at the mandatory journal club conference held monthly.

An evidence-based education journal club also has been reported. ²⁴ Designed for the benefit of faculty interested in medical education, this format also can be applied to almost any clinical topic, specialty area, or topic of interest. In this format, participants bring to the conference all of their unread journals related to the particular topic area selected. In small groups, the participants read aloud the titles of the articles in the journals and ask three questions to determine what articles achieve the relevancy criteria. The three relevancy questions are: (1) does the article have an impact on the topic?; (2) does the article focus on a common issue?; and (3) will the article change practice? If the answer to all three questions is yes, the article is tagged for critical review and assigned to at least two participants for review and discussion at the next conference. Participants have reported high levels of satisfaction with this approach, which challenges them to think critically about which articles from journals they choose to read in detail.

A very structured evidence-based approach has been reported in an obstetrics and gynecology residency program. ¹⁸ A 12-month curriculum was designed to cover 24 concepts encountered in epidemiology, biostatistics, and experimental design. The journal club met monthly for two hours, and two sets of articles were distributed for each session. The first set consisted of literature on topics in epidemiology, biostatistics, and experimental design, with two to six articles distributed for each session. The second set of articles consisted of studies for critical review, selected from all medical specialties to show the concepts presented in the first set of articles for each session. Although this format involves a great deal of resident preparation, it was reported to be well-received by the residents. The only change suggested by residents is that many of them would like to see all the studies for review come from the obstetrics and gynecology literature, rather than from all medical specialties. The full listing of articles selected for the 12-month curriculum is available in the research report. ¹⁸

Journal Clubs Focused on Research Methods and Statistics

Perhaps not entirely distinct from evidence-based journal clubs, but somewhat different in focus, are those providing examples of formats teaching research methods and statistics. One such example has been reported from an internal medicine residency program. ²¹ In this format, one internal medicine resident volunteers to select and present one clinical study for journal club. Two weeks before the meeting, all residents are provided copies of the selected article, and a series of stimulus questions, written by the resident volunteer and a faculty mentor, to aid the residents in probing the quality of the research, the appropriateness of the data and methods, and the validity of the conclusions. The discussion at journal club then focuses on the issues raised by the stimulus questions, rather than on the clinical aspects of the paper. This format requires intensive faculty involvement and facilitation to accomplish its goals.

A journal club format using a structured review instrument was mentioned previously. ⁴ In this format, a resident and faculty member jointly choose a topic, and five articles on this topic are distributed for journal club. Three of these articles are selected for structured review at the conference. Each resident then does a critical review of the three articles using the structured review instrument. At the conference, the resident leader (who chose the conference topic) selects a resident to begin the structured discussion of the article, chairs the ensuing discussion, and expands on the group’s comments. At the end of the conference, the attending coordinator summarizes the discussion and the lessons to be learned from each study. This format ensures that each article for the conference is reviewed in a consistent format, reinforcing for the residents the importance of a consistent, structured, and critical appraisal of the literature.

Journal Clubs for Transmission of Clinical Information

This journal club format may be, with time, the most popular format used in residency programs. Although many examples of this format of journal club exist, all share the goal of transmitting to participants a superficial knowledge of new developments in a particular topic or subspecialty area. One published example is from a survey of orthopaedic hand surgery fellowship programs. ²² In the journal club format used by the majority of fellowship programs surveyed, various journals are surveyed in the area of hand surgery, as many as 12 articles are discussed in a 1-hour conference, and the foremost goal of the conference is to review the current literature in hand surgery. A slight modification of this format is to have each participant review one article from a journal or topic area; each participant then presents briefly to the rest of the group the key points of the article, that all may understand the key contributions of the article to the specialty or topic area.

The No Preparation Journal Club

One unique journal club format has been reported, in which the article for discussion at journal club is not distributed to participants in advance of the conference. ¹⁰ In this format, one resident chooses a research article, and then assists the faculty moderator at the journal club session. Conference participants do not read the article in advance of the conference. The conference begins with a brief presentation of the research question of the selected study; participants then are asked to suggest appropriate study designs for addressing the research question, followed by presentation of the author’s chosen method of study. Participants then are asked to assume the author’s chosen study design and suggest and discuss additional details of study methodology. This process continues until all relevant aspects of the study methodology have been discussed. The results and conclusions of the article then are reviewed. The authors reported that more positive discussion of methodologic issues occurs, rather than the tendency toward method bashing that occurs with review of articles in advance of the journal club. Other advantages cited are that resident work load is decreased and attendance increased by not having to review articles in advance of conference, and that residents gain a greater appreciation of authors’ design choices in studies.

The Internet-Based Journal Club

One otolaryngology residency program has reported on a journal club format that uses e-mail and the internet to facilitate the completion and archival of reviews of the current literature in otolaryngology. ¹⁶ This format involves each class in the residency program (PGY2, PGY3) being assigned a particular journal to review each month. Each class divides the articles in the assigned journal, and each resident prepares written reviews of the articles assigned (one to five articles per month). Each participant submits his or her reviews by e-mail to the online journal club mailing list, which then distributes the reviews to all of the residents and faculty, providing a current and up to date review of that month’s otolaryngology literature. The best four to 10 articles then are selected by the residents and discussed at the journal club meeting that month. The monthly reviews are archived and stored on a server accessible from the departmental website. This database, consisting of reviews of the most current literature, is accessible to all faculty and residents and is searchable by title, date, journal, and thread.

Cataloguing Critically Appraised Articles From Journal Club

Several medical programs maintain computerized (or internet-based) logs of studies that met predefined criteria for validity on various topics as a method of providing a database of current evidence for residents, fellows, and faculty. One popular structured appraisal format for keeping presenting information about a study in a meaningful and succinct manner has been termed Critically Appraised Topics. Ideally, individuals prepare a Critically Appraised Topic before a journal club and post a log of past Critically Appraised Topics on a website or computer for later referral. A good example of a Critically Appraised Topics bank can be found at the University of Rochester’s Internal Medicine and Pediatric Residency Website. ¹³

SUMMARY

Journal clubs always have played a critically important role in graduate medical education; they are not less valuable today than they were when first introduced by Osler. Journal clubs serve numerous purposed in training programs and may have varied goals in different programs. Whatever the goals of the journal club for a particular residency program, and whatever format (or formats) is chosen by which to conduct journal club, the residency program director should keep in mind those factors published in the literature to be associated with journal clubs that have longevity and consistently good resident attendance. ^1,24,26 These factors include having a designated, committed leader, having good faculty participation and attendance, the regular provision of food, mandatory attendance requirements for residents, and formal instruction in epidemiology and biostatistics. It is important to note that, although the format(s) of a journal club may aid it in achieving curricular objectives, the format of journal clubs has not been associated with the overall satisfaction score of residents.

Those organizing journal clubs should articulate clearly the goals of the conference, select the most appropriate format(s) for achieving those goals, and ensure the direct involvement and advocacy of the department chairperson and other respected faculty members. The use of a structured review instrument may improve the residents’ understanding of the material, the focus of their presentations, and their satisfaction with the educational experience. Finally, periodic evaluation of the conference and the institution of appropriate changes will ensure that the journal club remains a valuable and successful part of the training program for many years.