Gruppen, Larry D. PhD; Yoder, Ernie MD, PhD; Frye, Ann PhD; Perkowski, Linda C. PhD; Mavis, Brian PhD
A number of authors in recent years1,2 have noted problems in the quality of medical education research (MER). Some of these problems relate to the nature of the questions asked,3 the level of funding available for MER,4–7 and the balance between practical solutions and theory building.8 Efforts to improve and support MER have taken a number of forms. Some of these use institutional structures9,10 and resources, such as special interest groups11 and PhD faculty in clinical departments.12
Faculty development has become a key initiative in improving the sophistication of MER. Research topics are a basic element of many intensive faculty development programs within individual medical schools,13,14 but the sessions devoted to research are typically a small part of the larger curricula. Embedding an instructional strand on research skills into a broad faculty development curriculum has some definite strengths, but it also has limitations. One limitation is the risk that research skills receive diminished visibility when “competing” with topics like teaching skills and educational methods that are often perceived by many faculty participants as being more immediately relevant to their daily work. Another limitation is that many institutions lack an adequate faculty development platform to support an extensive focus on research skills. Other institutions struggle with minimal resources and expertise in MER topics and methods, particularly in providing a broad-spectrum curriculum or specialized expertise.
Origins of the Medical Education Research Certificate Program
In this context, the Association of American Medical Colleges (AAMC) Group on Educational Affairs (GEA), as part of its strategic planning process, identified “advancing the careers of medical educators through skill development and mentoring” as one of three major areas of focus.15 The GEA leadership formed a Professional Development Committee to identify ways the AAMC could address this focus, and, in 2002, the committee reported on the initial plans for a program to promote faculty development in MER skills, the Medical Education Research Certificate (MERC) program. A MERC steering committee was organized to define the program content, format, and logistics. The committee consisted of approximately nine established researchers (both MDs and PhDs) in medical education from medical schools across the United States.
By the summer of 2004, the identity of MERC had crystallized as a six-workshop series leading to a certificate from the AAMC-GEA. Three workshops were offered under the MERC name at the AAMC annual meeting in 2004, followed by additional workshops at each of the regional GEA meetings the next year. Each year since then, four to five workshops have been offered at the national meeting, and two workshops take place at each of the four regional meetings. The program is open to anyone interested in developing MER skills and understanding. Participants sign up for the program through the AAMC-GEA, which records their progress through the program and determines when they are eligible for the certification.
The original goal of the MERC program was to help medical educators acquire and enhance their research skills so that they could effectively collaborate with more experienced medical education researchers and be better consumers of medical education scholarship. The steering committee specifically designed the program for clinical faculty who were less familiar with the techniques and approaches common to educational research. Thus, the purpose was to prepare these clinical faculty to consult effectively with an expert in MER rather than to develop independence as educational researchers. However, over the past few years the purpose of the MERC program has expanded to include two additional goals: (1) to provide options for faculty development in MER to address the MER needs and goals of individual institutions and (2) to provide a refresher and upgrade of MER skills for experienced as well as novice researchers.
Developing a national program was deemed important by the steering committee for a variety of reasons. As noted above, many institutions lack the readily available expertise to provide such education to their faculty. There were also anticipated economies of scale that could be realized at a national level that might not be feasible at a local, institutional level. Additional advantages included developing a consistent, structured curriculum in research skills that could be disseminated broadly and promoting interactions among the MER community. Finally, it became clear that many in the target audience found the AAMC's imprimatur on the MERC program very important as an indication of the program's quality, which made participation easier to negotiate with their home institutions.
Since the planning and initiation of the MERC program, MERC steering committee members have continued to guide the evolution of the program by leading MERC workshops themselves, as well as through ongoing evaluation of the program, new workshop development, and faculty development and peer review of the workshop presenters. The committee continues to meet at least once a year to review workshop content and program evaluation data, determine the need for new workshops, and develop a brand identity for the certificate program.
Program Curriculum and Session Format
The scope of the skills targeted by the MERC program falls between local faculty development that individual schools might provide and formal degree training. It is more intensive than the occasional local workshop but considerably less intensive than a formal degree curriculum.
The MERC curriculum includes only the topics essential to conducting effective MER. These topics were initially defined by the MERC steering committee and then complemented by a needs assessment of audience members of early workshops to determine potential new topics. The fact that the MERC program is explicitly not designed to develop independent researchers is reflected in the absence of several topics that, although essential to MER, are too technical and complex to convey in the workshop format of the MERC program. Two examples of these topics are statistical procedures and a comprehensive survey of methods for measuring educational outcomes. Both of these were judged to be beyond the scope of MERC, in large part because the teaching of each of these topics requires a level of expertise that is well beyond that feasible under the constraints of the MERC program. The program started with 8 original workshop topics and expanded to 11 based on feedback and needs assessment.
Original curriculum topics
Formulating research questions and designing studies.
In this workshop, participants brainstorm research ideas, write and refine a measurable research question, and discuss when institutional review board (IRB) approval is required. Basics of research design are discussed and applied to their selected research question.
Searching and evaluating the medical education literature.
This workshop is intended for medical educators and clinician educators who want to learn how to effectively search the published medical education literature and to evaluate the value of those searches.
Data management and preparing for statistical consultation.
Participants prepare their data for analysis and address questions about their data that a statistician will likely ask when providing consultation.
Measuring educational outcomes with reliability and validity.
This session introduces participants to the principles of score reliability and validity, using a combination of didactic presentation and review of MER projects.
IRBs and research ethics.
This workshop has been converted to a Web-based module that provides an overview of how research questions and information sources common to MER are related to the need for IRB review. The workshop provides a brief history of the development of IRBs and an overview of the basic concepts and definitions guiding IRB review.
Qualitative analysis methods in medical education.
Participants receive an overview of the purposes, uses, and standards for rigor of qualitative research methods and have an opportunity for skill development in the design of a small, qualitative study and preparation for analysis of qualitative data.
Program evaluation and evaluation research.
This workshop introduces participants to fundamental principles of educational program evaluation and provides participants with a strategy for developing an evaluation plan.
Questionnaire design and survey research.
This workshop describes basic principles in designing surveys and questionnaires and gives workshop participants an opportunity for hands-on experience designing a questionnaire.
Additions to the curriculum
Overview of qualitative data collection methods.
This workshop is intended for physicians and generalists in medical education, as well as faculty and staff involved in student affairs, who wish to develop perspectives and skills for collecting qualitative data, through methods such as focus-group discussions, interviews, observation field notes, and open-ended questionnaires—used in admissions processes, program development, curriculum evaluation, needs assessments, performance evaluation, and various scholarship and research applications.
Scholarly writing: Publishing MER.
In this session, the skills of scholarly writing are explored through the lens of analyzing one manuscript that was not accepted for publication and another that was accepted. Participants examine review criteria that are used by health education journals and apply them to a sample manuscript. In discussing scholarly writing, the participants make a decision about the type of feedback they would give to the authors of the sample paper.
This workshop focuses on planning and conducting research intended to test a specific hypothesis, in contrast to research intended for exploration and hypothesis generation.
The workshop format is designed to be consistent across all topics and emphasizes the practical application of relevant research principles to the individual participants' particular interests and responsibilities. The facilitator begins with an overview of the workshop goals and facilitates introduction of all members of the participant group. Introduction of attendees facilitates interaction within and among small groups of participants and enables the facilitator to gauge the participants' experience and interest and to identify examples and applications to use later in the workshop. Key elements of the workshop topic are then presented in an interactive lecture format. This presentation is guided by a slide set and speaker notes provided to the facilitator with the expectation that the facilitator will adhere to the framework while adapting the presentation to the specific needs and interests of the audience.
Each workshop's focus on application is enabled by dedicating time within the workshop for small-group work including sharing of ideas, identification of challenges, and discussion of applications of the workshop material to their individual or institutional context. The facilitator circulates among these groups to provide commentary, answer questions, and adapt the key points of the workshop to specific situations. In sessions sponsored by an external group (e.g., an individual medical school), these small work groups often focus on issues relevant to the institution, such as a curriculum change under consideration or residency program teaching. At the end of the workshop, participants complete an evaluation to facilitate improvement of the workshop and feedback to the facilitator. Each participant receives handouts that contain the principles addressed in the didactic portion, various examples and exercises used in the workshop, and a list of resources and references relevant to the topic.
The interactive workshop format creates balance between conveying information about research principles and making that information relevant and useful to the participants, enabling them to readily apply the principles. In keeping with the limited goals of the MERC program (i.e., that it is not intended to develop independent researchers), facilitators routinely emphasize the importance of identifying relevant expertise to assist participants in pursuing their research interests. For workshops sponsored by an individual institution, these linkages are often to local units and resources, but the workshops have also revealed the need for support and expertise that are absent within the institution.
Six workshops are required to obtain the certificate, but the only mandatory workshop is “Formulating research questions and designing studies”; individual participants are free to select the remaining five workshops. Flexibility in choosing topics was intended to enable participants to focus on areas relevant to their own needs and situations. This flexibility also facilitates completion of certificate requirements in a reasonable period of time. Individuals may also participate in the workshops without registering for the MERC program.
The original schedule of the program was structured around the AAMC annual meeting in November of each year and the spring regional GEA meetings. Typically, four to five workshops are offered as part of the AAMC annual meeting and two workshops at each of the spring regional GEA meetings. Thus, a participant could complete a certificate in 13 months if needed by attending two annual meetings and one regional spring meeting.
Facilitators for the initial iterations of the MERC workshops were originally drawn from the MERC steering committee, but new facilitator candidates are continuously sought from the ranks of established and productive medical education researchers. The steering committee identifies facilitator candidates, provides them the opportunity to learn about the MERC workshop format, and then “apprentices” them to an experienced facilitator to learn the details of a specific workshop. New facilitators are observed by established facilitators and receive formative feedback on their performance. If mutually agreeable, new facilitators then enter the pool of facilitators for future workshops. Facilitators only lead workshops for which they have been trained or have adequate prior experience and expertise. Each facilitator typically receives a $500 honorarium per session.
Ongoing evaluation by participants and observations by other facilitators helps monitor the quality and consistency of the various facilitators' implementation of the workshops. The goal is to develop a pool of MERC facilitators with both excellent teaching skills and an experience base that will enable participants to learn and apply the workshop objectives to their own work and settings.
As the program has grown and the first certificates have been awarded, interest in the program has also grown. Several individual medical schools, medical education consortia (regional or intracity groups of hospitals and medical schools), and specialty societies have contacted the AAMC-GEA and the MERC steering committee to find out how to provide the MERC program as part of their faculty development efforts. Individual medical schools have offered the MERC program in a number of different formats. Schools have presented MERC workshops as part of monthly or regularly scheduled faculty development seminar series, included MERC workshops as programming offered to formal medical education scholars programs, and built intensive three- to five-day academy programs around the content of the MERC program. In many cases, these institutions provided additional activities to supplement and support the MERC workshops, such as discussion groups, mentoring programs, e-mail groups, research consultations, and other ongoing group activities. Regardless of the local context, the MERC workshops are consistent in content and format with those offered at national and regional venues for an audience from multiple institutions.
Medical education consortia have followed a similar approach, offering a schedule of MERC workshops throughout the year. In addition to these local offerings, two specialty societies have offered one or more MERC workshops as part of their annual national conferences. At least one of these societies has used the MERC program as the basis for developing its national scholars program as a means of promoting medical education scholarship within its field.16
In all of these iterations of the MERC program, the workshop content is consistent, regardless of venue, and the same pool of facilitators leads these workshops. This ensures consistency of learning among the various groups.
The funding for the MERC program is managed by the AAMC and comes from participant tuition. Each individual registering for a certificate pays $100. In addition to this is a workshop fee of $50 for each workshop attended, for a total of $400. Individuals not registered for the certificate can also participate in individual workshops and pay the same workshop fee. Medical schools that arrange to offer the MERC program to their constituent faculty may pay the individual workshop fees for the number of participating faculty or may require individual faculty to pay. Other organizations (e.g., specialty societies) must pay a $1,500 licensing fee to the AAMC and fund the local meeting arrangements, facilitator travel, and honoraria from their own resources, but they are free to determine whether to charge individual workshop participants and, if so, how much.
A key issue for a program like the MERC is evaluation of its success in achieving intended outcomes. Participants evaluate each MERC workshop in terms of appropriate level of content, knowledge gained, presentation quality, and resources provided. These evaluations are reviewed by the facilitators and by the MERC steering committee to monitor workshop quality and relevance. Overall, MERC participants have been very positive in their evaluations of all the workshops.
In addition to the individual workshop evaluations, a formal program evaluation was conducted to assess the effectiveness of MERC. In 2008, an online questionnaire was sent to all MERC graduates (n = 33), of whom 21 responded (64%). The instrument was a retrospective pre/post questionnaire with 16 items (12 closed-ended and 4 open-ended). Twelve items asked participants to indicate their participation in a variety of medical education activities “prior to enrolling,” “during the time you were a MERC participant,” and “after completing MERC.” The results suggest that participants gained knowledge and increased their participation in MER after the program. A second survey of the MERC graduates will be conducted in the fall of 2010.
Continuing discussion focuses on defining the appropriate outcomes for the MERC program. Despite frequent encouragement to modify the purpose of the MERC to develop participants who can conduct MER independently, the advisory committee has sought to stay true to its original and more modest goal of providing participants with enough knowledge about MER to ask informed and focused questions of consultants and experts who can help plan studies and analyze results.
The degree to which the MERC program has met its goals is readily visible in the number of program graduates, listed in Table 1. Over 140 individuals have now completed the MERC program, which indicates that the program is valued and meets a demonstrable need. This need is further supported by the several hundred individuals who have attended workshops without seeking the MERC certificate.
Although the MERC program has been successful in identifying and addressing the need for improved MER skills, it continues to face several ongoing challenges. Scheduling workshops becomes more logistically complex as individuals seek new opportunities to fulfill their required six workshops. It has become common for individuals to have to make special travel arrangements to attend specific workshops to complete their requirements. The need to schedule workshops concurrently at the annual meeting of the AAMC also limits access for some participants.
Another logistical challenge is the need for a minimum number of participants to make a workshop financially viable. This has dictated firm criteria for minimum attendance that often goes against the natural proclivities of program leaders and workshop facilitators, who enjoy teaching and value the broad dissemination of the MERC goals, regardless of the number of participants.
When the MERC program was originally established in 2004, the expectation was that there would be a certain amount of pent-up demand for the program because there had not been a similar resource available on a large scale. The size of this demand is still unknown, however, and the question remains of what might happen after that demand is met. The attendance at regional GEA workshops has shown some signs of declining, but not consistently so. In contrast, the number of institutional requests to offer the program locally has increased. Thus, it is difficult to determine ongoing demand with confidence.
The fact that volunteers lead the MERC program and facilitate its workshops is both a strength and a limitation. The number of volunteers and breadth of institutions represented help ensure that no single, narrow perspective is emphasized. However, relying on volunteers also means that there may be a high rate of leadership turnover and that the time and attention that any of the volunteers can provide the program are limited. For example, early in the development of the MERC program, the steering committee considered a required “capstone” project that would serve as the practical application of what the participant had learned through the program. However, a careful consideration of the faculty resources required for this requirement led the committee to abandon this idea.
Participants in each of the MERC workshops represent a diverse set of skills, training, and experience. Although most are the clinical faculty members for whom the program was designed initially, MERC has drawn participants from other groups, including allied health, medical librarians, basic science faculty, and administrative support staff. Whereas some participants have reported feeling lost in the complexities of the topic, others found a given topic repetitious and already familiar. The emphasis on interactivity in the workshop is one procedural solution for this problem, but, more fundamentally, the MERC workshops have been designed to be as accessible as possible for participants with minimal training or experience.
Meeting the Needs of Medical Education Researchers
In conclusion, the MERC program provides a method for improving and supporting the quality and quantity of MER by educating faculty in fundamental research principles and methods. It is far from the only solution for improving the quality of MER, but it meets a defined need for a segment of the MER community. It also demonstrates the feasibility of a centrally organized and operated faculty development program that is national in scope. This model may be appropriate for other domains in which faculty development is needed at a national level.