Faculty must commit to continuous development of their medical education knowledge and skills to keep pace with escalating demands to expand their teaching responsibilities, learn new pedagogic methods, and adapt to curriculum reform initiatives. Not surprisingly, the number and scope of faculty development programs in medical education have expanded considerably over the past two decades, including fellowships in medical education.1,2 A number of articles describe existing medical education fellowships’ curriculum content and process strategies3–14; however, there is a paucity of rigorous program evaluation studies, especially investigations that employ systematic use of qualitative and quantitative methods to determine outcome data.1,2,15 Moreover, evaluation of educational fellowship programs has relied predominantly on descriptive methods using single-site data collected from participants’ satisfaction surveys that lack assessment of change over time or use of comparison groups.1,2,4,13,15,16
In 1998, members of the Harvard Medical School (HMS) faculty established one of the first longitudinal medical education fellowships in the United States. The program was created to develop and further enhance faculty’s skills as educators; provide opportunities for scholarly research in medical education; support fellows as leaders and agents of change within their organizations; and build a community of educators dedicated to supporting each other and for the system at large.6,17 To evaluate the effectiveness of the fellowship, we designed a two-phase, mixed-methods study to determine the personal and professional impact of the program on our fellows and to examine measureable outcomes of their academic careers and analyze how these outcomes change over time.
As described in a previous article,18 in the first phase of the study we conducted semistructured interviews of program graduates and, through qualitative analysis, demonstrated that “essential elements” of the fellowship fostered transformation of participants’ personal identity as educators and enhanced their confidence and self-efficacy. These elements include knowledge about medical education and teaching skills, a supportive learning environment, community building and networking, peer recognition, and emphasis on reflection and self-awareness.18
In this report we describe the quantitative phase of our mixed-methods study. Using 10 years of data, we conducted pre and post curriculum vitae (CV) analyses to examine 10 measurable outcomes of our education fellowships. We also measured changes in fellows’ educational outcomes with a comparison group of matched peers. We present the outcomes of our investigation using a modified version of Kirkpatrick’s evaluation framework.1 Kirkpatrick described a four-level progressive sequence for evaluating training programs (learner reaction, learning measurement, behavior change, and program results).19 To help guide our own evaluation efforts, we adopted a modified framework developed by Leslie and colleagues,1 which encompasses a seven-level sequence: level 1—learner reaction; level 2a—modification of attitudes/perceptions; level 2b—acquisition of knowledge/skills; level 3—behavioral change; level 4a—changes in organizational practice; level 4b—benefits to students/residents; and level 4c—benefits to patients/communities.
Study participants and recruitment process
The 44 faculty who participated in the fellowship between academic years (AYs) 1999 and 2005 at two HMS sites using a shared curriculum were invited to participate in this two-part study. The sites were the Rabkin Fellowship in Medical Education at a large academic teaching hospital—Beth Israel Deaconess Medical Center (BIDMC)—in Boston, Massachusetts, and the Mount Auburn Hospital Fellowship in Medical Education at a smaller regional teaching hospital—Mount Auburn Hospital (MAH)—in Cambridge, Massachusetts. Together, these programs are termed the “HMS Fellowship in Medical Education” or “fellowship.” The study protocol was approved by the institutional review boards at BIDMC and MAH.
Recruitment of fellowship graduates.
We sent to the 44 fellowship graduates a letter of invitation, study description, and requests for written informed consent and copies of their updated CVs formatted according to HMS guidelines.
Recruitment of comparison group.
For the second part of the study, in June 2005 we asked each of the 12 AY2004–2005 HMS fellowship graduates (6 at each of the 2 sites) to nominate one or two physicians to serve as a comparison group. We sent written instructions to the graduates to identify clinician–educators at their institutions who had demonstrated strong interest in clinical teaching. “Interest” was defined as consistent commitment of time and effort in directing or participating in ongoing teaching activities. In addition, we asked the graduates to nominate faculty who were closely matched in terms of the following criteria: gender, number of years since medical school graduation, academic appointment, and clinical discipline. We included specific instructions not to select peers who were graduates of the HMS fellowship program. The AY2005 fellows submitted names of 19 nominees, to whom we sent invitations along with informed consent forms and requests for their CVs. Fifteen faculty consented and submitted their HMS-formatted CVs. We reviewed each CV to verify strong interest in medical education and commitment to being active teachers, as well as to determine how closely they matched the fellows according to the criteria outlined above. Twelve faculty peers were chosen as the comparison group. We computed a Mann–Whitney U test to determine whether there were significant differences between the fellows and their peers. All study participants are described in Table 1.
Data collection, analysis, and approach
We created a coding manual based on the required HMS CV format to conduct analysis on the collected data (Appendix 1). We chose CV analysis because this represents verifiable data that are detailed and comprehensive.20–23 The HMS CV format requires standard organization and reporting of criteria required for academic advancement. These criteria were derived from a consensus-driven process performed by a faculty task force appointed by the HMS Dean of the Faculty of Medicine. Therefore, academic metrics for this study were the number of longitudinal teaching activities, education-related committee work, teaching presentations, educational leadership roles, medical education funding sources, total publications, medical education publications, teaching awards, newly developed curricula, and academic promotion. Two authors (B.L., L.N.) reviewed and coded each CV together and, after reaching agreement on each entry, entered data into SPSS statistical software, version 19 (IBM, Armonk, New York) for each year relative to fellowship entry (for graduates) or study initiation date (for the comparison group).
Fellowship graduate data collection.
For the first part of the study, we conducted a pre and post analysis of the 42 graduates’ CVs to determine within-subject academic achievement before and after participating in the HMS Fellowship in Medical Education. The starting point for all graduates’ CV analysis was 2 years prior to entering the fellowship. Beginning with the class of AY1999, we collected and entered data at 2-year intervals through June 2009. This provided up to 10 years of data on the earliest fellows.
We compared changes in pre and post outcomes for the fellowship cohort by performing repeated-measures ANOVA. Bonferroni corrections were made to all P values as a conservative measure. Data were entered for all variables two years before fellowship enrollment (F−2), at the end of the fellowship year (F), and at two and four years post fellowship (F+2, F+4). To determine the rate of outcome productivity as a result of the fellowship, we calculated the number of specified outcomes per year for all 42 graduates. We then compared outcomes at constant, two-year time intervals (e.g., F compared with F+2). To provide a conservative estimate of the fellowship’s impact, we removed outlier data from the analysis of publications of one fellow with a large number of publications (163 total publications and 24 medical education publications). We performed further analysis of the 23 participants who were six years post fellowship (F+6) in 2009. We also conducted a cross-tabulation t test analysis to determine the association of academic outcomes with gender and year of medical school graduation.
Comparison group data collection.
Using our coding manual, we conducted a pre and post comparative CV analysis of the same academic metrics for the 12 participants in the AY2005 fellowship cohort and the 12 nonfellowship physicians who served as the comparison group. The starting point for the analysis was two years before study entry in 2005; data collection and analysis continued for both groups through June 2009. We computed cumulative outcomes for the fellows and their matched peers to compare academic outcomes. We tested the difference in outcome changes between HMS fellows and matched peers using the Mann-Whitney U test.
Of the 44 invitees, 42 composed the group of study participants; we were unable to contact 1 graduate, and another individual died during the course of the study. These 42 fellowship graduates who responded to our 2005 and 2009 CV requests constituted a 95% participation rate. The 12 AY2005 matched peers who also responded to our 2005 and 2009 CV requests constituted a 100% participation rate. Statistically significant findings were set at P ≤ .05
Pre and post changes in academic achievement among fellowship graduates
Baseline analysis of academic outcomes for the 42 HMS fellows showed that 4 out of the 10 metrics—academic promotions (P ≤ .001), teaching leadership roles (P ≤ .001), medical education committees (P = .013), and medical education funding (P ≤ .001)—increased significantly when comparing F−2 and the end of the fellowship year. Post analysis showed significant increases in 2 of the 10 metrics between F and F+2—teaching leadership roles (P = .007) and medical education committees (P = .050). There was an apparent trend toward significance for academic promotions between F+2 and F+4 (P = .089) (Table 2). These outcomes correspond with a modified version of Kirkpatrick’s evaluation framework at level 4a—changes in organizational practice1 (Figure 1).
Analysis of the 23 fellows, for whom we had data extending six years post fellowship (F+6), showed no significant increases or declines in measured outcomes. Our analysis of outcomes by gender showed that women, compared with men, did significantly more committee work at F+2 (P = .014), obtained significantly more medical education funding at F+2 and F+4 (P = .004 and .013), and achieved a significantly higher rate of promotion at F+4 (P = .011). There were no significant findings when we analyzed the number of years between medical school graduation and the postfellowship years.
Differences in changes of academic achievement between fellows and comparison group
We found no significant differences in the demographic characteristics of AY2005 fellowship graduates compared with their matched peers for two consecutive years prior to the study year (AY2003, AY2004) or through the fellowship year AY2005.
We found that the AY2005 fellows significantly outperformed their matched peers in 5 of the 10 metrics at F+2 (longitudinal teaching, teaching leadership, curricular offerings, medical education committees, and teaching presentations) and 5 metrics at F+4 (longitudinal teaching, teaching leadership, medical education committees, teaching presentations, and medical education publications) (Table 3). These outcomes correspond with a modified version of Kirkpatrick’s framework at levels 3 to 4b.
We used a rigorous, mixed-methods approach to evaluate the outcomes of an HMS longitudinal faculty development program. Our initial qualitative study of interviews with fellowship graduates explored programmatic elements that contributed to personal and professional transformation as medical educators.18 By design, the qualitative study focused on Kirkpatrick’s levels 1 to 2b: learners’ reactions, modifications of attitudes, and acquisition of knowledge and skills. In the current study we used a standardized CV analysis approach to determine outcome data among fellowship graduates and a comparison group of peers, focusing on Kirkpatrick’s levels 3 to 4b: behavioral change, changes in organizational practice, and benefits to other learners.
CV analysis is a verifiable source of data used by researchers to assess faculty development programs.1,2,4,15,16,20–23 Studies using this method have reported increases in participants’ teaching presentations, publications, and academic promotions.20,21 Several investigators have also found significantly increased education leadership activities, although the lack of control groups limits the generalizability of this finding.5,8,22,23
The strengths of this current evaluation include 10 years of follow-up data, use of a well-matched comparison group, high response rate, and conservative analysis of verifiable outcomes at two different sites. Limitations of this study should also be noted. First, CV analysis depends on self-reported data. Second, we used a small, self-selected, nonrandomized cohort for comparison with the AY2005 fellows. However, it was not feasible to randomize selection of the peer group because the fellows were located at two separate institutions and represented four different clinical disciplines. Our data suggest that the cohorts were well matched, and this quasi-experimental design contributes significantly to our understanding of the fellowship’s impact beyond a pre and post, within-subject study design.
We found significant changes in fellowship graduates’ rate of academic achievement for teaching leadership roles and medical education committees, and a positive trend in academic promotions. Of note, HMS changed its criteria for academic promotion to include teaching and education excellence only one year before we concluded data entry. Further, fellows significantly outpaced a comparison group of matched peers in six of the academic metrics: longitudinal teaching activities, teaching leadership roles, curricular offerings, medical education committees, teaching presentations, and number of medical education publications.
Our findings strongly suggest that we successfully met the fellowship’s goals to graduate and support individuals who become academic leaders and change agents as well as active members in a community of educators dedicated to teaching excellence at HMS. The finding that female graduates had significantly more medical education funding and higher rates of promotion than male graduates four years post fellowship suggests that longitudinal faculty development may be an important strategy for decreasing historical academic gender disparities.24,25
We also considered system-wide programmatic impacts of the HMS Fellowship in Medical Education. Since the inception of the hospital-based fellowships in 1998 to the present, we have graduated a total of 166 fellows from the programs at BIDMC and MAH. These include faculty from junior to senior career levels, representing 18 different clinical departments. Given the success of the initial two hospital-based programs, the fellowship leaders introduced an adapted version of their curriculum into the existing HMS Academy medical education fellowship in 2004. Given the number of learners and peers that each fellow interacts with, and assuming that the findings of our current study are generalizable among the graduates, we believe that the fellowship has had a significant system-wide impact on HMS’s teaching endeavors and educational programs.
Heeding the call for more rigorous research studies of longitudinal faculty development programs,15,26 we conducted a mixed-methods study to determine the personal and professional impact of the HMS Fellowship in Medical Education on our graduates and to examine measureable outcomes in areas of academic achievement over time. In accordance with Kirkpatrick’s evaluation framework, our findings indicate that investment in a yearlong fellowship in medical education produces positive change in participants’ behaviors and improves institutional practice. Further research needs to be done to determine effective methods to assess programmatic impact on patients, families, and health care communities. We encourage other medical education researchers to use common coding manuals when assessing faculty development programs to aid in cross-institutional comparisons and standardization of evaluation practices.
Acknowledgments: The authors wish to thank Charles Hatem, MD, for his vision in creating the fellowships, and for his mentorship and unceasing support. The authors also wish to thank Grace Huang, MD, for reviewing the manuscript and providing important suggestions for improvement.
1. Leslie K, Baker L, Egan-Lee E, Esdaile M, Reeves S. Advancing faculty development in medical education: A systematic review. Acad Med. 2013;88:10381045.
2. Thompson BM, Searle NS, Gruppen LD, Hatem CJ, Nelson EA. A national survey of medical education fellowships. Med Educ Online. 2011;16. http://www.med-ed-online.net/index.php/meo/article/view/5642
. Accessed September 28, 2015.
3. Searle NS, Hatem CJ, Perkowski L, Wilkerson L. Why invest in an educational fellowship program? Acad Med. 2006;81:936940.
4. Gruppen LD, Simpson D, Searle NS, Robins L, Irby DM, Mullan PB. Educational fellowship programs: Common themes and overarching issues. Acad Med. 2006;81:990994.
5. Steinert Y, McLeod PJ. From novice to informed educator: The teaching scholars program for educators in the health sciences. Acad Med. 2006;81:969974.
6. Hatem CJ, Lown BA, Newman LR. The academic health center coming of age: Helping faculty become better teachers and agents of educational change. Acad Med. 2006;81:941944.
7. Simpson D, Marcdante K, Morzinski J, et al. Fifteen years of aligning faculty development with primary care clinician–educator roles and academic advancement at the Medical College of Wisconsin. Acad Med. 2006;81:945953.
8. Wilkerson L, Uijtdehaage S, Relan A. Increasing the pool of educational leaders for UCLA. Acad Med. 2006;81:954958.
9. Muller JH, Irby DM. Developing educational leaders: The teaching scholars program at the University of California, San Francisco, School of Medicine. Acad Med. 2006;81:959964.
10. Rosenbaum ME, Lenoch S, Ferguson KJ. Increasing departmental and college-wide faculty development opportunities through a teaching scholars program. Acad Med. 2006;81:965968.
11. Frohna AZ, Hamstra SJ, Mullan PB, Gruppen LD. Teaching medical education principles and methods to faculty using an active learning approach: The University of Michigan Medical Education Scholars Program. Acad Med. 2006;81:975978.
12. Robins L, Ambrozy D, Pinsky LE. Promoting academic excellence through leadership development at the University of Washington: The Teaching Scholars Program. Acad Med. 2006;81:979983.
13. Searle NS, Thompson BM, Perkowski LC. Making it work: The evolution of a medical educational fellowship program. Acad Med. 2006;81:984989.
14. Plack MM, Goldman EF, Wesner M, Manikoth N, Haywood Y. How learning transfers: A study of how graduates of a faculty education fellowship influenced the behaviors and practices of their peers and organizations. Acad Med. 2015;90:372378.
15. Steinert Y, Mann K, Centeno A, et al. A systematic review of faculty development initiatives designed to improve teaching effectiveness in medical education: BEME guide no. 8. Med Teach. 2006;28:497526.
16. O’Sullivan PS, Irby DM. Reframing research on faculty development. Acad Med. 2011;86:421428.
17. Hatem CJ, Lown BA, Newman LR. Strategies for creating a faculty fellowship in medical education: Report of a 10-year experience. Acad Med. 2009;84:10981103.
18. Lown BA, Newman LR, Hatem CJ. The personal and professional impact of a fellowship in medical education. Acad Med. 2009;84:10891097.
19. Kirkpatrick DL. Evaluating Training Programs: The Four Levels. 1994.San Francisco, Calif: Berrett-Koehler Publishers.
20. Morzinski JA, Simpson DE. Outcomes of a comprehensive faculty development program for local, full-time faculty. Fam Med. 2003;35:434439.
21. Gruppen LD, Frohna AZ, Anderson RM, Lowe KD. Faculty development for educational leadership and scholarship. Acad Med. 2003;78:137141.
22. Moses AS, Skinner DH, Hicks E, O’Sullivan PS. Developing an educator network: The effect of a teaching scholars program in the health professions on networking and productivity. Teach Learn Med. 2009;21:175179.
23. Rosenbaum ME, Lenoch S, Ferguson KJ. Outcomes of a teaching scholars program to promote leadership in faculty development. Teach Learn Med. 2005;17:247252.
24. Lautenberger DM, Dandar VM, Raezer CL, Sloane RA. The State of Women in Academic Medicine: The Pipeline and Pathways to Leadership. 2014. Washington, DC: Association of American Medical Colleges; https://members.aamc.org/eweb/upload/The%20State%20of%20Women%20in%20Academic%20Medicine%202013–2014%20FINAL.pdf
. Accessed September 28, 2015.
25. Terosky AP, O’Meara KA, Campbell CM. Enabling possibility: Women associate professors’ sense of agency. J Divers High Educ. 2014;7:5876.
26. McLean M, Cilliers F, Van Wyk JM. Faculty development: Yesterday, today and tomorrow. Med Teach. 2008;30:555584.
CV Analysis Coding Manual Developed for the Harvard Medical School Fellowship in Medical Education, From a Study of Academic Achievement and Longitudinal Faculty Development, 1999–2005, Boston, Massachusetts
All activities are coded by academic year (July 1–June 30, 200X)
- Fellow ID
- Year of medical education fellowship graduation
- Year relative to the fellowship (e.g., F, F+1, F+2)
- Year of medical school graduation
- Clinical department
- Appointment at hospital/affiliated institutions
Note: Kirkpatrick level 1—Learner reactions; level 2a—Modifications of attitudes, perceptions; and level 2b—Acquisition of knowledge, skills were reported in a previous qualitative study.18
Kirkpatrick level 3 outcomes: Behavior change
- Number of episodic (“single”) teaching presentations
- Total number of publications (all inclusive)
- Total number of publications related to teaching, medical education, and medical education research
- Funding related to medical education innovation or medical education research
- (Note: Count each activity for the first year of funding only)
Kirkpatrick level 4a outcomes: Impact on organizational practice
- Academic appointment
- Number of academic and/or teaching leadership roles including director of a medical school course, clerkship, graduate medical education residency program, clinical division, fellowship curriculum, faculty development program, national continuing medical education course, medical education center, or medical education academy
- Number of major new educational curriculum offerings/materials developed
(Note: Count for the first year of curriculum implementation only)
- Total number of major local, regional, and national committee assignments related to medical education or medical education research
Kirkpatrick level 4b outcomes: Benefits to students, residents, fellows, and faculty
- Number of teaching awards
- Number of longitudinal medical student, resident, fellow, or faculty development teaching activities per year relative to the fellowship (Note: This includes commitment to teaching in a course or a regularly occurring elective, inpatient attending duties, or other activities that entail consistent 20+ face-to-face contact hours. “Mentoring” hours are not counted because interpretation of this activity varies widely among fellows.)
Kirkpatrick level 4c outcomes: Benefits to patients, communities (not measured)