Effective mentoring is critical to the success of early-career investigators. Strong mentorship has been linked to enhanced mentee productivity, self-efficacy, and career satisfaction.1–11 Yet, despite its importance, mentoring is typically learned by example, trial and error, and peer observation.12,13 Given this approach, mentor capabilities are highly variable. Current mentor development and training contrasts sharply with the contemporaneous rigor of instruction and assessment characteristic of competency-based and practice-centered health sciences education. Encouraging progress toward more formal mentor training is evident at several academic health centers (AHCs),13–17 yet its general merits have been questioned in the absence of experimental evidence to support its value.
We hypothesized that a systematic, formal training strategy would result in significantly improved mentoring competency. To test this hypothesis, we adapted and implemented a published mentoring curriculum18,19 for health sciences researchers.20,21 We conducted a randomized controlled trial (RCT) with mentors of clinical and translational researchers at 16 academic sites to test the curriculum’s ability to improve self-reported mentoring skills across six core competencies, and to examine its influence on mentoring behaviors.
Method
Study sites and population
We conducted this study at 16 insti tutions in the United States and Puerto Rico (see Acknowledgments for full listing), 15 of which have National Institutes of Health Clinical and Translational Science Awards (CTSAs). Prior to the study, sites had mentoring programs of variable size, focus, and structure. For example, 12 participating institutions (75%) provided orientation for mentors, 10 (63%) formally evaluated mentors, 8 (50%) assisted with mentor selection, 7 (44%) used mentor contracts, 6 (38%) had formal mentor training in place, and 5 (31%) provided financial incentives for mentors.
Eligible participants were faculty currently mentoring trainees who were conducting clinical and/or translational research at least 50% of the time. Recruitment targeted mentors of CTSA KL2 scholars, others with K or similar career development awards, and junior faculty, and secondarily mentors of postdoctoral fellows and graduate students.22 A total of 283 pairs were recruited from a convenience sample of the 1,261 mentors contacted.22 No incentives were offered. The institutional review boards of the participating AHCs reviewed the study protocol and determined it to be either exempt (n = 11) or approved via expedited review as minimal risk (n = 5).
Study design
The trial, conducted between June 2010 and July 2011, employed a mixed-methods approach through which mentoring skills were measured quantitatively and mentoring practices described qualitatively. Following the baseline interviews (see below), we randomized 283 mentors into either the intervention group or the control group, with block randomization by study site (12–27 pairs/site). Mentors and mentees alike completed a 15- to 30-minute baseline interview (pretest, prior to the intervention), and a 30- to 45-minute posttest interview three months after the mentors completed the intervention. Baseline structured interviews, which contained no open-ended questions aside from descriptions of previous mentor training, were conducted in person by trained site research assistants (S.C.H., K.C.S.).22 Mentees remained blinded to the group allocation of their mentors throughout the trial. Because of resourcing and the need for consistency in the posttest qualitative data collection, all postintervention interviews were administered via phone by one of three University of Wisconsin–Madison (UW) staff members (including S.C.H. and K.C.S.).
Intervention
We adapted the curriculum used for the intervention group, titled Mentor Training for Clinical and Translational Researchers,20 from Entering Mentoring,18 which was originally created for mentors in the biological sciences. The curriculum was tailored for clinical and translational scientists by a multi-institutional team and implemented at 16 sites by trained facilitators (W.C.H., R.M., E.D.S.).21 The process-based curriculum focuses on six key competencies: (1) maintaining effective communication, (2) establishing and aligning expectations, (3) assessing mentees’ understanding of scientific research, (4) addressing diversity within mentoring relationships, (5) fostering mentees’ independence, and (6) promoting mentees’ professional career development. Facilitators led discussion of case studies and activities intended to engage mentors in critical reflection of their mentoring philosophy. This pedagogical approach encouraged peer discourse in which participants considered an intellectual framework for mentoring, explored strategies to improve their mentoring relationships, solved mentoring dilemmas, and created mentoring action plans.
Facilitators were trained together by UW staff (C.P., S.C.H., P.A.) in Madison, Wisconsin, for 1.5 days prior to implementing the curriculum at their individual sites. Intervention fidelity was monitored via facilitator surveys and conference calls. Although the order in which curriculum content was delivered was uniform, flexibility in the length and spacing of the sessions was essential to accommodate schedules and maximize attendance. The 8 hours were most commonly delivered as four 2-hour sessions (n = 13)21; others offered two 4-hour (n = 2), or one 2-hour and two 3-hour sessions (n = 1). The sessions occurred over a period ranging from 2 days to 11 weeks; the average was 5 weeks.
Outcome measures
Though the study’s curriculum was adapted from Entering Mentoring,18 the evaluative measures originally used for that curriculum were not applicable to our study population, and we determined that other existent measures were inadequately aligned with the intervention. Therefore, the primary outcome measure was based on mentors’ self-reported scores on the validated Mentoring Competency Assessment (MCA).22 This 26-item research mentoring skills inventory was designed to align with the six competencies of the curriculum. Mentors rated their mentoring skill levels on a seven-point Likert-type scale (1 = not at all skilled, 4 = moderately skilled, 7 = extremely skilled), rating their own skills globally as they pertained to all their active mentees. Mentees also completed the MCA but rated only the mentor enrolled in the trial; in addition to choices on the seven-point scale, mentees could choose 0 for “not observed” when rating mentoring skills.
We administered the MCA at baseline (pretest) and post intervention (posttest) to all enrolled mentors and mentees. The a priori primary outcome of the trial was the change in mentors’ MCA composite score between these two time points.
The posttest version of the MCA also included a “retrospective pretest” that asked respondents (mentors and mentees alike) to reconsider and re-rate mentors’ baseline skills; mentors and mentees were asked, first, to re-rate mentors’ skills at the onset of the study period (designated as “before”) and then to rate them at the present time (designated as “now”) for each MCA item. This established approach captures changes in the respondents’ perceptions over the study period.23,24
Secondary prespecified outcomes included the following:
- change in mentors’ awareness of their skills as measured by the change in their self-report composite scores from retrospective pretest to posttest;
- change in mentors’ self-report subscores for each competency (pre- to posttest and retrospective pre- to posttest);
- change in mentors’ self-report scores on each of the 26 MCA items (pre- to posttest and retrospective pre- to posttest);
- change in mentees’ composite assessment of their mentors’ skills (pre- to posttest and retrospective pre- to posttest); and
- change in mentors’ behavior, as reported qualitatively by mentors and their mentees.
The posttest structured interview for both mentors and mentees included a series of open-ended questions about changes in perceptions and practices since the baseline interview. Please see Supplemental Digital Chart 1 (https://links.lww.com/ACADMED/A192). The interviewers asked mentors and mentees to reflect on whether their understanding of effective mentoring had changed, and to report changes in mentors’ behavior for each of the six mentoring competencies. For example, interviewers asked mentors if they had changed the way they communicate with mentees, and they asked mentees if they noted any changes in their mentors’ communication with them. We applied content analysis to develop a codebook scheme.25
We coded mentor responses as follows: no change, awareness, intent to change, or implemented change.26,27 We assigned each mentor the highest “stage of change” reported.21 We coded mentee responses about mentors’ skills as follows: no change, negative change, neutral change, or positive change. To be defined as “positive,” mentees had to describe definitive improvement since the baseline interview; this could be articulated as either a general or specific change in mentoring behavior. We categorized responses as “neutral” when mentees described both positive and negative changes, or when the mentee’s opinion was uncertain. The full qualitative section for each respondent was the unit of analysis and was assigned to a mutually exclusive category; in other words, we did not split any mentee responses into two categories (e.g., both “negative” and “neutral”). Two UW researchers (S.C.H., K.C.S.) independently coded each interview, with interrater reliability of 98.7% for mentor data and 97.1% for mentees.
Statistical analysis
We summarized baseline mentor and mentee characteristics. We collected data on some of these characteristics, such as research focus, during the posttest assessment, as noted in the results and tables. The prespecified primary analysis was a test for a collective difference between the intervention and control groups in the change from pretest to posttest (self-reported) MCA composite score using the Wilcoxon rank-sum test. We summarized the magnitude of the intervention’s effect on composite score by calculating the mean group difference in pre- to posttest change with a 95% confidence interval. All analyses were intention to treat, so they included all participants completing the posttest, regardless of intervention adherence.
We also analyzed the prespecified secondary outcomes using mean group differences with 95% confidence intervals. For MCA composite score and subscales, mean scores with 95% confidence intervals were calculated for the pretest, retrospective pretest, and posttest assessments. We tested for differences between groups in the pre- to posttest change and retrospective pre- to posttest change using Wilcoxon rank-sum tests without adjustment for multiple tests.
We examined the consistency across sites of the distribution of improvements for mentors in the intervention group composite scores (pre- to posttest and retrospective pre- to posttest), and we used analysis of variance to test for site and site–treatment interaction effects for the primary outcome. We also determined estimates and 95% confidence intervals for the group difference in pre- to posttest composite score by gender and academic rank using a linear model to calculate P values for interaction between treatment group and subgroup.
We used chi-square tests to determine statistical significance of differences in the proportion of changes in mentors’ behavior, as reported qualitatively by mentors and their mentees.
For all statistical tests, we considered a P value ≤ .05 to be significant.
Results
Recruitment and follow-up
We screened a total of 1,261 mentors for eligibility; 283 mentors were randomized and allocated (with their paired mentee) to either the intervention (n = 144) group or the control group (n = 139; Figure 1). Of the 144 mentors in the intervention group, 94% (n = 136) participated in the training, and of these, 82% (n = 111/136) completed all eight hours. For completion of the postintervention interviews, there was a retention rate of 98% (mentors: n = 277; mentees: n = 275).
;)
Figure 1: Flow of both mentor and mentee participants through a multi-institutional randomized controlled trial of a formal mentoring curriculum for mentors of mentees working in clinical and translational research, 2010–2011.
Characteristics of participants
Mentors’ demographic and academic characteristics at baseline were similar in the two study groups (Table 1). The majority (60%, n = 170) of mentors were male. They had a mean age of 50.5 years (range: 31–81) and were primarily white (91%, n = 257). The majority were full or associate professors and reported extensive mentoring experience (average of 15 years, standard deviation [SD] 8.0 years). Only 21% (n = 59) experienced prior mentor training. The mentors’ research focus areas included laboratory, clinical, behavioral, and community engaged, though the largest percentage conducted clinical research (66%, n = 187).
Table 1: Characteristics of Mentors and Mentees in Multi-institutional Randomized Controlled Trial of a Formal Mentoring Curriculum for Mentors of Mentees Working in Clinical and Translational Research, 2010–2011a
The baseline characteristics of the paired mentees in the two groups were also comparable. Their mean age was 35.9 years (range: 25–61), and 42% (n = 118) were male. Whereas the majority of mentees self-identified as white (74%, n = 208), 30% (n = 85) self-selected other racial categories. Most mentees were funded by career development awards or postdoctoral fellowships. They were engaged in the full spectrum of clinical and translational research, and similar to mentors, the majority conducted clinical research (69%, n = 196).
Effects of the intervention
Baseline. Between-group self-report MCA composite scores for mentors were not significantly different at baseline (data not shown).
Primary outcome. The mean change in pretest to posttest MCA composite scores was larger in the intervention group compared with the control group (+0.40 versus +0.18, P < .001; Figure 2).
Figure 2: Comparisons by group of mean Mentoring Competency Assessment composite score self-assessed by mentors, with changes shown from pretest (“Pre”) to posttest (“Post”) and from retrospective pretest (“Retro-pre”) to posttest (“Post”). Group means with 95% confidence intervals are shown. P values test for group difference in the indicated change.
Mentor secondary outcomes. Likewise, three of the six subscale competency self-report scores—communication (+0.53 versus +0.23, P < .001), expectations (+0.45 versus +0.14, P < .001), and professional development (+0.37 versus +0.16, P = .009)—were also significantly larger for the intervention group (Figure 3A-3F).
Figure 3: Comparisons by group of mean Mentoring Competency Assessment subscale scores self-assessed by mentors, with changes shown from pretest (“Pre”) to posttest (“Post”) and from retrospective pretest (“Retro-pre”) to posttest (“Post”). Subscales shown are communication (3A), expectations (3B), understanding (3C), independence (3D), diversity (3E), and professional development (3F). Group means with 95% confidence intervals are shown. P values test for group difference in the indicated change.
For the secondary outcome examining change in retrospective self-report pretest to posttest MCA scores, there was also a significant improvement in composite scores in the intervention group compared with the control group (+0.70 versus +0.20, P < .001; Figure 2), as well as in all six of the subscale competency scores (P < .001; Figure 3A-3F). These reflective gains were greater because mentors in the intervention significantly decreased their assessment of their baseline MCA composite score as compared with the control (P < .001).
The mean changes for the intervention and control groups, as well as the estimates and 95% confidence intervals for the intervention/control differences in mean change, are given in Supplemental Digital Figure 1, https://links.lww.com/ACADMED/A193, for all MCA self-report scores, including MCA composite scores, subscale scores, and each of the 26 individual items.
Mentee secondary outcomes. We also examined the mentees’ assessment of their mentors’ skill (pre- to posttest and retrospective pre- to posttest). Post intervention interviews confirmed that 93% (256/275) of the mentees were blinded to the group allocation of their mentor. MCA composite scores, as measured by mentees, were not significantly different at baseline (data not shown). The mean change in pretest to posttest MCA composite scores as rated by the mentees was not significantly different for the intervention group compared with the control (P = .20); however, there was a significant mean change in the retrospective pretest to posttest MCA composite scores (+0.35 versus +0.16, P = .003; Figure 4). Five subscore ratings assessed by the mentees (retrospective pre- to posttest) were also found to be significant: expectations (+0.48 versus +0.21, P = .007); understanding (+0.37 versus +0.16, P = .002); independence (+0.33 versus +0.15, P = .04); diversity (+0.21 versus +0.07, P = .05); and professional development (+0.33 versus +0.18, P = .01; not shown). There were no significant differences in improvements of retrospective pre- to posttest MCA composite score in the intervention versus control group across mentee gender (P = .42) or mentee academic rank (P = .67).
Figure 4: Comparisons by group of mean Mentoring Competency Assessment composite score assessed by mentees, with changes shown from pretest (“Pre”) to posttest (“Post”) and from retrospective pretest (“Retro-pre”) to posttest (“Post”). Group means with 95% confidence intervals are shown. P values test for group difference in the indicated change.
Qualitative results. In addition to MCA skill gains, intervention group mentors reported a significantly greater degree of change in their awareness of mentoring competencies and need to implement behavioral changes. Qualitative data analysis indicated that 97% (137/141) of intervention group mentors reported an increased awareness, intent to change, or actual behavioral change since the baseline interview, as compared with 53% (72/136) of the control group mentors (P < .001, Figure 5). The majority (87% [123/141]) reported that they had implemented at least one behavioral change, as compared with 42% (57/136) in the control (P < .001). Consistent results were reported by the mentees: 68% (95/140) of mentees whose mentors were in the intervention group reported that they noted at least one positive change in their mentors’ behavior as compared with 57% (77/135) in the control (P = .053). Further, 44% (61/140) noted two or more positive changes as compared with 24% (33/135) in the control (P = .002).
Figure 5: Self-reported stages of change at the six-month postinterview for the intervention and control groups. Given is the percentage of mentors by group who reported the given stage together with 95% Wilson confidence intervals. The difference in proportion of intervention versus control mentors who reported implementing at least one behavioral change is significant (P < .001).
Subgroup analysis results. Improvements in pre- to posttest MCA composite score in the intervention versus control groups were observed across gender and mentor academic title; see Supplemental Digital Figure 2, https://links.lww.com/ACADMED/A194. We observed consistent improvements in the intervention group MCA composite self-report scores for each of the 16 sites; see Supplemental Digital Figure 3, https://links.lww.com/ACADMED/A195. There was no evidence of a site or site–treatment interaction effect on the primary outcome (P = .73).
Discussion and Conclusions
Our study aimed to address the impact of a competency-based mentor training curriculum designed for the mentors of junior investigators embarking on clinical and translational research careers. To our knowledge, this is the first multisite RCT to provide evidence of perceived skill gains from research mentor training based on reports from both mentors and their mentees.
We detected a statistically significant difference in the primary outcome variable, pre- to posttest changes in the self-report MCA composite score for mentors in the intervention versus the control group, suggesting improvement in mentoring skills associated with mentor training. Moreover, we observed improvement in three of the six specific competency subscores. When examining retrospective pre- to posttest changes, we observed even larger gains (and significant improvement in all six subscores), reflecting the intervention groups’ reassessment of their baseline skills. Such evidence of reflective learning was also supported by data collected from participating mentors immediately following training.21
We detected skill gains at each of the diverse sites, with intervention-attributable improvements irrespective of gender and career stage, suggesting that the gains are robust. Moreover, we observed control-adjusted gains in nearly every item in the MCA. These measured skill gains are also evidenced by qualitative data that suggest an impact on mentoring practices. The observation that even those with more than 15 years of mentoring experience recognized areas for improvement in their skills and practice suggests that a structured mentor training approach could be a generalizable strategy to improve the quality of clinical and translational research mentoring at academic institutions. We recognize, however, that mentors who are not self-motivated to enroll in such training may demonstrate different skill gains and behavioral outcomes.
A potential limitation of any measure of self-reported learning is the reliability of respondents to accurately assess their own skills. Mentors in the intervention group who participated in eight hours of training are likely to expect benefits from this experience and may be more inclined to rate their skills higher; however, responses to the posttest interview questions indicate that a greater number of mentors in the intervention group, as compared with the control, reported specific changes in their mentoring behavior, indicating an impact of the training beyond reported skill gains. Furthermore, we are not limited to self-reported data; initial analyses from mentees, who were blinded to their mentors’ group allocation, suggest that mentees of trained mentors perceived changes in their mentors’ competency. These mentees retrospectively reported greater skill gains as well as a greater number of positive changes in their mentors’ practices. These mentees’ observations externally validate the positive impact of mentor training on clinical translational mentoring skills and suggest that a small change in skill assessment may translate to meaningful improvements in mentoring practices. More extensive qualitative data analyses will provide insights into critical factors in men toring relationships as perceived by both mentors and mentees, as well as the influence of research mentor training.
Although these results are promising, they are limited to short-term outcomes; it is unknown whether they will translate into sustained improvement in mentoring skills. The study timeline further limited the ability to track whether mentor training was associated with enhanced mentee success in terms of grants, publications, and career trajectories. However, this quantitative and qualitative dataset (283 mentor–mentee pairs) may serve as a baseline from which to examine sustained improvements in mentoring skills and behaviors and to track mentee outcomes long-term.
Our RCT findings document the impact of a structured, competency-based research mentor training curriculum to enhance mentoring skills and to stimulate concrete behavioral change. The availability of this evidence-based training program is timely given the national call to train clinical and translational researchers.11,28 To facilitate broader implementation, we have published the complete curriculum.20 This curriculum, as well as adaptations for the mentors of clinical and behavioral researchers, community-engaged researchers, and biomedical researchers, is easily accessible on a new legacy Web site that allows users to download full curricula, build their own curriculum, evaluate their training, and access a range of resources.29 We hope that providing public access to these training materials and assessment tools will improve career development for health science researchers.
Acknowledgments: The authors wish to thank Karin Silet, MA (Institute for Clinical and Translational Research, University of Wisconsin–Madison [UW]) for discussions leading to the conception, conduct, and completion of this study; Stephanie (Schiro) Wilson (Department of Radiology, UW) for assistance with instrument design and data collection; Larissa I. Zakletskia, MA (Department of Family Medicine, UW) for assistance with data organization and cleaning, and Brandon Deaner (Department of Engineering Mechanics, UW) for data entry.
References
1. Bland CJ, Taylor AL, Shollen SL, Weber-Main AM, Mulcahy PA Faculty Success Through Mentoring. 2009 Lanham, Md Rowman & Littlefield Education
2. Cho CS, Ramanan RA, Feldman MD. Defining the ideal qualities of mentorship: A qualitative analysis of the characteristics of outstanding mentors. Am J Med. 2011;124:453–458
3. Feldman MD, Arean PA, Marshall SJ, Lovett M, O’Sullivan P. Does mentoring matter: Results from a survey of faculty mentees at a large health sciences university. Med Educ Online. April 23, 2010;15
4. Garman KA, Wingard DL, Reznik V. Development of junior faculty’s self-efficacy: Outcomes of a National Center of Leadership in Academic Medicine. Acad Med. 2001;76(10 suppl):S74–S76
5. Palepu A, Friedman RH, Barnett RC, et al. Junior faculty members’ mentoring relationships and their professional development in U.S. medical schools. Acad Med. 1998;73:318–323
6. Raggins BR, Kram KE The Handbook of Mentoring at Work: Theory, Research and Practice. 2007 Thousand Oaks, Calif SAGE Publications
7. Ramanan RA, Phillips RS, Davis RB, Silen W, Reede JY. Mentoring in medicine: Keys to satisfaction. Am J Med. 2002;112:336–341
8. Sambunjak D, Straus SE, Marusić A. Mentoring in academic medicine: A systematic review. JAMA. 2006;296:1103–1115
9. Shea JA, Stern DT, Klotman PE, et al. Career development of physician scientists: A survey of leaders in academic medicine. Am J Med. 2011;124:779–787
10. Steiner JF, Curtis P, Lanphear BP, Vu KO, Main DS. Assessing the role of influential mentors in the research development of primary care fellows. Acad Med. 2004;79:865–872
11. Fleming M, Burnham EL, Huskins WC. Mentoring translational science investigators. JAMA. 2012;308:1981–1982
12. Keyser DJ, Lakoski JM, Lara-Cinisomo S, et al. Advancing institutional efforts to support research mentorship: A conceptual framework and self-assessment tool. Acad Med. 2008;83:217–225
13. Silet KA, Asquith P, Fleming MF. Survey of mentoring programs for KL2 scholars. Clin Transl Sci. 2010;3:299–304
14. Tillman RE, Jang S, Abedin Z, Richards BF, Spaeth-Rublee B, Pincus HA. Policies, activities, and structures supporting research mentoring: A national survey of academic health centers with clinical and translational science awards. Acad Med. 2013;88:90–96
15. Johnson MO, Subak LL, Brown JS, Lee KA, Feldman MD. An innovative program to train health sciences researchers to be effective clinical and translational research mentors. Acad Med. 2010;85:484–489
16. Feldman MD, Steinauer JE, Khalili M, et al. A mentor development program for clinical translational science faculty leads to sustained, improved confidence in mentoring skills. Clin Transl Sci. 2012;5:362–367
17. Thorndyke LE, Gusic ME, George JH, Quillen DA, Milner RJ. Empowering junior faculty: Penn State’s faculty development and mentoring program. Acad Med. 2006;81:668–673
18. Handelsman J, Pfund C, Lauffer S, Pribbenow C Entering Mentoring: A Seminar to Train a New Generation of Scientists. 2005 Madison, Wisc University of Wisconsin Press
19. Pfund C, Maidl Pribbenow C, Branchaw J, Miller Lauffer S, Handelsman J. Professional skills. The merits of training mentors. Science. 2006;311:473–474
20. Pfund C, House S, Asquith P, Spencer K, Silet K, Sorkness C Mentor Training for Clinical and Translational Researchers. 2012 New York, NY W.H. Freeman & Co.
21. Pfund C, House S, Spencer K, et al. A research mentor training curriculum for clinical and translational researchers. Clin Transl Sci. 2013;6:26–33
22. Fleming M, House S, Hanson VS, et al. The Mentoring Competency Assessment: Validation of a new instrument to evaluate skills of research mentors. Acad Med. 2013;88:1002–1008
23. Allen JM, Nimon K. A retrospective pretest: A practical technique for professional development evaluation. J Ind Teach Educ. 2007;44:27–42
24. Nimon K, Zigarmi D, Allen JM. Measures of program effectiveness based on retrospective pretest data: Are all created equal? Am J Eval. 2010;32:8–28
25. Neuendorf KA The Content Analysis Guidebook. 2002 Thousand Oaks, Calif SAGE Publications
26. Carnes M, Handelsman J, Sheridan J. Diversity in academic medicine: The stages of change model. J Womens Health (Larchmt). 2005;14:471–475
27. Prochaska JO, DiClemente CC. Stages and processes of self-change of smoking: Toward an integrative model of change. J Consult Clin Psychol. 1983;51:390–395
28. Abedin Z, Biskup E, Silet K, et al. Deriving competencies for mentors of clinical and translational scholars. Clin Transl Sci. 2012;5:273–280
29. University of Wisconsin–Madison, University of Wisconsin Institute for Clinical and Translational Science. . Research Mentoring: Cultivating Effective Relationships.
https://mentoringresources.ictr.wisc.edu. Accessed January 14, 2014
