Fleming, Michael MD; House, Stephanie MA; Hanson, Vansa Shewakramani MS; Yu, Lan PhD; Garbutt, Jane MD; McGee, Richard PhD; Kroenke, Kurt MD; Abedin, Zainab MPH; Rubio, Doris M. PhD
Skilled research mentors are critical to the training of new investigators.1 Although studies indicate that effective research mentors exhibit a variety of characteristics and skills,2–8 there is limited information about how to determine and measure the competencies that are crucial for mentors in the field of clinical and translational science.9–11
How, then, do we know if someone has the skill set to be an effective research mentor?12,13 What measures can be used to evaluate the effectiveness of mentor training workshops? How can program leaders decide whether to encourage a young investigator to seek the mentorship of a given established investigator?
In this article, we describe how the Clinical and Translational Science Awards (CTSA) Mentor Working Group approached the problem of designing, administering, and evaluating an instrument to assess skills of research mentors in clinical and translational science. The primary aim of the study reported here is to describe the psychometric properties of the new 26-item measure, the Mentoring Competency Assessment (MCA).
We developed the MCA to serve as the primary outcome measure for a national trial of an educational intervention involving more than 200 pairs of mentors and mentees at 16 U.S. universities. A secondary goal of the trial was to determine skills norms for research mentors working in the area of clinical and translational science.
In the CTSA research mentor training trial (hereafter, the trial), 283 mentor–mentee pairs were assigned either to participate in the experimental eight-hour training curriculum designed to improve the skills of research mentors or to receive no training. (The training was limited to the mentors.) The 26-item MCA served as the primary outcome measure for the trial. Before and after the intervention, researchers used the MCA to ask mentors to rate their own skills and mentees to rate the skills of their mentors. In this study, we analyzed the baseline data collected in 2010 to evaluate the psychometric properties of the MCA.
A CTSA administrative supplement awarded to the University of Wisconsin–Madison (UW-Madison) supported the overall project and facilitated the recruitment of investigators and staff members at 15 additional sites* to conduct the trial and to develop and test the MCA, as described below. Each institution involved in the project also contributed resources to participate in the trial and testing of the MCA.
The institutional review board (IRB) at UW-Madison approved the study and served as the primary IRB of record for the multisite trial. Investigators at the other 15 sites received IRB approval from their own institutions.
Project overview and participants
In fall 2009, three groups began to work simultaneously to develop the research methodology for the trial. One of the groups focused on developing the workshop curriculum (educational intervention) and the accompanying training manual, entitled Mentor Training for Clinical and Translational Researchers.11
Another group—consisting of the 16 site project investigators, many of whom were members of the CTSA Mentor Working Group—focused on recruiting convenience samples of mentor–mentee pairs at their own institutions. The investigators generally recruited the mentors first, by e-mail or direct contact, and then asked them to identify all of the qualifying mentees with whom they currently worked. (In a few cases, investigators recruited the mentees first and asked them to identify their mentors.) Faculty mentors of trainees who were conducting clinical/translational research at least 50% of the time were eligible. The number of mentor–mentee pairs recruited at each site ranged from 12 to 27. The final sample consisted of 283 mentor–mentee pairs.
The third group was a national measures working group composed of researchers, evaluators, and training leaders and led by the first author (M.F.). As members of this group, we were charged with designing an instrument to assess mentoring competencies and then evaluating the psychometric properties of the instrument. Our work is the focus of this report.
Development of the MCA
First, our group reviewed the literature and examined existing mentoring assessment instruments. We identified three instruments that focused on research mentors and provided sufficient information on the development and testing of the measure.9,10,14 One of these was the Mentorship Effectiveness Scale, developed by researchers in the School of Nursing at Johns Hopkins University.14 We believed that the primary limitations of this instrument were the ceiling effect of its scale and the paucity of information about its psychometric properties. We also assessed two instruments from mentoring programs: the Wisconsin Mentoring Seminar9 and the Mentor Development Program at the University of California, San Francisco.10 Although the curriculum working group adopted some of the Wisconsin program’s curriculum for the trial, we believed that these two programs’ instruments were not adequately aligned with the trial curriculum and were not suited to our more diverse and senior audience of clinical and translational researchers. Therefore, although we used these instruments to provide a framework for our assessment instrument, we built the MCA around the competencies used for the trial.11
Second, we participated in numerous conference calls during a six-month period in 2010 as members of our group proposed iterations of skills to be covered in the MCA. We aligned the proposed skills with the learning objectives of each of the six mentor competencies targeted for training during the trial’s eight-hour workshop, as described in Mentor Training for Clinical and Translational Researchers.11 These competencies were maintaining effective communication, aligning expectations, assessing understanding, addressing diversity, fostering independence, and promoting professional development. They were similar to competencies targeted in previous successful mentor training programs.9,10 We then developed two to six items to cover skills in each of the competencies, based on the literature and expert opinion of our group. The goal, as noted above, was to use the MCA to assess the level of these competencies in mentors before and after the trial’s educational intervention.
Third, before the MCA was used in the trial, it was reviewed by UW Survey Research Center staff with expertise in survey design. We then conducted cognitive interviews15,16 with six mentors and six mentees to better assess the instrument’s reliability. In these interviews, we asked participants to “think out loud” about their responses to the items and asked targeted questions to assess consistency of interpretation. After making adjustments based on this feedback, we conducted several traditional pilot interviews, primarily to assess flow and timing, before making final refinements to the MCA. All participants in the cognitive and pilot interviews were excluded from participation in the trial.
Administering the MCA
At the 16 participating sites, 24 researchers were responsible for administering the MCA via face-to-face interviews with trial participants. To ensure consistency and high follow-up rates, the UW-Madison team provided each researcher with extensive training, including reviewing the recruitment and data collection protocols, modeling an interview, practicing interviews, providing feedback on performance, and using conference calls to ensure data integrity.
The final MCA consisted of 26 items designed to evaluate research mentoring competency in six areas (maintaining effective communication, aligning expectations, assessing understanding, addressing diversity, promoting professional development, and fostering independence). When the study researchers administered the MCA, they began by asking each mentor to “please rate how skilled you feel you are in the following areas” and each mentee to “please rate how skilled your mentor is in the following areas.” Both the mentors and mentees responded using a seven-point Likert-type scale in which 1 = “not at all skilled,” 4 = “moderately skilled,” and 7 = “extremely skilled.” Mentees could also choose 0 for “not observed.” (For the complete mentor and mentee instruments, see https://mentoringresources.ictr.wisc.edu/EvalTemplates.)
The primary data used for this report were collected in summer and fall 2010, prior to the intervention. These baseline data were collected on paper, and hard copies were sent to UW-Madison for coding and analysis.
To characterize the study sample, we used descriptive statistics. To characterize mentor competency levels, we determined the mean score and standard deviation (SD) for each of the 26 items as ranked by the mentor and mentee groups.
To assess the reliability (internal consistency) of the instrument used by mentors and the instrument used byd mentees, we calculated the coefficient alpha for each group.17
To measure the construct validity of the instruments, we conducted confirmatory factor analysis.18 In this analysis, we used maximum likelihood (ML) estimation to assess how well the 26 items measured the six domains (i.e., the six latent constructs). ML is desirable for its asymptotic properties and conduciveness to hypothesis testing, given its assumption that all eigenvalues are greater than zero.19 ML methods generate parameter estimates that are most likely to produce the observed correlation matrix, assuming the sample is from a normal distribution. The reproduced correlation matrix is compared with the actual correlation matrix through an iterative process that ideally concludes with convergence between the two matrices.20
In fitting a measurement model, we tested whether each competency ought to be measured together or separately through an additive scale or common factor score. Given that previous mentoring research focused on specific elements underlying the concept of mentoring,9,10 we also searched for commonalities among items grouped by competency. Although there are merits to aggregating the individual items to form additive indices, we found examining the parameter estimates for each item to be a superior analytic strategy. This strategy enabled us to assess the validity of the factor structure for the MCA. We used Mplus18 to calculate the final standardized model results.
Table 1 summarizes the four goodness-of-fit statistics that we used in the analysis21,22: chi-square, root mean square error of approximation (RMSEA), comparative fit index (CFI), and Tucker-Lewis index (TLI). We primarily relied on the RMSEA of < 0.08 as recommended by Browne and Cudeck22 to assess model fit. In addition, because chi-square is sensitive to sample size we used the relative chi-square test (chi-square divided by degrees of freedom [df]), with < 5 indicating an acceptable goodness of fit.23
Characteristics of the study sample
The sample consisted of 283 mentors and 283 mentees. The mean ages of the mentor and mentee groups were 50.5 and 35.9, respectively (Table 2). The gender distribution varied; more men participated as mentors (n = 170; 60.1%), whereas more women participated as mentees (n = 165; 58.3%). The racial and ethnic distribution of the groups also varied, with 257 mentors (90.8%) and 208 mentees (73.5%) identifying themselves as white. Hispanic/Latino participants made up the largest minority groups for both mentors (n = 20; 7.1%) and mentees (n = 32; 11.4%). The majority of the mentors (n = 161; 56.9%) had achieved the academic rank of professor. Many mentees held the academic rank of assistant professor (n = 118; 41.7%), although almost half (n = 139; 49.1%) were students or fellows. Of the mentees, 68 (24.0%) had a professional degree (e.g., MD, DVM, PharmD), 75 (26.5%) had a PhD, and 103 (36.4%) had both a professional degree and a PhD or master’s degree (e.g., MD/PhD, DVM/MS). Only 59 mentors (20.8%) and 52 mentees (18.4%) had participated in a prior mentor training workshop.
Scores for mentor skills
There was considerable variation in the mean scores for the 26 mentor skills assessed on the MCA, as rated by the mentors themselves and by their mentees (Table 3).
The mentors’ ratings of their own skills were consistently lower than the mentees’ ratings of the mentors’ skills. For example, mentors’ mean self-reported scores were below 5 for eight items; they reported the lowest scores for helping establish a work/life balance (4.41), pursuing strategies to improve communication (4.52), and accounting for biases and prejudices (4.68). In contrast, mentees’ mean ratings of mentors’ skills did not fall below 5 for any item.
Although mentors’ self-reported mean score was over 6 for only one item (acknowledging mentees’ professional contributions, 6.05), mentees gave mean scores higher than 6 to six items: accounting for different backgrounds of mentors and mentees (6.32), developing a trusting relationship (6.17), acknowledging mentees’ professional contributions (6.16), providing constructive feedback (6.12), setting research goals (6.09), and helping mentees acquire resources (6.01).
Psychometric properties of the MCA
The coefficient alpha scores for completion of the 26 items by the mentor group and the mentee group were 0.91 and 0.95, respectively. The coefficient alpha scores for the six competencies as rated by the mentor group versus the mentee group were as follows:
* Maintaining effective communication (6 items), 0.62 versus 0.59
* Aligning expectations (5 items), 0.76 versus 0.73
* Assessing understanding (3 items), 0.72 versus 0.72
* Addressing diversity (2 items), 0.65 versus 0.73
* Fostering independence (5 items), 0.91 versus 0.90
* Promoting professional development (5 items), 0.80 versus 0.62
Although some of the competency subscales had alpha coefficients < 0.7, this was not unexpected because of the small number of items.
All of the correlation coefficients for the mentor group and mentee group were in the expected direction, with the expected magnitude of relationships among the factors (Table 4).
We used Mplus to test our factorial model, as illustrated in Figure 1, with the items assigned to the constructs they were intended to measure.
For the mentor instrument, the hypothesized model with the six latent constructs and 26 items resulted in an acceptable fit to the data: chi-square = 663.20; df = 284; P < .001; relative chi-square = 2.34, RMSEA = 0.069 (90% confidence interval [CI], 0.062–0.076); CFI = 0.85; and TLI = 0.83. The correlations among the six competencies were high, ranging from 0.49 to 0.87 (Table 4). All of the parameter estimates for the individual items were significant, with standardized factor loadings ranging from 0.32 to 0.81 (Table 5).
For the mentee instrument, the hypothesized model with the six latent constructs and 26 items also resulted in an acceptable fit to the data: chi-square = 840.62; df = 284; P < .001; relative chi-square = 2.96; RMSEA = 0.080 (90% CI, 0.063–0.077); CFI = 0.87; and TLI = 0.85. The correlations among the six competencies were high, ranging from 0.58 to 0.92 (Table 4). Again, all of the parameter estimates for the individual items were significant, with standardized factor loadings ranging from 0.56 to 0.86 (Table 5).
Our findings support use of the MCA in the evaluation of research mentors who work in the area of clinical and translational science. In addition to demonstrating that the MCA has reliability and validity, our findings provide preliminary norms derived from a large national sample of mentors and mentees—norms that could be useful to program directors and others who wish to assess the mentoring skills of their faculty. Unlike previous instruments that rely on either mentors’ self-evaluation or mentees’ evaluation of mentors, the MCA can be used to assess mentor performance as perceived by both mentors and mentees. As a dual-purpose measure, it can identify areas in which mentor and mentee assessments diverge, providing information that could help target particular areas for faculty development initiatives.
The MCA was designed and tested for use with a primary mentor. However, most new investigators work with multiple mentors who guide their research and help them build sustainable research programs. Some of these secondary mentors may focus on specific areas with mentees (e.g., science, grant writing, career development, networking). From a competency perspective, though, all research mentors should demonstrate the 26 skills included in the MCA.
Results from both the mentor and mentee groups supported the measurement model (Figure 1). Our data suggest that the MCA’s 26 items are able to assess the skill levels of mentors across six competency domains. Although these domains are highly correlated, the fact that the model showed a good fit with the data suggests that they are nonetheless distinct. Therefore, we recommend using the MCA for both self-evaluation by mentors and evaluation of mentors by mentees. Of note, the MCA’s reliability coefficients were quite high; the overall alpha for both the mentor and mentee groups was greater than 0.90.
We believe there are several ways in which the MCA could be used to benefit research mentoring programs in clinical and translational science. First, the MCA could be used to provide direct feedback to mentors about their strengths and the areas in which they need improvement, especially when the instrument is completed by mentees. Second, the MCA could be used to assess the efficacy of mentor training curricula targeting the six competencies covered by the instrument. Third, simultaneous completion of the MCA by a mentor and his or her mentee may identify areas in which these individuals are “not on the same page” and could benefit from further discussion about the reasons for their divergent assessments. Indeed, not all mentees have the same needs with respect to mentoring, so the MCA could be used to initiate a discussion about the mentee’s needs.
This study had several limitations, including its use of cross-sectional data, absence of correlation with other measures of mentor skills, and the restriction to six mentor competency domains. Further, our sample focused on individuals working in clinical and translational science and drew heavily from mentors who were at the professor level. Additional research is needed to confirm the applicability and utility of the MCA in samples involving mentors at in different stages in their academic career (e.g., doctoral students, fellows, assistant professors) and from different areas (e.g., basic science) and to assess the generalizability of the MCA to other settings and other mentor training curricula. Further, as illustrated in our mentee data (see Table 3), there is a ceiling effect that limits positive changes in the MCA skills ratings after educational interventions. We had hoped to see lower baseline scores. The mentees’ high baseline ratings may reflect the nature of convenience samples and overall quality of the sample’s research mentors. Finally, when using the MCA to assess mentor skills, it will be important to consider other mentee outcomes, such as mentee publications, grants, and additional measures of career success.24,25 The MCA should be considered only one measure of mentor performance.
This study also had several strengths that deserve mention. These include the successful development of a skills inventory based on six competencies, use of a national expert panel to develop the instrument, pilot-testing and refinement of the MCA, large sample size, participation of senior research mentors at 16 sites with varying research cultures and mentoring expectations, and use of trained interviewers to ensure consistency and understanding of the instructions. In light of these strengths, we believe that the MCA provides department chairs and leaders of career development programs with a validated tool to assess mentor performance to help ensure the success of their young investigators. Additional studies with larger, more diverse samples of mentors that include more assistant professors may provide additional information on the reliability and validity of the MCA.
Acknowledgments: The authors are grateful to Stephanie (Schiro) Wilson, Kimberly Spencer, Anne Taylor, and Don Yarbrough for their work on instrument development and implementation. In addition, the authors thank the official collaborators and participating investigators. These include the following CTSA research mentor training trial site leaders: Adriana Báez Bermejo (University of Puerto Rico), Melissa D. Begg (Columbia University), Philip Binkley (The Ohio State University), Barbara F. Brandt (University of Minnesota), Ellen Burnham (University of Colorado, Denver), Paula Carney (Northwestern University), Suzann Campbell DeLapp (University of Illinois, Chicago), Julie Eichenberger-Gilmore (University of Iowa), Janice Gabrilove (Mount Sinai Medical Center), Jane Garbutt (Washington University in St. Louis), Rubén García García (University of Puerto Rico), W. Charles Huskins (Mayo Clinic), Kurt Kroenke (Indiana University), Bruce Rollman (University of Pittsburgh), Gary Rosenthal (University of Iowa), Julie Schweitzer (University of California, Davis), Eugene D. Shapiro (Yale University), and Stephanie Vecchiarelli (The Ohio State University). The authors also thank the following CTSA research mentor training trial site research assistants: Susan Anderson, Joanna M. Barr, Lauren Boyle, Ann Castellano, Antaeus Coe, Fred De Pourcq, Lida Gilbertson, Blair Gonsenhauser, Nancy Hodgkins, LaToya Howard, Jessica Hyink, Nicolas Licht, Melissa Logsdon, Stephanie McCalley, Reid McCamish, Vardhini Mohan, Madrid Poultney, Suyapa Sarmiento, Miyisha Slay, Lauren Thomann, and Tracy Yale.
Funding/Support: This research was supported by National Institutes of Health (NIH) grants UL1 RR025011, K24AA015390, UL1 RR02415, and UL1TR000005.
Other disclosures: None.
Ethical approval: The institutional review board (IRB) of the University of Wisconsin at Madison approved the study and served as the primary IRB record for the multisite trial. The other 15 sites received IRB approval at their respective institutions.
Disclaimer: The ideas expressed in this report are solely the views of the authors and do not necessarily represent the official views of the National Center for Research Resources or National Institutes of Health.
* Participating institutions included Columbia University; Indiana University; Mayo Clinic; Mount Sinai School of Medicine; Northwestern University; The Ohio State University; University of California, Davis; University of Colorado, Denver; University of Illinois, Chicago; University of Iowa; University of Minnesota; University of Pittsburgh; University of Puerto Rico; Washington University in St. Louis; and Yale University. Cited Here...