Secondary Logo

Journal Logo


Measuring the Effectiveness of Faculty Mentoring Relationships

Berk, Ronald A. PhD; Berg, Janet MS, RN; Mortimer, Rosemary MS, MSEd, RN; Walton-Moss, Benita DNS, RN; Yeo, Theresa P. MSN, MPH, RN

Author Information
  • Free


Over the past 25 years, there has been a lack of clarity about the characteristics and outcomes of mentoring relationships, despite a growing body of research.1 “Mentor” has taken on numerous meanings and has been applied in a variety of corporate2–9 and educational10–15 contexts since its origin about 2,600 years ago, give or take a month or two: In The Odyssey, the ancient Greek poet Homer recounts the saga of Odysseus, the Greek king and warrior. When he knew he would be away from home for many years, he chose a trusted friend, Mentor, to educate, tutor, protect, and guide his son.

Since the mid-1970s, more than 20 definitions of mentoring or mentors have appeared in the literature.16–18 These definitions are extremely diverse,1,19 plus there is no professional consensus on any “acceptable” definition. Wrightsman's observation over 20 years ago still seems apropos today: “There is a false sense of consensus, because at a superficial level everyone ”knows“ what mentoring is. But closer examination indicates wide variation in operational definitions, leading to conclusions that are limited to the use of particular procedures.”20, pp 3–4

Another strategy to provide some meaning to the construct of mentorship has been to identify the basic elements or functions of the mentoring relationship. Jacobi1 distilled five elements in the mentoring relationship on which there is general agreement. A mentoring relationship (1) focuses on achievement or acquisition of knowledge; (2) consists of three components: emotional and psychological support, direct assistance with career and professional development, and role modeling; (3) is reciprocal, where both mentor and mentee (aka protégé) derive emotional or tangible benefits; (4) is personal in nature, involving direct interaction; and (5) emphasizes the mentor's greater experience, influence, and achievement within a particular organization.

The literature on mentoring within the health care field has run the gamut, from describing the value of mentoring in leadership,21–26 documenting a long-distance mentorship program,27 mentoring new faculty,28–30 using preceptors as mentors,31 and determining participation in mentoring relationships,32 to surveying the extent of administrative support for mentoring.33 Although much has been written on mentoring in health care, the research has not addressed the effectiveness of the mentoring relationship in the academic setting or the tools to measure that effectiveness.

Formal and informal mentoring programs have been popping up in colleges and universities nationwide, especially in medical schools such as ours.34 There are even a few books that describe guidelines for developing such programs.9,35–37 Unfortunately, criteria for evaluating the effectiveness of these programs are either not reported24,27 or not quantifiable.32,33 Within the context of this need, the Ad Hoc Faculty Mentoring Committee at Johns Hopkins University School of Nursing undertook to define the construct of “mentorship” and to develop new generic instruments to measure the effectiveness of a faculty mentoring relationship.

Interested faculty members in our school established the Ad Hoc Faculty Mentoring Committee to investigate faculty's mentoring activities as they related to the criteria for promotion to associate professor and professor ranks. Although no formal mentoring program existed at our school, evidence of successful mentoring was required for promotion. That evidence was being submitted in the form of letters, written by faculty and student mentees, which described the mentoring relationship with the faculty candidate. There were no specific guidelines or criteria to define the mentoring relationship or to evaluate the mentor's effectiveness. The evidence presented was nonstandardized and anecdotal.

The committee's four tasks were to (1) define “mentorship,” (2) specify concrete characteristics and responsibilities of mentors that are measurable, (3) develop an instrument that provides a comprehensive profile description of the mentoring relationship, and (4) build a scale that measures the effectiveness of that relationship in terms of specific outcomes that a mentee could evaluate.

“Mentorship”: The Construct

The following products related to the definition and characteristics of mentorship are the results of the committee's work.


Building on the previous definitions, but allowing for flexibility in the nature of the mentoring relationship, the committee proposed the following definition:

A mentoring relationship is one that may vary along a continuum from informal/short-term to formal/long-term in which faculty with useful experience, knowledge, skills, and/or wisdom offers advice, information, guidance, support, or opportunity to another faculty member or student for that individual's professional development. (Note: This is a voluntary relationship initiated by the mentee.)


The desirable characteristics of a faculty mentor include, but are not limited to, expertise, professional integrity, honesty, accessibility, approachability, motivation, respect by peers in field, and supportiveness and encouragement.


In order to put some teeth into the role of mentor, faculty must commit to certain responsibilities for which he or she will be held accountable by the mentees. Those concrete responsibilities are:

  • Commits to mentoring
  • Provides resources, experts, and source materials in the field
  • Offers guidance and direction regarding professional issues
  • Encourages mentee's ideas and work
  • Provides constructive and useful critiques of the mentee's work
  • Challenges the mentee to expand his or her abilities
  • Provides timely, clear, and comprehensive feedback to mentee's questions
  • Respects mentee's uniqueness and his or her contributions
  • Appropriately acknowledges contributions of mentee
  • Shares success and benefits of the products and activities with mentee

New Mentorship Effectiveness Instruments

Instruments related to mentoring programs in the 1980s relied on global questions about whether or not someone had a mentor38–40 or on a wide array of mentoring characteristics or functions.6,41–46 These tools mirrored the inconsistency in definitions and lack of consensus on a generic set of functions mentioned previously. In the 1990s, the health care research reporting instruments to measure the effectiveness of mentoring programs consisted of three studies,32,33,47 including only one in medicine.

Morzinski et al.47 described the evaluation of a formal mentoring program for junior faculty in academic family medicine. This program was based on skills deemed important for socialization in medicine and professional development. The mentees evaluated the impact of a mentoring program based on their achievement of three dimensions: development of career management skills, improved understanding of values and norms of the environment, and the ability to develop professional relationships. The study showed that junior faculty improved their professional and academic skills after participating in the mentoring program. The benefits were greater when the participants engaged in joint academic projects.

Critique of mentorship instruments

The current evaluation tools used in mentoring programs have several limitations in the context of the mentoring relationship. These tools are designed to evaluate only specific mentoring programs. They measure the importance of mentoring functions, and/or they measure the frequency of mentoring behaviors. And, these evaluation tools may or may not apply to faculty mentoring. Their questionnaire formats consist of short-answer constructed response and bipolar anchor scales with different anchors for each item. For example, the extreme anchors may be “Very Unsatisfied” and “Very Satisfied,” “Very Unimportant” and “Very Important,” “No Support” and “High Support,” and “No Impact” and “High Impact.” These bipolar anchors measure different characteristics about specific items. The scores on these items cannot be summed using a summated ratings procedure to produce subscale or scale scores. There are, however, other bipolar importance and activity scales with the same anchors. When anchors are presented only at the extreme ends of a scale continuum, the meaning of the scale's values and the interpretation of responses at points in between these extremes are unclear or ambiguous. Only the respondents will know the true meaning of these points.

Despite all of the aspects of mentoring that these instruments measure, none measured the critical dimension, “To what extent were any of the relationships effective?” A rating scale that evaluates the degree of effectiveness was needed. The challenge was to develop such a scale, plus address the structural limitations of previous scales.

Given the variation and complexity of faculty mentoring relationships, measuring effectiveness seems inextricably linked to the nature of each unique relationship. Consequently the committee developed two instruments: a questionnaire that described the characteristics of the mentoring relationship (albeit a profile of the relationship) as seen from the perspective of the mentee, and a formal rating scale that measured the effectiveness of the mentor against the aforementioned characteristics and responsibilities.

The Mentorship Profile Questionnaire

The Mentorship Profile Questionnaire (see Appendix A) was developed to describe the exact nature of the mentoring relationship and to specify the outcome measures produced from the relationship. The Description Section requests the mentee define the role of his or her mentor (teacher, counselor, advisor, sponsor, advocate, resource), the frequency and mode of communication, length of the relationship, and its strengths and weaknesses. The Outcomes Section asks the mentee to identify, describe, and provide supporting documents for the products of the relationship, such as publications, presentations or posters, new teaching methods, clinical expertise, conducting research, service activities, job change or promotion, and grant writing.

The Mentorship Effectiveness Scale

Item generation.

The committee constructed a formal rating scale to provide an efficient, comprehensive, and standardized tool for rating the mentorship experience and, especially, the effectiveness of the mentor (see Appendix B). Deriving the content from the pool of positive or desirable characteristics and responsibilities of mentors listed previously, 12 statements were generated to reflect a comprehensive assessment of the mentorship's effectiveness. The statements were written to meet established scale-item criteria,48–51 gleaned from a variety of classic sources on scale construction.52–56 The items were then reviewed by a five-member faculty committee for their psychometric form as well as for their mentor-characteristic substance to provide evidence of content-related validity. Item revisions required several iterations until unanimity by the committee was attained.

Scale structure.

A Likert-type summated rating scale format was used to elicit each mentee's responses to the items. A highly discriminating six-point agree–disagree continuum was developed: 0 = Strongly Disagree, 1 = Disagree, 2 = Slightly Disagree, 3 = Slightly Agree, 4 = Agree, 5 = Strongly Agree. These anchors seemed most appropriate to evaluate responses to a wide range of mentors’ characteristics. No uncertain or neutral position was presented in order to force an agree–disagree rating. Nunnally and Bernstein57 have indicated there is a slight advantage to using an even-numbered scale with no middle “undecided” position because a neutral position response gives no rating information. A “Not Applicable” option was also listed in case a characteristic was not appropriate for a particular mentor–mentee relationship.

Response bias.

Two types of response bias were of concern: acquiescence (or yea-saying) and the halo effect. Although these biases are not common with Likert-type scales,51 a mentee's close working relationship with his or her mentor may affect the rater's objectivity. The tendency to give positive responses to the “positive” characteristics, irrespective of the item content, or to rate the specific characteristics highly because of an overall positive impression of the mentor, can inflate the ratings and, consequently, favorably skew the responses. Given the nature of the mentor–mentee relationship, no psychometric antidote for this potential subjectivity and ratings bias appears possible. These effects should be considered in the interpretation of the final ratings.

Scale administration and scoring.

Mentors nominate mentees to complete this scale. Each mentee rates the extent to which the mentor exhibited each of the 12 characteristics or met the behavioral descriptions. Degree of agreement represents a qualitative rating, albeit an ordinal score value, from which the mentor's effectiveness could be inferred. The ratings may be presented item-by-item based on the 0–5-point quantitative scale or summed across all 12 items for a total rating, ranging from 0–60.

If several mentees rate the same mentor and the relationships are comparable, a median rating for the sample of mentees can be computed by item and for the total scale. This comparability, however, is quite rare. Most often, each mentor–mentee relationship is unique on one or more characteristics. This precludes aggregating ratings across mentees for a single mentor. If such ratings were combined, the results could be misleading and misrepresent the effectiveness of the mentoring.

Psychometric issues

Although there is considerable variation in the types of formal and informal mentoring programs in medical schools, minimal attention has been devoted to the development of instruments to evaluate mentors and the mentoring relationship. The Ad Hoc Faculty Mentoring Committee spent more than a year reviewing the literature and constructing the two tools described in this section. Despite the effort expended, there are built-in intractable psychometric issues that limit the collection of validity and reliability evidence. Such evidence is required by the Standards for Educational and Psychological Testing.58

Content-related validity.

The most important validity evidence is content related. The items on the Mentorship Profile Questionnaire and the Mentorship Effectiveness Scale must be congruent with the definition of mentoring and the domain of mentor characteristics and responsibilities in whatever mentoring activities occur. If there is a formal program, then the items must match the salient characteristics of the program as well. A panel of faculty members knowledgeable about mentorship should formally review the scale items against the mentoring characteristics and the panel should attain consensus or, preferably, unanimity.

Response bias.

As noted previously, acquiescence and halo biases can inflate the ratings by the mentees. Although the direction of the bias can be anticipated, the degree cannot be measured. Either or both sources of response bias can lower the validity of the ratings, and the inferences drawn from them, about mentor effectiveness.

Other validity and reliability evidence.

The most common indices of item analysis, validity, and reliability computed from sample data cannot be estimated for most scales of mentors’ effectiveness. Although a common set of criteria and scale items are administered using standardized procedures, typically each mentor–mentee relationship is unique. For example, the details of the relationships on the Mentorship Profile Questionnaire preclude the aggregation of ratings across mentees for the same mentor (see Appendix A). The ratings by each mentee are usually based on different role profiles. Hence, the ratings are not comparable and do not have the same meaning. Since a statistical sample of mentor ratings cannot be obtained, validity coefficients and standard indices of internal consistency reliability, such as coefficient alpha, as well as other group-based psychometric statistics, cannot be computed.


The research and experience on faculty mentoring relationships in academia, and medical schools in particular, over the past 25 years have produced lists of definitions, functions, and programs, but miniscule evidence of effectiveness. The concept of mentoring remains unclear and imprecise and instruments designed to evaluate mentoring programs rarely do. The effectiveness of formal and informal medical faculty mentoring programs intended to promote the professional growth of junior faculty and the academic success of students is based more on assumption than on demonstrated empirical evidence.

In view of this shaky foundation, the Ad Hoc Faculty Mentoring Committee at the Johns Hopkins University School of Nursing has contributed a generic definition, set of characteristics and responsibilities, and a mentorship profile and rating scale to measure faculty mentorship effectiveness. Although these products were developed in the absence of a formal mentoring program, the content, items, and instrument structure can be applied, or easily modified, to fit most informal as well as formal programs already in operation.

There is a critical need for research on mentoring that must address the definitional and conceptual issues plaguing this domain for years. Neither the empirical nor theoretical published research has kept pace with the development of mentoring programs. The scarcity of rating scales that directly measure characteristics of the mentoring relationship, essential to evaluate any program's effectiveness, requires immediate attention. Although the psychometric issues we have identified tend to limit quantification of results to person-, relationship-, and program-specific contexts, a deliberate effort should be devoted to tackling these scaling problems. Hopefully, our contribution will furnish a springboard, direction, and instrument prototypes to direct future research.


1 Jacobi M. Mentoring and undergraduate academic success: a literature review. Rev Educ Res. 1991;61:505–32.
2 Chao GT, Gardner PD. Formal and informal mentorship: a comparison on mentoring functions and contrast with nonmentored counterparts. Pers Psychol. 1992;45:619–36.
3 Collins GC, Scott P. Everyone who makes it has a mentor. Harv Bus Rev. 1978;56(4):89–101.
4 Dreher GF, Ash RA. A comparative study of mentoring among men and women in managerial, professional, and technical positions. J Appl Psychol. 1990;75:539–46.
5 Jeruchim J, Shapiro P. Women, Mentors, and Success. New York: Fawcett Columbine, 1992.
6 Kram KE. Mentoring at Work: Developmental Relationships in Organizational Life. Glenview, IL: Scott, Foresman, 1985.
7 Mobley GM, Jaret C, Marsh K, Lim YY. Mentoring, job satisfaction, gender, and the legal profession. Sex Roles. 1994;31(1-2):79–98.
8 Roche GR. Much ado about mentors. Harv Bus Rev. 1979;57(1):14–28.
9 Scandura TA. Mentorship and career mobility: an empirical investigation. J Organ Behav. 1992;3(2):69–174.
10 Daloz LA. Mentor: Guiding the Journey of Adult Learners. Indianapolis, IN: Wiley/Pfeiffer, 1999.
11 Davidson MN, Foster-Johnson L. Mentoring in the preparation of graduate researchers of color. Rev Educ Res. 2001;71:549–74.
12 Miller A. Mentoring Students and Young People: Handbook of Effective Practice. London: Kogan Page, 2002.
13 Schoenfeld AC, Magnan R. Mentor in a Manual: Climbing the Academic Ladder to Tenure. 2nd ed. Madison, WI: Magna Publications, 1994.
14 Villani S. Mentoring Programs for New Teachers: Models of Induction and Support. Thousand Oaks, CA: Corwin Press, 2002.
15 Wang J, Odell SJ. Mentored learning to teach according to standards-based reform: a critical review. Rev Educ Res. 2002;72:481–546.
16 Blackwell JE. Mentoring: an action strategy for increasing minority faculty. Academe. 1989;75:8–14.
17 Moore KM, Amey MJ. Some faculty leaders are born women. In: Sagaria MAD (ed). Empowering Women: Leadership Development Strategies on Campus. New Directions for Student Services. San Francisco: Jossey-Bass, 1988:39–50.
18 Moses YT Black women in academe: issues and strategies. Paper presented at the Conference of the Association of American Colleges, Washington, DC, 1989. ERIC Document Reproduction Services No. ED 311-817.
19 Merriam S. Mentors and protégés: a critical review of the literature. Adult Educ Q. 1983;33(3):161–73.
20 Wrightsman LS Research methodologies for assessing mentoring. Paper presented at the Conference of the American Psychological Association, Los Angeles, CA, 1981. ERIC Document Reproduction Service No. ED 209-339.
21 Boyle C, James SK. Nursing leaders as mentors: how are we doing? Nurs Adm Q. 1990;15(1):44–8.
22 Hamilton MS. Mentorhood: A key to nursing leadership. Nurs Leadersh. 1981;4(1):4–13.
23 Larson BA. Job satisfaction of nursing leaders with mentor relationships. Nurs Adm Q. 1986;11(1):53–60.
24 Madison J. The value of mentoring in nursing leadership: a descriptive study. Nurs Forum. 1994;29(4):16–23.
25 Rawl SM, Peterson LM. Nursing education administrators: level of career development and mentoring. J Prof Nurs. 1992;8(3):161–9.
26 White JF. The perceived role of mentoring in the career development and success of academic nurse administrators. J Prof Nurs. 1988;4(3):178–85.
27 Owens BH, Herrick CA, Kelley JA. A prearranged mentorship program: can it work long distance? J Prof Nurs. 1988;14(2):78–84.
28 Brown H.N. Mentoring new faculty. Nurs Educ. 1999;24(1):48–51.
29 DeJong C, Hartman M, Fisher-Hoult J. Mentoring new faculty. J Staff Prog Organ Dev. 1994;12(1):41–9.
30 Genrich SJ, Pappas A. Retooling faculty orientation. J Prof Nurs. 1997;13(2):84–9.
31 Bellinger SR, McCloskey JC. Are preceptors for orientation of new nurses effective? J Prof Nurs. 1992;8:321–7.
32 Short JD. Profile of administrators of schools of nursing, Part II: Mentoring relationships and influence activities. J Prof Nurs. 1997;13(1):13–8.
33 Kavoosi MC, Elman NS, Mauch JE. Faculty mentoring and administrative support in schools of nursing. J Nurs Educ. 1995;34:419–26.
34 Fried LP, McDonald SM. Mentoring: A Report from the Task Force on Women's Academic Careers in Medicine. Baltimore, MD: Department of Medicine, The Johns Hopkins University, 1997.
35 Reinarz AG, White ER (eds). Beyond Teaching to Mentoring. Indianapolis, IN: Wiley/Pfeiffer, 2001.
36 Vance C, Olson RK (eds). The Mentor Connection in Nursing. New York: Springer, 1998.
37 Zachary LJ. The Mentors Guide: Facilitating Effective Learning Relationships. Indianapolis, IN: Wiley/Pfeiffer, 2000.
38 Burke RJ. Mentors in organizations. Group Organ Stud. 1984;9:353–72.
39 Fagenson EA. The mentor advantage: perceived career/job experiences of protégés versus non-protégés. J Organ Behav. 1989;10:309–20.
40 LeCluye EE, Tollefson N, Borgers SB. Differences in female graduate students in relation to mentoring. Coll Stud J. 1985;10:411–5.
41 Busch JW. Mentoring in graduate schools of education: mentors’ perceptions. Am Educ Res J. 1985;22:257–65.
42 Erkut S, Mokros JR. Professors as models and mentors for college students. Am Educ Res J. 1984;21:399–417.
43 Knox PL, McGovern TV. Mentoring women in academia. Teach Psychol. 1988;15:39–40.
44 Kogler-Hill SE, Bahniuk MH, Dobos J, Rouner D. Mentoring and other communication support in the academic setting. Group Organ Stud. 1989;14:355–68.
45 Noe RA. An investigation of the determinants of successful assigned mentoring relationships. Pers Psychol. 1988;41:457–79.
46 Riley S, Wrench D. Mentoring among women lawyers. J Appl Soc Psychol. 1985;15:374–86.
47 Morzinski JA, Diehr S, Bower DJ, Simpson DE. A descriptive, cross-sectional study of formal mentoring for faculty. Fam Med. 1996;28:434–8.
48 Berk RA. The construction of rating instruments for faculty evaluation: a review of methodological issues. J Higher Educ. 1979;50:650–69.
49 DeVellis RF. Scale Development: Theory and Applications. 2nd ed. Thousand Oaks, CA: Sage Publications, 2003.
50 Netemeyer RG, Bearden WO, Sharma S. Scaling Procedures. Thousand Oaks, CA: Sage Publications, 2003.
51 Streiner DL, Norman GR. Health Measurement Scales: A Practical Guide to Their Development and Use. 2nd ed. New York: Oxford University Press, 1995.
52 Edwards AL. Techniques of Attitude Scale Construction. New York: Appleton-Century-Crofts, 1957.
53 Likert R. A technique for the measurement of attitudes. Arch Psychol. 1932;140:44–53.
54 Payne SL. The Art of Asking Questions. Princeton, NJ: Princeton University Press, 1951.
55 Thurstone LL, Chave EJ. The Measurement of Attitude. Chicago: University of Chicago Press, 1929.
56 Wang KA. Suggested criteria for writing attitude statements. J Soc Psychol. 1932;3:367–73.
57 Nunnally J, Bernstein IH. Psychometric Theory. 3rd ed. New York: McGraw-Hill, 1994.
58 AERA (American Education Research Association), APA (American Psychological Association), and NCME (National Council on Measurement in Education) Joint Committee on Standards. Standards for Educational and Psychological Testing. Washington, DC: AERA, 1999.

Appendix A

Mentorship Profile Questionnaire Developed by the Ad Hoc Faculty Mentoring Committee, Johns Hopkins University School of Nursing

Appendix B

Mentorship Effectiveness Scale Developed by the Ad Hoc Faculty Mentoring Committee, Johns Hopkins University School of Nursing
© 2005 Association of American Medical Colleges