Searle, Nancy S. EdD; Teal, Cayla R. PhD; Richards, Boyd F. PhD; Friedland, Joan A. MD; Weigel, Nancy L. PhD; Hernandez, Rachael A.; Lomax, James W. MD; Coburn, Michael MD; Nelson, Elizabeth A. MD
In this article, we provide the rationale, design, and description of a unique teaching award that, since 2001, has enhanced the teaching environment and has also become highly valued by the Baylor College of Medicine (BCM) promotions and tenure (P&T) committee as a resource when determining a faculty member’s readiness for promotion. The Fulbright and Jaworski L. L. P. Faculty Excellence Award (hereafter, the F&J Award) is unique because it is self-nominating and standards based, not norm based.* Those who can demonstrate to the peer reviewers that they meet the F&J Award’s published standards receive the award, no matter the number of applicants. We also present the results of our internal evaluation of the program in 2010, which includes findings from an online survey of recipients of the award and interviews of various stakeholders.
Need for the award
Academic health centers have multiple missions, including caring for patients, discovering new knowledge, and training the next generations of clinicians and medical scientists. Historically, research has eclipsed the other missions and has shaped the promotion criteria and processes, making it difficult for faculty with a focus on education to get promoted.1–3 The evaluation of educators for promotion is often conducted primarily by those extensively involved in research, where the number of papers published and the amount of grant funds received are of highest value, or by those involved in clinical effort, where revenue and clinical leadership are of highest value. It is not surprising, then, that accomplishments in education are often not valued on a par with research or clinical accomplishments in promotions decisions.
The problem described above is not a new one. In the late 20th century, Ernest L. Boyer,4 then president of the Carnegie Foundation for the Advancement of Teaching, addressed it by stating that in addition to the scholarship of discovery (i.e., research), there are three other types of scholarship: integration, application, and teaching. Glassick et al5 expanded on Boyer’s work by presenting six criteria for promoting quality of all types of scholarship: have clear goals, be adequately prepared, use appropriate methods, achieve meaningful results, communicate results effectively, and reflectively critique the work in planning subsequent work. Simpson et al3 report the recommendations from a 2006 consensus conference on educational scholarship that built on Glassick’s criteria by requiring not only documentation of the quantity and quality of educational activities, but the demonstration of two types of engagement with the educational community: (1) drawing on that which is known in the community (i.e., using best practices that have passed the litmus test of peer review), and (2) contributing something of value back to the community (i.e., sharing products or techniques for others to use that have also passed the test of peer review).3
In response to curricular changes in the mid-1990s,6 BCM began an extensive program to build a more cohesive community in support of the educational mission of the college.7 One important component of this program, developed to recognize excellence in faculty educational contributions, was the F&J Award,8,9 in which educators recognize the accomplishments of other educators using the litmus test of peer review. Faculty who earn the award automatically become members of the college’s Academy of Distinguished Educators for five years. The developers of the award included a small group of very dedicated faculty, including four of the authors (B.R., J.F., J.L., M.C.), who were concerned with the enhancement and recognition of teaching at the college. They worked for over two years with the leadership of the college to create the award.
Description and review process
The review panel meets three times per year. These meeting dates and the final date for each review cycle when materials will be accepted are posted on the Web. Information about the award process is published on the BCM Web site and is distributed frequently by some department chairs to their faculty. Workshops on how to prepare an application are presented three times per year by the program’s director (N.S.), with the dates published on the online faculty development workshop calendar and on BCM’s paper calendar.
All faculty who believe that they can meet the designated standards of quality, quantity, and breadth to receive recognition for their sustained (at least three to five years) and exemplary educational contributions to the college are encouraged to apply. This is a self-nominating process. If a faculty member believes that he or she can make a case that his or her contributions meet or exceed the award’s published standards, that individual may apply for the award by submitting a mini-portfolio. The mini-portfolio consists of a personal statement and structured summary of accomplishments, supporting documentation, and a curriculum vitae.8 The format of the mini-portfolio has been carefully matched to the type of evidence of quantity, quality, and breadth for each of the four categories. Using a peer-review process modeled after the National Institutes of Health review process for research proposals, a representative panel of respected faculty (both within and outside of BCM) rate each faculty member’s educational accomplishments as presented in the mini-portfolio for its quality, quantity, and breadth as compared with the F&J Award’s published standards (described below). Currently, there are 25 people on the review panel, but in each session approximately 15 reviewers participate. This process includes assigning each mini-portfolio to one primary and one secondary reviewer, who lead the discussion about the portfolio at the review panel meeting. The primary and secondary reviewers rate each submission with a maximum total of 100 points, using weighted criteria:
1. 50 points maximum for quality, judged using Glassick’s criteria of scholarship.5
* 5 points for clarity of goals
* 10 points for adequacy of personal preparation and ongoing self-reflection/improvement
* 35 points for adequacy of methods, quality of presentation, and results
2. 40 points maximum for quantity (i.e., amount of educational activity in a given category).
3. 10 points maximum for breadth (i.e., diversity of educational activity).
After discussing a mini-portfolio in the review panel meeting, all panel members assign an overall score to the mini-portfolio. Faculty whose portfolios are assigned an average score greater than or equal to 80, and who are given a score greater than or equal to 80 from at least 75% of the panelists, receive the award.
Reviewers make their ratings by comparing each mini-portfolio with a standard—not with each other. The standard for each award category is defined by three or four example mini-portfolios.8,9 These initial prototypes were developed between 1997 and 2001 and are based on the actual educational activities of noted teaching faculty at BCM. The F&J Award developers carefully equated quantity and quality within and across award categories. These prototypes establish the minimum amount of evidence of quality, quantity, and breadth needed to receive the award. The prototypes are published on the Web8 so that faculty can judge in advance if they believe that the sum of their evidence will compare favorably to the prototypes. The prototypes also serve as examples of the types of evidence one might want to include and the structure for creating a mini-portfolio.
For the first three years, award developers had a calibration meeting where all review panel members evaluated mock portfolios and justified their scores in each area to each other prior to each review panel meeting. Just as when reviewing grants, there can be some differences in scoring a submission. But, generally, the review panel members were, and continue to be, consistent in their scoring.9 In the past five years, the panel members have decreased sharply the amount of discussion and the length of time of the review panel meetings by submitting their scores to the chair of the review panel two days before the review panel session. When there is a great variance in the scores, the reviewers are asked to discuss their scores with each other to learn each other’s thinking. There is no requirement that reviewers change their scores in this situation.
When the award was first being designed, there were long discussions about whether there should be one composite award that covered all categories of the scholarship of teaching or whether there should be different categories. Award developers opted for the latter option for two primary reasons. First, they wanted to be more inclusive of faculty educational contributions and believed that some educators whose efforts were concentrated in just one area would be disadvantaged by a single composite award. Second, they wanted to create multiple opportunities for faculty involved in more than one area to receive recognition. Consistent with this second reason for using multiple categories, the developers also explicitly decided to limit evidence in any one portfolio to approximately five years of effort, thus creating the opportunity for faculty to apply for the award, even in the same category, approximately every five years.
In an environment that promotes teaching, it is essential that members of the P&T committee understand and value excellence in education, to the same degree that they understand and value excellence in research. The traditional method of using publications and grants to judge the quantity and quality of research has merit because it is based on a well-established system of peer review. In effect, the P&T committee defers much of its task of making judgments about faculty scholarship of discovery to other groups of qualified judges. The goal in establishing this faculty excellence award was to provide a system for peer review of educational accomplishments on par with peer review of research.
Peer review can take many forms and can occur at different points during professional activity. Usually, peer review in education is equated with direct observation and critique of a specific activity such as a lecture. This type of peer review is important but not equivalent to a synthesis of prior activities (e.g., a grant or publication). Consequently, the F&J Award’s developers sought to establish a form of peer review that also considered prior activities synthesized into a coherent presentation (i.e., the mini-portfolio) containing evidence specific to nationally accepted criteria for educational scholarship.3–5
When and how the F&J Award program was developed presented both opportunities and challenges. The development of the award began in 1997, just a few years after the work of Boyer4 and Glassick and colleagues.5 Although today many people emphasize the criteria of dissemination beyond the medical center for determining educational scholarship, even these early definers of educational scholarship understood that internal dissemination of excellent and innovative educational activities strengthens educational practice within the institution. Therefore, criteria for the F&J Award do not primarily emphasize dissemination of educational activities outside BCM in the categories of teaching and evaluation and educational leadership.
Award developers strongly believed that award selection criteria needed to be aligned with the college’s values and thereby promote realistic expectations for educators as they strive individually and collectively to make contributions consistent with those values. Illustrating this alignment is the apparent effect of using Glassick’s criterion about adequate preparation on faculty attendance at educational skill-building activities; such attendance has significantly increased over the years since the award has been in place.
Statistics on award recipients
At the time of the survey described in the next section, there were 1,700 full-time faculty at BCM, of whom 171 (10.1%) had received an F&J Award. Three hundred thirty-six F&J Awards have been granted to 219 individuals between spring 2001 and July 1, 2010. One hundred ninety-three awards (57.4%) were presented in teaching and evaluation, 83 (24.7%) in educational leadership, 49 (14.6%) in the development of enduring educational material, and 11 (3.3%) in educational research. Of the 219 individuals, 81 (37.0%) have received multiple awards, with 52 receiving two F&J Awards, 22 receiving three F&J Awards, and 9 receiving four F&J Awards.
At the time of our analysis, all 171 F&J Award recipients were active members of our faculty. We determined that for 130 individuals, the award may have had an influence on their promotion. These individuals had a rank of instructor, assistant, or associate professor at the time they received their first F&J Award, reached their current rank in 2002 or later, and were awarded the F&J Award in 2009 or earlier. Of the 130, 88 (65.6%) received an award before gaining their current rank (χ2 (1) = 16.3, P < .001). Of the current BCM faculty who have received at least one award, 110 (64.7%) are currently either associate or full professors. There is no statistical difference in the percentages of men and of women who have been promoted.
Internal Review of the F&J Award Program
To learn more about what recipients of the award, members of the P&T committee, and department chairs believe about the award, in 2010 we surveyed them either by an online survey or by in-person interviews. We gained approval from the BCM institutional review board to conduct the surveys.
All of the 171 F&J Award recipients who were faculty at BCM in 2010 were asked to complete an online survey that included questions about why they applied for the award and what effect they believed the award had on teaching at the college. One hundred ten of 171 (64.3%) recipients completed the survey. They were asked the importance of applying for their first F&J Award in three areas: promotion, recognition of educational activities, and academy membership. Sixty-seven (61%), 61 (56%), and 23 (21%), respectively, responded that they considered that applying in each of these areas was of “great importance,” and 16 (15%), 6 (6%), and 10 (9%), respectively, responded that they considered that applying in each area was of “little importance.” The majority of the other respondents considered that applying for the award was “important” in each of the three areas. Two open-ended questions were also asked: (1) What additional reasons did they have for applying for the award, and (2) what influence, if any, did they believe the F&J Award has had on teaching at the college? Selected responses can be seen in Tables 1 and 2. The survey also had an additional, open-ended question that asked if there was anything else faculty wanted to say about the award. Selected responses can be seen in Table 3.
We interviewed eight individuals positioned to provide insight into the effect of an award on faculty promotion, including four members of the P&T committee and four department chairs. One person was randomly selected to be interviewed from each of the following eight different groups:
* One basic scientist and one clinician who had been on the P&T committee since before the award was developed
* One basic scientist and one clinician who had been on the P&T committee only since the development of the award
* One basic science and one clinical department chair from whose department few, if any, faculty had applied for the award
* One basic science and one clinical department chair from which few if any faculty had applied for the award
Each interview consisted of six open-ended questions that were developed by two of the authors (N.S. and C.T.) and averaged 20 minutes in length. Interviews were transcribed and analyzed by two of the authors (N.S. and R.H.) for repeated themes and interrater reliability.
Interviews with members of the P&T committee
Both of the P&T committee members with extended service on the committee said that promotion for those involved in the educational mission of the college had improved since the establishment of and acceptance of the award by the P&T committee. When asked, “What effect has the F&J had on P&T?” the basic science member said, “The F&J serves as an independent stamp or label that signifies that they have objectively fulfilled the criteria of what we would want in an educator.” The clinical science member stated,
It’s a small percent of faculty—5% I think—that have it, so the award is something extraordinary. It’s leveled the playing field across departments. We usually think that two F&Js mean you are ready to get promoted to associate professor.… The F&J has increased the visibility and importance of education amongst research scientists and clinicians.
Both the basic science and clinical science members who had been on the P&T committee only since the development of the award reported that the F&J Award is important to signal the readiness of an applicant for promotion and/or tenure. When asked, “What effect has the F&J had on P&T?” the clinical science member responded: “A huge impact. The F&J [is] considered an objective measure. This program vigorously evaluated their contribution and quantifies it.”
When asked, “What effect has the F&J Award had on promotions?” the basic science member said: “A dramatic effect … particularly from assistant to associate professor and in the majority of cases from associate to full professor. In gaining tenure, the F&J is a gold stamp.” When he was asked why and how the F&J Award gained credibility with the P&T committee, he said:
About 10 years ago, there was a campaign by deans and others with influence to make education more important to faculty appointment and promotion. The F&J has been a very important tool because it’s a criterion-based award.… During the campaign, the committee became sensitized and values the rigor of that tool. With it, we can go forth comfortably with promotion.
When he was asked if there was anything else he wanted to say, the basic science member responded:
Before you get it, it’s an impetus to teach, and teach well.… It [the award] is a valuable tool to get faculty members involved in teaching and to improve teaching at the school.
Interviews with department chairs
Both of the department chairs with two or more F&J Award recipients in their departments encouraged their faculty to apply for an award and valued the award for recognizing teaching in their individual departments.
In response to the question: “Do you believe that it is important for teaching faculty to receive an F&J? If so, why?” the basic science chair responded:
Teachers tend to be considerably less recognized in a medical school than they should be. Anything that could be done to enhance recognition is good.
When he was asked if there was anything else he wanted to say about the F&J Award, he responded:
I think it’s something that is a little different, that we’ve done here. A couple of other schools have this recognition but it is more organized here.
The two department chairs who had few faculty who had received an F&J Award had little understanding of the award and its appropriateness for their faculty. For example, the clinical science chair was not aware that faculty must nominate themselves. The basic science chair was unaware that mentorship and other one-on-one teaching in the lab are included in recognized educational activities. He stated that “most of my faculty don’t do enough regular teaching to be seriously competitive for those awards.”
Individual recipients have stated their beliefs that the award is good for the institution in a variety of ways. It encourages reflection on teaching, validates teaching as important, increases passion for teaching, motivates teachers to seek excellence in teaching, and recognizes teaching a variety of learners from pre-K to faculty. Of the 214 open-ended responses to our survey questions of faculty, more than half of them made comments about the value of the award and its effect on promotion. There were only four neutral responses, and those were to the question concerning the influence of the award on the teaching environment. There was only one negative response in all 214 responses.
Although the medical education literature indicates that it is difficult for those involved in education to receive appropriate credit for their educational activities from P&T committees,10,11 our data indicate that faculty who receive an award for teaching are being promoted at our institution.
Current promotion requirements at our institution state that there must be documentation of excellence in a primary mission with evidence of good to excellent performance in at least one of the other academic mission areas of education, research, and patient care. One possible explanation for the promotion of educators who have received an award is that we have educators rigorously reviewing educators. The process is respected and accepted by those making promotion and tenure decisions. Early discussion and presentations introduced our P&T committee to the F&J Award process, which contributed to their acceptance of the program as a rigorous measure. We also invited two members of the P&T committee to become members of the F&J review panel.
In 1990, Boyer4 stated that scholarship should be broadened beyond an emphasis on discovery and first proposed the idea of the scholarship of teaching. To engage in the scholarship of teaching, one must study teaching and learning systematically, reflect on his or her teaching, and make it public for others to learn from or build on. Building on the seminal work of Boyer, Glassick et al5 proposed an answer to the question “How do we judge the work of faculty as scholars?” in the form of six criteria for scholarship. Boyer’s requirement for the scholarship of teaching and Glassick’s six criteria for scholarship are included in the preparation and evaluation of our mini-portfolio, enhancing its rigor.
Another explanation for the promotion of educators who have received the award is that the existence of the award has expanded the understanding of learners to include the entire educational mission of the college. This includes teaching medical and graduate students, postdocs, residents and fellows, staff, faculty, patients, and in some instances K–12 and college-level students and other members of the public. The first year of the award, 10 departments, 7 clinical science and 3 basic science, were represented. Existence of the award has also expanded the valued methods of teaching to include such areas as mentoring and educator development as well as recognizing a widened variety of educational techniques (e.g., simulation and standardized patients). Many educational resources are considered as enduring materials (e.g., textbooks, case vignettes, audiovisual materials) and have clear usage requirements. Work in the educational mission also includes educational leadership activities and educational research.
The proportion of medical school graduates who are women has risen over the past three decades,11 but the literature indicates that those women who choose academic medicine are not promoted at the same rate as their male counterparts.11,12 In contrast, our data suggest that there is no difference in the promotion of men and women who receive an award. The award might be helping to ensure that men and women are judged equitably and, therefore, are promoted at equal rates.
Another possible reason for the success of this program is the diligence with which the award was created. Those who established the award spent over two years developing the process and the examples. Rather than require a number of specific achievements, this unique award references both the quantity and types of activities and supporting documentation of quality. The award serves as a respected symbol for excellence in education, recognizing both the recipient and the value of education in our institution.
From the beginning, members of the P&T committee supported the development of the award, but after a couple of members of the P&T committee joined the F&J review panel, support of the award by members of the P&T committee increased substantially. Those faculty who serve on both committees report to members of the P&T committee the rigor of the F&J Award process. It took approximately five years for the award to be fully supported by members of the P&T committee, and it took about this long for educators considering promotion to recognize the importance of the reception of an F&J Award in the P&T process.
Viewing our data, one might ask, “Why haven’t more people applied for the award?” For the last five years, we have had a steady increase each year in the number of new people receiving the award: 25 in 2009; 27 in 2010; and 43 in 2011. We anticipate more people applying for and receiving the award as word spreads that reception of the award can be helpful for promotion, especially in basic science departments.
One significant barrier for faculty applying for an award, especially those not interested in using the award for promotions (e.g., full professors), is the time and effort required to prepare the mini-portfolio. In response, award developers have created an alternative approach for full professors that maintains the quality of the award but decreases the amount of time required to prepare the application. With this alternative, several senior faculty have received recognition, joined the Academy of Distinguished Educators, and are encouraging junior faculty in their departments to apply using the regular process.8
One limitation to our internal review was the lack of a comparison group. Because of the confidentiality and unavailability of specific information surrounding promotion and tenure at our institution, there are no records from which we could determine the actual number of educators without an F&J Award who were promoted during this time period. Our interviews of P&T committee members do clearly substantiate the increase in value now placed on educational work as documented with an F&J Award.
Regarding generalizability, the variation by institution of P&T rules and outcomes should be taken into account by those interested in developing a similar program at another institution. Also, it is important to involve all stakeholders in the planning process and to find the funding for the salary and administrative costs associated with implementing an award. With these recommendations in mind and using our templates and materials, the awards process can be adapted to other institutions and has already been adapted and used as a method for admission to two other academies of educators at the American Association of Colleges of Osteopathic Medicine and the University of Texas Medical Branch, Galveston, and the Council of Emergency Medicine Residency Directors (CORD) Academy for Scholarship in Education in Emergency Medicine “Distinguished Educator” Award.13–15
In conclusion, we feel confident that this self-nominating, standards-based, peer-reviewed program to assess the scholarship of teaching of medical faculty could serve as a model for others interested in leveling the academic playing field for researchers, clinicians, and educators.
Acknowledgments: The authors would like to thank those far-sighted faculty members who gave freely of their time to develop this important and influential program. They would also like to thank those who participated in the online survey and the interviews. They also thank Dr. Stephen Greenberg, dean of medical education, Baylor College of Medicine, for advice concerning this article.
Funding/Support: The Fulbright and Jaworski L. L. P. Faculty Excellence Award Program is supported by an endowment from the international law firm of Fulbright & Jaworski L. L. P.–Houston.
Other disclosures: None.
Ethical approval: Ethical approval was obtained from Baylor College of Medicine’s institutional review board.
Previous presentations: Previous presentations describing this program but not the data collected have been presented at the Association of American Medical Colleges meeting in 2002, the GFA meeting in San Francisco in 2009, the Oakland University William Beaumont School of Medicine in 2010, and in 2011 at the following places: Monterrey Tech, Monterrey, Mexico; Texas A&M Health Science Center, College Station, Texas; Society for Education in Anesthesia Annual Meeting, San Antonio, Texas; and the American University of Antigua, Coolidge, Antigua.
* A search of the literature found no other teaching award defined as self-nominating, standards-based, and peer-reviewed. Medical schools usually have a variety of teaching awards given each year, but they are generally norm referenced. In such cases, applicants are usually judged against each other and not judged against a standard, and usually the two or three individuals with the highest scores receive awards. Cited Here...