Secondary Logo

Journal Logo

Innovation Reports

Rethinking the Educator Portfolio: An Innovative Criteria-Based Model

Shinkai, Kanade MD, PhD; Chen, Chen (Amy) MD; Schwartz, Brian S. MD; Loeser, Helen MD, MSc; Ashe, Cynthia; Irby, David M. PhD

Author Information
doi: 10.1097/ACM.0000000000002005

Abstract

Problem

Medical educators often struggle to achieve academic advancement at the same rate as their research-focused colleagues.1 Achieving parity in academic promotions between educators and researchers has largely been hindered by a narrow definition of scholarship, a lack of clear criteria for measuring educator excellence, and a paucity of strategies to make educational contributions available for rigorous, efficient peer review. As a consequence, educators are promoted more slowly and are less satisfied with their promotions than their research-focused counterparts.1

Over the past several decades, education scholars have made significant strides in addressing these impediments. In a seminal work, Boyer2 argued that institutions should move past the research versus teaching debate and broaden the term scholarship to include education work. Later, Glassick and colleagues3 created six standards of educator excellence (adequate preparation, clear goals, appropriate methods, significant results, effective presentation, and reflective critique), and the Association of American Medical Colleges (AAMC) identified five core educator roles (teaching, curriculum development, advising and mentoring, education leadership, and learner assessment).4 Since then, others have built on this work and proposed guidelines for evaluating educator excellence.5

Increasingly, those in education have recognized that educational scholarship, like publications, must be made available for peer review.6 To address this issue, the international teaching community developed the concept of the educator portfolio (EP).7 The EP was designed both as a place for individuals to document their teaching activities and accomplishments in education and as a tool to contribute to their professional development.7 In medicine, EPs are often included in applications to academies of medical educators (AMEs), which in turn play a crucial role in advocating for improving teaching quality and for recognizing distinguished teachers.8

Although EPs are intended to display educators’ best work, their introduction to medical education has been flawed. EPs are highly variable in content, often excessively long, and difficult to assess consistently because few evaluation guidelines exist.9 Furthermore, EPs are frequently prepared as optional attachments to educators’ curricula vitae (CVs) and may not focus on individuals’ recent work, clearly define their teaching and educator roles, or describe the significance of their contributions to their fields. Therefore, EPs have not adequately contributed to the objective evaluation of educators’ academic contributions.

After recognizing many of these deficiencies, we redesigned our institutional EP to display faculty members’ current academic activities related to education, facilitate rigorous and efficient peer review, enhance academic promotions, and provide an improved tool for professional development. In this Innovation Report, we describe the development of our new EP—a streamlined and criteria-based template for succinct, relevant, and standardized EPs that can be integrated into existing CVs and used to objectively evaluate the quality of an educator’s work. We report the implementation of this new model first into our Academy of Medical Educators (AME) application process and subsequently into the campus-wide academic advancement system.

Approach

Redesigning the EP template

In 2013, the AME at the University of California, San Francisco (UCSF) appointed a working group to redesign the existing EP (EP 1.0) template for use in the AME application process and in academic promotions. The working group modeled the new EP (EP 2.0) template on two narrative sections in the existing CV platform used at UCSF, entitled Research Program and Significant Publications. The former encourages faculty members to describe their current area(s) of research and its significance, and the latter requests information about faculty members’ most important, recent publications with details of the impact of those publications. This approach resulted in a redesigned EP template that consists of a one-page executive summary that lists up to five of the faculty member’s key, recent contributions to education (including time allocation to educator roles, changes in those roles, and evidence of impact) and up to three 2-page detailed role descriptions based on the five AAMC core educator roles, which provide more granular information on the educational activities outlined in the executive summary.

The EP 2.0 template also provides criteria to evaluate educator excellence, which were adapted from Glassick’s criteria for standards of educator excellence and the AAMC toolbox for evaluating educators (see Table 1).3,10 These criteria were designed to address the long-standing lack of standardized metrics to assess EP quality. Of note, there is no rubric or scoring system connected with these criteria, as they are intended to be used as qualitative tools to assess a faculty member’s educational contributions based on broad indicators of excellence. Overall, the critical elements of the new EP 2.0 template include a structured framework and standardized layout which define different roles, provide information on the significance of a faculty member’s recent contributions, and convey clear, qualitative standards for evaluation.

Table 1
Table 1:
Criteria for Evaluating the Scholarly Contributions of Educators and Examples of Broad Indicators of Educator Excellence Used in the New Educator Portfolio (EP 2.0) Template at the University of California, San Francisco (UCSF), Adapted From Glassick’s Criteria for Standards of Educator Excellence3 and the Association of American Medical Colleges (AAMC) Toolbox for Evaluating Educators10

Piloting the EP 2.0 template with the AME

To provide proof of concept, the AME working group applied the new criteria to EPs from the previous round of AME applications and found the included information to be inadequate for evaluating educator excellence. The working group members then completed EP 2.0 templates as examples to share with the AME members and conducted two workshops, first to gather feedback on the new model and then to coach AME members in preparing their own EP 2.0s. The examples and workshops were enthusiastically received, and suggestions for improvement were incorporated into the new EP 2.0 template.

In 2014, AME candidates were required to submit applications that included the new EP 2.0 template (see Appendixes 1 and 2 for blank copies of this template and Supplemental Digital Appendixes 1 and 2 at http://links.lww.com/ACADMED/A503 for completed examples). The AME supported candidates throughout the application process by providing an optional portfolio coach and an administrative review of a draft of each candidate’s application to ensure alignment with the requisite standards. Candidates and coaches also received a frequently asked questions document (see Supplemental Digital Appendix 3 at http://links.lww.com/ACADMED/A503).

Integrating the EP 2.0 template into the UCSF academic advancement system

Throughout the pilot, the AME engaged the Office of Academic Affairs in discussions about the limitations of the current campus-wide CV platform and the need for improved tools to demonstrate educator excellence. In 2015, UCSF convened a faculty academic advancement task force to review the existing CV platform and to recommend changes. An AME member (K.S.) served on that task force and advocated for inclusion of the EP 2.0 template in the online CV platform, based on the relevance of educator excellence to the academic promotions process (particularly in those advancement series in which educator roles are a major academic activity) and the need for improved, rigorous evaluation of educator roles. The task force agreed that faculty members whose major academic contribution is in education faced bias in the existing CV platform due to the emphasis on discovery-oriented research publications; absence of a succinct, consistent, and robust format to document teaching excellence; and lack of specific criteria to rigorously evaluate educator contributions. Furthermore, in the existing promotions process, the EP was considered supplemental information, appended as an electronic attachment, and inconsistently transmitted with the faculty member’s CV to those writing letters of support and to the promotions committee.

The task force’s concerns about integrating the EP 2.0 template into the CV platform included the potential for increasing the length of the CV and the time needed for review, as well as the possibility that those with minimal teaching roles would inappropriately use it. Addressing these concerns, the EP 2.0 template is succinct (see Appendixes 1 and 2) and offers specific criteria for evaluating excellence. Furthermore, the flexible electronic format permits faculty members to select either the traditional teaching narrative entry (a free-text description of their educator activities and/or teaching philosophy) or an alternative view with the EP 2.0 template, and embedded instructions encourage only faculty members with significant educator roles to use the EP 2.0 template. Importantly, the EP 2.0 template is also integrated directly into the online CV platform, making it consistently available to all letters of support writers and promotions committee reviewers.

Throughout this entire process, administrative staff provided answers to any queries the new EP 2.0 template users had via e-mail and telephone.

Outcomes

A brief narrative survey was administered to AME candidates after they completed the academy selection cycle in 2014–2015. Overall, candidates reported high satisfaction with the EP 2.0 template. They found it easy to complete and lauded its easy-to-follow directions and transparent evaluation system, citing that “the examples were excellent” and that the “direction[s] were very clear.” Candidates also appreciated the examples of completed portfolios and benefited from working with the portfolio coaches (especially in determining which roles to emphasize and what documentation to include), stating that the coaches “helped hone my application,” “gave feedback to my approach,” and “provided guidance on fields to apply to.” Some challenges included uncertainty about mentoring activities and identifying and accessing data for the Results and Impact section (particularly providing comparative teaching evaluation scores). In debriefing discussions with the AME selection committee, committee members unanimously affirmed that the EP 2.0 template provided superior information to the original EP 1.0 for conducting peer review and making membership decisions, and they enthusiastically praised the new template’s clarity, consistency, and conciseness.

The EP 2.0 template successfully addressed the major shortcomings of the EP 1.0 template, including variability, excessive length, lack of a structured framework and specific criteria for evaluation, and exclusion from the academic promotions process. Importantly, the new template may serve as a career development guide for new faculty members who aspire to medical education careers, and it offers a framework for how to best demonstrate educational scholarship achievements through explicit definitions of excellence.

We also learned several important lessons through the process of designing the EP 2.0 template. First, AME candidates posed common questions and portfolio coaches encountered typical challenges, which resulted in the creation of a frequently asked questions document (see Supplemental Digital Appendix 3). Second, the series of workshops helped introduce the EP 2.0 template to AME members and the selection committee, and trained portfolio coaches enhanced the implementation process. If other institutions plan to introduce a template like ours, faculty development for candidates, coaches, and promotions committees will be important to ensure its successful implementation. Third, the importance of having passionate faculty members to advocate for key changes to the advancement process cannot be overstated. Promoting the understanding that an improved, rigorous assessment of educator excellence is highly relevant to a promotions process in which educator roles are a major academic activity was central to the eventual adoption of the EP 2.0 template across the UCSF system. Fourth, faculty members appreciated the directions on the new template to select and highlight their best, current contributions in education. They liked that they controlled the process and could reflect on their accomplishments, and as a result, they felt empowered and engaged.

Next Steps

We will continue to increase awareness of the EP 2.0 template at UCSF and at other institutions as a model that offers educators the opportunity to display their important, recent scholarly contributions to education. Work continues at UCSF to inform faculty members on how to complete the EP 2.0 template and to direct promotions committees on how to apply the evaluation criteria to faculty members’ EPs. We also seek to partner with other academic institutions and educational leadership organizations that are using, improving, or developing documentation for educator excellence to achieve national consensus on effective EP templates and to broadly disseminate these educator advocacy tools. Ultimately, we expect that the rigor introduced by the EP 2.0 template will reduce the disparities in academic promotions between educators and researchers and change the academic culture in positive ways.

In the future, we aim to carry out more quantitative investigations on the impact of the EP 2.0 template and to further characterize the responses of all stakeholders to its implementation. Specifically, we plan to study the impact of the new template on promotions rates, its utility to promotions committees that may not include educator faculty and/or to departments that may not include many educators, and the feedback from AME candidates in terms of satisfaction, ease of use, and time required for completion. Overall, we anticipate that our streamlined and criteria-based EP 2.0 template will enable educators to highlight their current creative academic work in education, provide guidance for professional development, and decrease disparities in academic promotions between medical educators and discovery-oriented researchers.

Acknowledgments: The authors are grateful to Denise Connor, MD, Karen Hauer, MD, Maxine Papadakis, MD, Kathleen Land, Sharad Jain, MD, Rebecca Jackson, MD, Brian Alldredge, PharmD, Elena Fuentes-Afflick, MD, MPH, Renee Binder, MD, Cynthia Leathers, MBA, and Ned Hamilton for their collaboration in the development and advocacy of the new educator portfolio template at the University of California, San Francisco.

References

1. Thomas PA, Diener-West M, Canto MI, Martin DR, Post WS, Streiff MB. Results of an academic promotion and career path survey of faculty at the Johns Hopkins University School of Medicine. Acad Med. 2004;79:258264.
2. Boyer EL. Scholarship Reconsidered: Priorities of the Professoriate. 1990.Princeton, NJ: Princeton University Press.
3. Glassick CE, Huber MT, Maeroff GI. Scholarship Assessed: Evaluation of the Professoriate. 1997.San Francisco, CA: Jossey-Bass.
4. Simpson D, Fincher RM, Hafler JP, et al. Advancing educators and education by defining the components and evidence associated with educational scholarship. Med Educ. 2007;41:10021009.
5. Gusic M, Amiel J, Baldwin C, et al. Using the AAMC toolbox for evaluating educators: You be the judge! MedEdPORTAL. 2013;9:9313.
6. Schulman LS. From Minsk to Pinsk: Why a scholarship of teaching and learning? J Scholarsh Teach Learn. 2000;1:4853.
7. Niebuhr V, Johnson R, Mendias E, Rath L, Sandor K, Szauter K. Educator portfolios. MedEdPORTAL. 2013;9:9355.
8. Irby DM, Cooke M, Lowenstein D, Richards B. The academy movement: A structural approach to reinvigorating the educational mission. Acad Med. 2004;79:729736.
9. Snadden D, Thomas M. The use of portfolio learning in medical education. Med Teach. 1998;20:192199.
10. Gusic ME, Baldwin CD, Chandran L, et al. Evaluating educators using a novel toolbox: Applying rigorous criteria flexibly across institutions. Acad Med. 2014;89:10061011.
Appendix 1
Appendix 1:
Executive Summary, New Educator Portfolio (EP 2.0) Template, University of California, San Francisco
Appendix 2
Appendix 2:
Detailed Teaching Role Description, New Educator Portfolio (EP 2.0) Template, University of California, San Francisco (UCSF)

Supplemental Digital Content

Copyright © 2017 by the Association of American Medical Colleges