In the United States and throughout the world, the annual published reports ranking the quality of medical schools and universities have been controversial in part because of the heavy reliance on how much a school emphasizes research and how successful a school is in competing for grants. Although the publishers of these rankings make an effort to factor in the quality of the educational program, their surveys of medical school faculty are both subjective and vulnerable to conflicts of interest. As the surveyed faculty provide their personal evaluations of the quality of another school’s education program, they may be influenced by wanting their own school to rank higher. Yet, as educators, we have no one to blame but ourselves, because the publishers are doing the best that they can with the data available to them. Unlike publicly available lists of research grants, we do not provide any objective benchmarks of excellence in medical education.1,2 Strong research programs, the number of Nobel Prize winners, and the dollar amounts of external grants are mistakenly taken as sufficient evidence of the quality of the educational program. Indeed, Dr. Barbara Ischinger,3 director for education at the Organisation for Economic Co-operation and Development, stated that the current university rankings do more harm than good because they largely ignore a key measure of quality—namely, what goes on in the seminar rooms and lecture theaters.
That is, until now.
In August 2012, the Association for Medical Education in Europe (AMEE) introduced an initiative entitled A Schools Programme for International Recognition of Excellence in Education (ASPIRE) awards. Through a board of international leaders in medical education and panels of experts, this initiative, sometimes called the ASPIRE-to-Excellence program, began to establish internationally peer-based standards of excellence in medical education. Leaders in dentistry and veterinary medicine requested involvement, and those disciplines were incorporated into the ASPIRE panels and board in 2014. At present, the ASPIRE board has 24 members from 16 countries.
Initially, the goal was to establish these standards of excellence for an entire medical education program. However, that goal was soon seen to be overly ambitious because of the number of variables that make up an excellent education program, including criteria covering both high- and low-resourced environments and the need to factor in cultural differences. It was also accepted that a school might demonstrate excellence in some aspects of its education program but not in others. This led to a decision to define the components of an excellent medical education program. Once that had been accomplished, experts from around the world were impaneled to develop criteria for excellence in each component.
One might ask, “To determine excellence, why not simply use a country’s accreditation standards and the outcomes of the reports that derive from the accreditation visits?” That approach was attempted in the United States over an eight-year period when the two co-secretaries (one being the first author of this Commentary) of the Liaison Committee on Medical Education tried three different approaches to using the extensive information from an accreditation visit to devise a reliable and valid approach to ascertaining excellence—but failed to do so. The underlying cause of this failure is that accreditation standards are designed to ensure minimum levels of quality and do not address levels of excellence. In the process, they also found that listing strengths in these accreditation reports was so inconsistent across schools that this practice was eliminated in 2015.
Five Years of the ASPIRE Program
The ASPIRE program initially formulated criteria for social accountability, student assessment, and student engagement, and these were followed in years 3 and 4 by faculty development and simulation. Curriculum development and design are undergoing pilot review during 2018 and should be available for schools to examine late in that year. While as of this writing, 88 schools (see Table 1) have taken the time to apply for recognition of excellence in one or more of these five criteria, perhaps the most significant contribution of the ASPIRE initiative is that there are now criteria that recognize best practices in each of the five areas mentioned above; these criteria can serve as benchmarks for a school’s own quality management.
One must not underestimate the enormity of the challenge in bringing 15 to 20 content experts from around the world together on a volunteer basis and asking them to develop criteria for excellence. The ASPIRE website4 presents the five criteria established so far. For a fee of £2,500 (U.S. $3,230 at the time of this writing), a school can submit evidence of excellence for these criteria and be judged by a panel of international experts in the corresponding content area(s). If successful, an ASPIRE prize is awarded during the annual AMEE conference. If unsuccessful, targeted feedback to the school is provided to identify the areas that can be improved, and a reduced fee for reapplication is available. In an informative essay, Cianciolo et al5 offer an interesting perspective in their description of the challenges and benefits of one school’s experience with their ASPIRE application. As evidence for the rigor of the criteria, as noted in Table 1, across all of the categories, only 38 (43%) of the 88 submissions were awarded the designation of excellence.
We have demonstrated that it is possible to develop criteria for best practices in several areas of medical education, that excellence in medical education can be assessed using these criteria, and that schools find the process of applying the criteria useful when they review their education programs. Patterns have emerged as a result of this review process. For example, with the criterion of student engagement, it has become clear that schools not only must show evidence that their students have voting privileges on the curriculum management committee but must also show that change has come about based on student input. As well, the students’ evaluations of faculty teaching must be systematically used for faculty promotion and not just for annual teaching awards.
A surprise for all is the relatively low submission and success rate for the student assessment award; as of 2017, there have been 15 student engagement awards but only 4 for student assessment, as shown in Table 1. It appears that a common shortcoming is not having a variety of reliable and valid assessment tools in place that are aligned across the continuum of the educational experience.
Schools failing to achieve an ASPIRE award with their submission for social accountability appear to fall short in one main way: their ability to provide evidence that their graduates are in fact practicing in the targeted communities consistent with the school’s social accountability mission statement.
It is too early to identify trends in faculty development and simulation.
ASPIRE Academy: An unanticipated benefit
Although not initially anticipated, it soon became apparent that the representatives from the successful schools wanted more than just a designation of excellence. They organized a group of former award winners to help other schools learn from them. The ASPIRE Academy was formally recognized by the ASPIRE board in 2015. This group of educators from around the world now provides a resource to other schools, both in individual coaching and faculty development sessions. To date, 3 webinars and 19 workshops or symposia have been provided by the ASPIRE Academy. These faculty development offerings are careful not to explain how to apply for an award but, instead, to provide guidance and examples of the different ways a school can be excellent in the five ASPIRE areas.
In some ways, two of the limitations of the ASPIRE effort are intertwined. The absence of a visit to a school by peers to validate the school’s data is a concern. However, the cost of a visit to each applicant school would be prohibitive, as the existing fee, although already supplemented by AMEE, is often reported as a barrier to application. The cost to the school is not only the fee but also the time and effort it takes to complete an application. Another pattern that has recently emerged and is creating intense discussion among the ASPIRE board members is that schools from Australia, New Zealand, Europe, and North America (with 61 submissions) enjoyed a 51% success rate, whereas the 28 submissions from the rest of the world had only 25% success. How to judge the evidence provided by schools from regions with fewer resources is a central theme in discussions of the ASPIRE board and the panels.
As ASPIRE passes through its fifth year, discussions at the board level have returned to the original question of how to identify schools with overall excellence in medical education. With an ASPIRE award for curriculum development and design coming on board in 2018, could a school with awards in all, or a certain number of, these areas be recognized as having an excellent medical education program overall? What about other areas, such as interprofessional education? How would one constitute panels from around the world and from multiple professions to write criteria and judge excellence in these other areas? What about schools that have unique areas of excellence, such as the school that requires all of its teachers to have training in education and then goes the extra mile in providing support and protected time for the teachers to pursue the additional qualifications involved? What about schools that support their students to learn about education and become teachers themselves?
In a relatively short period of time, the ASPIRE criteria and awards have emerged as a systematic way to provide objective and peer-based benchmarks of excellence in medical education. Bringing international experts together to create these criteria is a significant contribution in and of itself. As the number of representatives from the schools that are awarded ASPIRE recognition continues to increase and those individuals find new ways to contribute, the hopes are high for this program. Challenges remain in how to better define excellence in low-resource settings, what new areas to take on, and how to keep infrastructure costs down. However, as an example of continuing global interaction for quality improvement, optimism prevails.