Innovation Reports: Successes and Limitations for Promoting Innovation in Medical Education

Abstract

In 2013, Academic Medicine introduced a new article type, Innovation Reports, with the intent to promote innovation by providing a forum for publishing promising new ideas at an early stage of development. In this article, the authors examine Innovation Reports as a means for promoting innovation within the medical education community.

The authors undertook a 2-part analysis. In July 2018, they examined submission data, Altmetric scores, and citations for the first 5 years of the feature. To explore authors’ perceptions of the impact of publishing an Innovation Report, in October 2018, they conducted a web-based survey of corresponding authors.

Between October 2013 and May 2018, 920 manuscripts were submitted under the Innovation Report category, of which 335 were sent for review and 151 were published (16% overall acceptance rate). The mean citation rate for the published Innovation Reports was 4.3 (range 0–47). The mean Altmetric Attention Score was 14.3 (range 0–441). An Altmetric score of 14 places an article in the top 20% of articles published in Academic Medicine.

The 151 published Innovation Reports had 148 unique corresponding authors, of whom 85 (57%) responded to the survey. Almost all respondents (n = 82; 96%) reported that publishing an Innovation Report promoted their individual career growth. For many corresponding authors, the publication of early ideas in an Innovation Report appeared to be an end point rather than a springboard for further development and innovation, as only 14 (16%) reported publication of a subsequent related study.

Reflecting on the successes and limitations of Innovation Reports over the first 5 years, the authors suggest that soliciting input from more stakeholders and being explicit about the goals of this article type would help inform how Innovation Reports should evolve in the future so they can better launch creative thought and spur innovation.

Innovation spurs creativity in thinking, analyzing, communicating, and teaching.¹ It plays an important role in scholarship, challenging how we conceptualize and define problems. It can drive the dispersion of new ideas and create a movement of change within a community. Business leaders and executives have long extolled the merits of being innovative, and innovations from companies such as Google, Apple, and Amazon have changed the way we live our daily lives. Within the business world, researchers have developed innovation frameworks² and studied innovative leaders to identify ways to replicate their successes.

Within medical education, in contrast, there has been a relative lack of new ideas over the past few decades.³ Medical schools and residency programs have historically standardized the way they deliver education to their learners to comply with strict regulatory requirements, set forth by accreditation bodies such as the Liaison Committee on Medical Education and the Accreditation Council for Graduate Medical Education, which have limited creative thought and exploration of new practices. Minimal grant funding within health professions education (HPE) domains has further limited opportunities for innovative ideas to be researched, tested, and developed. As such, optimal educational strategies and practices “may differ dramatically from current practice.”⁴

In this Perspective, we examine Academic Medicine’s Innovation Reports as a means for promoting innovation within the medical education community. We also propose ways to rework Innovation Reports so they can better launch creative thought and spur innovation.

Innovation Reports: The First 5 Years

In 2013, Academic Medicine introduced Innovation Reports as a new category of article. Then-editor-in-chief David P. Sklar wrote of his hope that “providing this venue for the presentation of promising new ideas at an early stage of their development may help speed the adoption of those that fulfill their promise.”¹ Some of us (M.B.D., J.D.C., S.J.D.) were involved in the development and implementation of Innovation Reports and have reviewed hundreds of submissions since the feature was introduced.

The original intent was to create space for publishing creative solutions to problems and challenges within medical education. The inspiration for this was in part previous editor-in-chief Steven L. Kanter’s 2008 editorial, “Toward Better Descriptions of Innovations,” which provided 9 criteria for describing an innovation that Kanter hoped would stimulate authors to publish their innovative ideas, curricula, and programs.⁵ Kanter’s framework encouraged “a reflective, analytical, and scholarly treatment” of the innovation, including “details about the problem and its generalizability; an account of the full array of solutions considered and why each was accepted or rejected; [and] a critical appraisal of the potential of the innovation to spur additional innovation.”

Sklar noted “the need to provide some additional options for the presentation of innovations,”¹ and described the scope of Innovation Reports as follows:

The ideas offered in Innovation Reports may be new ways of teaching, thinking, organizing care, communicating, analyzing information, or managing medical schools and teaching hospitals. They may describe disruptive products, ways of disseminating innovations, or incremental quality improvements that integrate education, research, and clinical care. They may also describe new configurations of clinical care teams and new methodologies and analytic approaches to problems. Innovation Reports will present ideas that challenge how we think about and define problems and will help us incubate ideas that can be tried out in our communities.¹

To underscore the preliminary nature of the innovations described in Innovation Reports, Academic Medicine outlined strict limits for manuscripts: no more than 2,000 words, 10 references, and 3 exhibits. The text of Innovation Report manuscripts is required to be organized using the headings Problem, Approach, Outcomes, and Next Steps. From the inception of the feature, the peer-review process has been the same as for any type of submission to Academic Medicine, although authors and reviewers had to acquaint themselves with the new submission criteria developed for Innovation Reports.⁶

To gain some insight into the successes and limitations of Innovation Reports over the first 5 years of the feature, in July 2018, we examined Academic Medicine submission data from Editorial Manager (Aries Systems Corporation, North Andover, Massachusetts) and Altmetric scores displayed at academicmedicine.org. Using Scopus (Elsevier), we identified the number of times each published Innovation Report had been cited; the number of previous publications by the corresponding author before the Innovation Report; and the number of subsequent publications by the corresponding author on the same topic.

In addition, to explore authors’ perceptions of the impact of publishing an Innovation Report, we conducted a brief web-based survey of 148 corresponding authors in October 2018 via REDCap (Vanderbilt University, Nashville, Tennessee). There was no incentive for participation, and the University of Washington Institutional Review Board determined the research project to be exempt. The survey was developed de novo and pilot tested. The final survey contained 13 items addressing corresponding author demographics, impact on career, and future directions (available as Supplemental Digital Appendix 1 at https://links.lww.com/ACADMED/B10.) Using a constant comparative approach, 3 of us (J.B.B., S.J.D., C.E.M.) conducted a thematic analysis to identify salient themes in the free-text responses to 3 survey questions that asked whether (1) anyone had contacted the corresponding author about the Innovation Report, (2) the Innovation Report had led to any new collaborations, and (3) there were specific ways in which publishing the Innovation Report had helped the author’s career growth.

Characteristics of submissions and authors

Between October 2013 and May 2018, 920 manuscripts were submitted under the Innovation Report category. Of these, 335 were sent for review and 151 were published, leading to an overall acceptance rate of 16%. The 151 published Innovation Reports had 148 unique corresponding authors, who represented 7 countries (the United States, Canada, the United Kingdom, the Netherlands, Zimbabwe, Israel, and Australia). The majority (n = 118; 76%) were from the United States.

The published Innovation Reports covered a wide range of topics, such as teaching and assessment strategies, understanding the social determinants of health, technological innovations, and medical school admissions processes. The most common key words selected by corresponding authors to categorize these 151 submissions were “curriculum development, reform, or evaluation” (n = 55; 36%) and “program evaluation” (n = 34; 23%). Half (n = 75; 50%) of the published Innovation Reports did not report any funding sources.

The interquartile range of the number of previous publications by the corresponding author in any journal before publication of the Innovation Report was 3–34. Subsequent related publications were identified for 38 (26%) of the 148 corresponding authors using Scopus. Of the corresponding authors, 85 (57%) responded to our survey. Respondents’ demographic data are presented in Table 1.

Table 1: Demographic Characteristics of the 85 Corresponding Authors Responding to 2018 Survey on Innovation Reports^a

Successes

The mean citation rate for an individual Innovation Report was 4.3 (range 0–47). The mean Altmetric Attention Score was 14.3 (range 0–441). A mean Altmetric score of 14 places an article in the top 20% of articles published in Academic Medicine.

Most of the 85 survey respondents (n = 75; 88%) indicated that the Innovation Report was “quite important” or “extremely important” as an outlet for scholarship. Most (n = 75; 88%) felt the Innovation Report was a unique outlet for scholarship (Table 2). Almost all respondents (n = 82; 96%) reported that publishing an Innovation Report promoted their individual career growth, with 15 (18%) reporting that it helped their career growth a “tremendous amount.”

Table 2: Corresponding Authors’ Perceptions of the Impact of Publication of an Innovation Report, 2018 Survey^a

In free-text responses on the survey, corresponding authors stated Innovation Report publications led to both personal career advancement and advancement of the academic community (Table 3). They indicated that they received increased recognition in the topic area of their published Innovation Report, opportunities for collaborations, and subsequent grant funding. Fifty-four (64%) of the respondents reported that they had been contacted via email with inquiries regarding their Innovation Report, and 37 (44%) reported that the publication led to increased opportunities for new collaborations. One respondent wrote, “I have collaborated with numerous other people over the years, and none of the following papers would have occurred if we had not done the Innovation Report first” (P48). Fourteen respondents (17%) indicated that their Innovation Report led to a subsequent published manuscript. Twelve (14%) reported that publication of their Innovation Report contributed to obtaining grant funding: “The Innovation Report legitimized my work such that a grant agency asked me (and others) to consider applying for funding. That led to 2 received grants” (P8).

Table 3: The Impact of Publishing an Innovation Report: Themes From Corresponding Authors’ Free-Text Comments, 2018 Survey on Innovation Reports^a

The respondents who published about a curriculum or a novel program reported that publishing an Innovation Report furthered their curriculum or program’s credibility and growth within their local institution: “[Publication led to] increased visibility of our program and increased visibility of my work” (P63). It also enabled dissemination of their curricula and programs nationally: “Numerous institutions asked for more information; [I] visited and gave seminars at 3 institutions on the topic” (P70). Authors described providing assistance with initiating similar programs at other institutions and invited professorships to share programmatic experience and details of curriculum implementation: “I received at least 2 requests to discuss our report at Medicine Grand Rounds around the country” (P62).

Limitations

While the majority (n = 50; 60%) of the 85 survey respondents reported being “quite likely” or “very likely” to submit another Innovation Report, 17 (20%) reported being “not at all likely” or “somewhat likely” (Table 2). One of the most common reasons authors provided for being hesitant to submit another Innovation Report was the lengthy review process. One described it as “extremely laborious” (P46), while another suggested “making the time to publication shorter” (P20). Another common reason was the Innovation Report structure. Authors suggested allowing more flexibility in the submission requirements. One wrote, “Only having 10 references was not enough to adequately reference my theory-heavy innovation” (P23). Some felt that the 4 mandatory section headings (Problem, Approach, Outcomes, Next Steps) were too restrictive, preferring instead that “section headers be up to the authors” (P9).

Only 14 (16%) of the survey respondents reported that they had published a subsequent follow-up or larger study that grew out of the original Innovation Report publication.

Opportunities to Promote Innovation

Academic Medicine is not alone in attempting to promote early-stage work within HPE scholarship. Journals such as Academic Emergency Medicine Education and Training, Journal of Graduate Medical Education, Medical Education, and The Clinical Teacher have article types intended to serve as outlets for dissemination of new ideas. Yet, although opportunities exist to publish new ideas, how much progress have we made toward promoting innovation, as described by Kanter⁵?

Although our examination of Innovation Reports revealed that publication appears to advance individual career development and spur larger academic discussions, we were able to identify a subsequent related publication in Scopus for only 1 in 4 corresponding authors of published Innovation Reports. It is possible that an Innovation Report lead to another group of HPE researchers publishing a research project that we failed to capture given our data collection methods. However, it is also possible that the innovation lacked sustainability, one of Kanter’s 9 criteria,⁵ and therefore did not result in deeper exploration. Another reason may be that many HPE researchers have limited time, funding, and infrastructure to conduct and publish work.^7,8 We believe the reality for many Innovation Report authors is that the publication of these preliminary ideas becomes the stopping point rather than the springboard for larger studies. If published reports of single-institution innovations do not yield more robust, longitudinal reports as these innovations mature, we must ask whether a steady stream of stand-alone publications describing new ideas is enough to sustain a culture of innovation.

Single-institution initiatives, like those described in many Innovation Reports, are only the first of many steps in building programs and theories to advance a field. Publishing these innovations should function as a kick-starter. It should serve as the first phase and help establish research momentum to promote rigorous, collaborative, multi-institution projects or, alternatively, attempts to reproduce similar results in different settings.

Academic Medicine and journals publishing article types similar to Innovation Reports have the opportunity to promote, create discussion around, and facilitate a larger culture of innovative thought that supports the advancement of new ideas. However, these articles should not serve as an outlet to publish early and often. To truly create and advance novel ideas, authors must look as critically at their innovations as they do at perceived gaps in the literature. While many Innovation Reports address what was done and what worked, they often fail to meet 4 of Kanter’s 9 criteria for describing innovations: a deeper exploration into why a specific innovation was chosen from other potential solutions, what failures and roadblocks were encountered along the way, a critical assessment of the innovation’s influence on a discipline, and generation of new ideas and new opportunities for exploration.⁵

To increase the relevance of Innovation Reports within HPE scholarship, this article type should be analyzed more critically. First, Academic Medicine should measure the impact of Innovation Reports by soliciting feedback from stakeholders, including authors, reviewers, and readers, regarding the influence and impact of this article type. We acknowledge that our soliciting feedback only from corresponding authors and evaluating submission data may not have provided a comprehensive evaluation of the reach of this article type. The impact of Innovation Reports should also be measured through citations and social media shares, and these data should be compared with data for the journal’s other article types.

Second, Academic Medicine should provide a clearer definition of what qualifies to be published as an Innovation Report. Can program descriptions, new curricula, preliminary research, or newly developed tools qualify? When is something no longer considered an innovation? What kind of outcomes are expected or acceptable for an Innovation Report? We suggest authors should aim to collect and report outcomes that go beyond satisfaction, attitudes, and awareness; we recommend striving for levels 2 and 3 of Kirkpatrick’s 4 levels of evaluation (i.e., learning and behavior).⁹ Greater clarity around the article type and encouraging higher levels of outcome data would not only benefit authors and reviewers but also help improve the quality of innovations.

Soliciting input from more stakeholders and being explicit about the goals of this article type would also help inform how this feature should evolve in the future and whether the benefits outweigh the limits.

Conclusions

Although publication of Innovation Reports is a step toward promoting innovation, there are potential improvements that would help the HPE community achieve Kanter’s⁵ and Sklar’s¹ goals of disseminating innovations. The methods by which Academic Medicine and other journals can assist in this work requires further consideration and input from educators and investigators in the field.

Acknowledgments:

The authors wish to thank Kunhui Zhang from the Department of Statistics, University of Washington.