Secondary Logo

Share this article on:

Making the First Cut: An Analysis of Academic Medicine Editors’ Reasons for Not Sending Manuscripts Out for External Peer Review

Meyer, Holly, S., MS, PhD; Durning, Steven, J., MD, PhD; Sklar, David, P., MD; Maggio, Lauren, A., MS-LIS, PhD

doi: 10.1097/ACM.0000000000001860
Research Reports

Purpose Manuscripts submitted to Academic Medicine (AM) undergo an internal editor review to determine whether they will be sent for external peer review. Increasingly, manuscripts are rejected at this early stage. This study seeks to inform scholars about common reasons for internal editor review rejections, increase transparency of the process, and provide suggestions for improving submissions.

Method A mixed-methods approach was used to retrospectively analyze editors’ free-text comments. Descriptive content analysis was performed of editors’ comments for 369 manuscripts submitted between December 2014 and December 2015, and rejected prior to external peer review from AM. Comments were analyzed, categorized, and counted for explicit reasons for rejection.

Results Nine categories of rejection reasons were identified: ineffective study question and/or design (338; 92%); suboptimal data collection process (180; 49%); weak discussion and/or conclusions (139; 37%); unimportant or irrelevant topic to the journal’s mission (137; 37%); weak data analysis and/or presentation of results (120; 33%); text difficult to follow, to understand (89; 24%); inadequate or incomplete introduction (67; 18%); other publishing considerations (42; 11%); and issues with scientific conduct (20; 5%). Manuscripts had, on average, three or more reasons for rejection.

Conclusions Findings suggest that clear identification of a research question that is addressed by a well-designed study methodology on a topic aligned with the mission of the journal would address many of the problems that lead to rejection through the internal review process. The findings also align with research on external peer review.

H.S. Meyer is assistant professor, Department of Medicine, Graduate Programs in Health Professions Education, Uniformed Services University of the Health Sciences, Bethesda, Maryland.

S.J. Durning is professor, Department of Medicine, and director, Graduate Programs in Health Professions Education, Uniformed Services University of the Health Sciences, Bethesda, Maryland, and deputy editor for research, Academic Medicine, Washington, DC.

D.P. Sklar is distinguished professor emeritus of emergency medicine, University of New Mexico School of Medicine, Albuquerque, New Mexico, and editor-in-chief, Academic Medicine, Washington, DC.

L.A. Maggio is associate professor, Department of Medicine, and associate director for technology and distributed learning, Graduate Programs in Health Professions Education, Uniformed Services University of the Health Sciences, Bethesda, Maryland.

Funding/Support: None reported.

Other disclosures: The views expressed in this article are those of the authors and do not necessarily reflect the official policy or position of the Uniformed Services University of the Health Sciences, the Department of Defense, or the U.S. Government.

Ethical approval: Reviewed by the Association of American Medical Colleges Institutional Review Board and deemed exempt from further review.

Correspondence should be addressed to Holly S. Meyer, 4301 Jones Bridge Rd., Bethesda, MD 20814; telephone: (202) 427-2288; e-mail: holly.meyer.ctr@usuhs.edu.

Written work prepared by employees of the Federal Government as part of their official duties is, under the U.S. Copyright Act, a “work of the United States Government” for which copyright protection under Title 17 of the United States Code is not available. As such, copyright does not extend to the contributions of employees of the Federal Government.

Submissions to peer-reviewed journals are on the rise for a variety of reasons, including the desire to further science and the pressure on faculty members to “publish or perish.” For each published article, many more manuscripts are submitted. Each manuscript, whether eventually published or not, requires the time and energy of editors, peer reviewers, and authors. The increased volume of submissions and publications presents the potential for strain on the publication system and for the stakeholders engaging with it. Researchers estimate that in 2013 peer reviewers dedicated 30.5 million hours evaluating manuscripts for publication.1 Reviewers spend on average six hours reviewing a single manuscript and review for, on average, five journals.2 Therefore, journals face increasing challenges for recruiting and retaining high-quality peer reviewers.

Internal editor review (e.g., reviewing manuscripts and selecting a portion to reject without sending them for external peer review) has been suggested as a way to mitigate reviewer burnout by subjecting external peer reviewers to fewer submissions and ensuring that those they receive are of higher quality.3 Academic Medicine (AM), a peer-reviewed health professions education (HPE) journal, uses internal editor review to cope with the influx of submissions, address authors’ needs for timely decisions, and protect the limited external peer review pool. While there is no standard model for internal editor review,4 since 2012 AM has chosen to operationalize internal review using a team of editors (typically two or more). This approach has been chosen because editors have an in-depth understanding of the journal, its mission (as described for authors on the journal’s Web site and in the instructions for authors), and what is desired (in terms of standards). The editors have discussed and reviewed general criteria for rejection of submitted articles, such as those proposed by Bordage,5 but did not have a formalized system created, such as a checklist to analyze their reasons for rejection without review. Editors at AM are also well read, have served as external peer reviewers for several years for the journal (as well as other related journals), and are typically aware of existing reviewer resources. For AM, internal review has resulted in more than half (65%) of the manuscripts being rejected without external peer review from 2013 to 2015. The rejection of these articles occurs quickly, two to three days after submission of the article, resulting in authors being able to make changes and resubmit promptly to a new publication.

Multiple articles have provided a rationale for submissions rejected by external peer review.5–7 However, now with many manuscript rejections occurring at the internal editor review stage, it is critical for authors to understand this process and avoid common pitfalls. To our knowledge this is the first report to investigate internal editor review by HPE journals and to make transparent a rigorous process such as is undertaken by AM.

The goal of this study was to inform scholars about common reasons for internal editor review rejections, increase transparency of AM’s internal editor review process, and provide suggestions for improving submissions. Given the high volume of articles rejected at this stage by AM, such an understanding could greatly benefit the HPE community. Additionally, like several other HPE journals, AM does not return specific information to authors about a manuscript regarding the reason(s) for rejection without review. Our hope is that authors will use this information to enhance the quality of manuscripts submitted and avoid rejection through internal review at AM and potentially other journals.

Back to Top | Article Outline

Method

Goal, design, and setting

We used a mixed-methods approach to retrospectively analyze editors’ free-text comments. Descriptive content analysis was used to identify reasons for the rejection of manuscripts submitted to AM. AM is the journal of the Association of American Medical Colleges (AAMC). This study was reviewed by the AAMC Institutional Review Board and deemed exempt from further review.

Each article submitted to AM is subjected to internal editor review as depicted in Figure 1. AM’s staff begins the internal editor review process by inputting manuscript-specific information into Editorial Manager (Aries Systems, 2016, North Andover, Massachusetts), the journal’s online submission system. Editorial staff distributes the manuscripts between the deputy editor, two associate editors, and the editor-in-chief based on expertise and capacity. Each editor, other than the editor-in-chief, provides a written justification and recommendation to reject following internal editor review or send the manuscript to external peer review.

Figure 1

Figure 1

The editor reads the manuscript, recommending whether it should be rejected or sent for external peer review and listing formal reasons for the editor-in-chief to review. The editors have devised criteria for these decisions based on bimonthly phone discussions and informed by the framework suggested by Bordage5 where challenging submissions are discussed, as well as recent efforts to update reviewer materials.8 These criteria are not formally codified (e.g., no checklist), in part to allow flexibility with individual submissions. Further, the editor-in-chief discusses manuscripts with individual editors outside of these phone calls where there are differences in opinion. In sum, AM uses a rigorous, yet flexible, process of “frame of reference” training between the editor-in-chief and the editorial team to determine whether the submission should proceed to external peer review or not. The editor-in-chief then reviews all recommendations for rejection from the editors and makes the final determination. Thus, when a manuscript is rejected without external reviews, if it went to the deputy editor or an associate editor, two editors agree on the decision.

During the study period, December 2014 to December 2015, AM received 1,273 submissions of a type requiring peer review. These were Research Reports (695), Innovation Reports (187), and Articles (391). Research Reports are original research, studying and addressing a concern, while Articles cover a topic of broad concern in medical education. Innovation Reports provide new solutions to problems at a smaller scale. Perspectives, Commentaries, and Review Articles were excluded from analysis as they are crafted using guidelines that differ from the above publication types.

During the study period, 840 (65%) manuscripts were rejected during the internal editor review process. Of the 840 manuscripts, 369 were routed to associate editors for internal editor review. The editor-in-chief reviewed the remaining manuscripts. This study only analyzed the associate editors’ free-text justifications for those rejections. Intentionally, we did not review the full text of the articles because our aim was to examine the rejection decisions and not to validate the accuracy of the decisions.

Back to Top | Article Outline

Analysis

Content analysis and constant comparison were used to characterize the internal editor reviewers’ comments, which were free-text comments from the associate editors. To structure the coding, L.M. and H.M. used Bordage’s “Top 20 Reasons Reviewers Recommend Rejection”5 as sensitizing concepts.9 For five rounds, L.M. and H.M. repeated the following process of analyzing 20 associate editors’ comments independently, collaboratively reaching consensus, and finally iteratively revising the codes. Once codes were set, L.M. and H.M. each coded 100 additional comments, reviewing any challenging comments together. H.M. coded the remaining 69 comments. L.M. performed a vertical check of all codes, reviewing each comment within a code to ensure it best fit in that coding. L.M. and H.M. discussed all discrepancies and reached consensus on 87% (344). A third author, S.J.D., served as a tiebreaker or provided additional context for the remaining coding decisions (25). S.J.D. also helped reach consensus on remaining editor comments, bringing consensus to 100%. Two of the authors (S.J.D. and D.S.) are editors for AM. Given their contextual knowledge behind editors’ written comments and to increase rigor, S.J.D. and D.S. also randomly reviewed a sample of editor comments (50).

Comments were assigned to as many codes as appropriately applicable. We established rules to keep codes from being overly applied. For example, when generalizability issues were related to the location of the study (e.g., country specific), the comment was placed in “not relevant to United States”; however, when generalizability was broader, comments were placed in “limited generalizability.” To support findings, where appropriate, we provide direct quotes from associate editors’ comments. Quotes include a unique numerical identifier that identifies the manuscript from which it was extracted.

Back to Top | Article Outline

Results

AM associate editors rejected 369 manuscripts after internal editor review, citing on average 3.11 reasons for rejection per manuscript. The number of reasons for rejection per manuscript ranged from 1 to 9, with a standard deviation of 1.704. In total, 96 Articles (26%), 245 Research Reports (66%), and 28 Innovation Reports (8%) were reviewed.

The qualitative coding process resulted in 29 reasons for rejection (Figure 2), clustered under 9 themes. These identified reasons overlap with Bordage’s “Top 20 Reasons Reviewers Recommend Rejection,”5 such that 9 identified reasons for rejection (31%) exactly matched Bordage’s earlier findings. Additionally, we divided 3 of Bordage’s reasons into 9 more granular reasons (31%). We also modified 3 of Bordage’s reasons (10%) and merged 2 of Bordage’s codes into 1 (3%). Three of Bordage’s codes were removed (10%) as they were not identified in the editors’ comments. Finally, 6 (23%) new codes emerged: “clinically, not particularly applicable”; “limited generalizability”; “other publishing considerations”; “prior, concurrent, or previous submission”; “IRB, copyright, or authorship issue”; and “weak effect size and/or insufficient power.”

Figure 2

Figure 2

The most prevalent theme was “ineffective study question and/or design” (338; 92%). Within this theme and across all codes, the reason “study design and/or question fail to move the literature forward” (161; 44%) was the most frequently occurring reason for rejection.

Table 1 outlines all the reasons for rejections, categorized into 9 themes: ineffective study question and/or design (338; 92%); suboptimal data collection process (180; 49%); weak discussion and/or conclusions (165; 45%); unimportant or irrelevant topic to the journal’s mission (137; 37%); weak data analysis and/or presentation of results (120; 33%); text difficult to follow, to understand (89; 24%); inadequate or incomplete introduction (67; 18%); other publishing considerations (42; 11%); and issues with scientific conduct (20; 5%). All data in the table, including themes and reasons for rejection within themes, are presented in decreasing order.

Table 1

Table 1

Sixty (16%) manuscripts were rejected based on editor comments that align with only one reason for rejection. The top four single reasons for rejection were “falls outside the scope of the journal” (13; 22%); “study design and/or question fail to move the literature forward” (10; 17%); “not relevant to United States” (7; 12%); and “prior, concurrent, or previous submission” (6; 10%). Thirteen additional single reasons for rejection accounted for the remaining 40% of manuscripts. Of the manuscripts rejected for a single reason, 5 reasons accounted for 60% of these rejections.

In several cases, we identified that a single comment could fit into multiple reasons for rejection. For example, an editor’s comment reads, “There is insufficient information in the manuscript to know if anything would be generalizable to the U.S. or other French hospitals” (#27). This comment relates to three reasons for rejection: not relevant to United States, insufficient data presented, and limited generalizability.

Back to Top | Article Outline

Discussion

Overview

The current study provides authors with nine themes why submitted manuscripts to AM were not sent on to external peer review. These can equip researchers with an organized set of categories for consideration in their manuscripts to improve their chances for a successful review. First, our analysis demonstrates that the lack of a clear and important research question and/or an inadequate study design often resulted in a rejection. We hope that knowledge of this flaw, prior to manuscript submission, can help authors avoid this common pitfall. Second, we have opened a window into the previously opaque process of internal editor review. This transparency stands to benefit not only authors but also potentially other journal editors seeking to implement similar peer review practices. Third, our analysis also provides some validity evidence that previously published reasons for rejection5 pertaining to external peer review are applicable to the internal editor review process.

In many cases, our identified reasons for internal editor review rejections aligned with those identified 20 years earlier by Bordage.5 For example, similar to Bordage, we also identified the importance of authors clearly articulating their research questions and study design. While in many cases we found alignment with Bordage’s work, we also modified and expanded his reasons. Of note, several of these expanded reasons addressed topics of responsible conduct of research, such as prior, concurrent, or previous submission. Additionally, our findings expand on Bordage’s concept of “unimportant or irrelevant topic to the journal’s mission” by providing additional granularity for the internal editor review process. We expanded Bordage’s single broad reason for rejection into three: “not relevant in United States,” “falls outside the scope of the journal,” and “unimportant or irrelevant topic to the journal’s mission.” This expansion demonstrates that Bordage’s reasons remain valuable and can be used by future researchers, and future studies likes this one can continually evolve the reasons for rejection.

A major aim of this study was to provide authors and journal editors with knowledge of the potential pitfalls for rejection at the internal editor review stage. We next highlight three suggestions based on results for submitting authors: finding the right fit with the journal, crafting a clear research question and design, and acting responsibly as a researcher.

Back to Top | Article Outline

Find the right fit with the journal.

One of the first barriers authors cross is finding the right fit between manuscript and prospective journals. In this study over a third of articles were rejected, in part because the manuscript’s topic did not fit the journal’s mission. The importance and applicability of the topic to readers were the key considerations of editors. Our analysis of AM editors’ comments delineated “relevance” into more discrete and actionable categories to help prospective authors consider the ways in which their manuscript may fit within the journal’s scope. To ensure fit with a journal, prior to submission authors should carefully read the journal’s mission and vision statements and familiarize themselves with the topics and types of manuscripts that the journal publishes.

Back to Top | Article Outline

Craft a clear research question and design.

Identifying a research question and designing the study should occur early in the research process. Failure to get these components right has the potential to destabilize an entire project. The research question sets the stage for the study design, and they both are informed by and subsequently inform the literature. Authors need to thoroughly review the current literature to identify how their research fits and advances the field. For help with formulating a clear research question and design, researchers should consult helpful resources such as the AAMC’s “Review Criteria for Research Manuscripts”8 and several useful articles on this topic.10–13

Back to Top | Article Outline

Act responsibly as a researcher.

Researchers have an imperative to conduct their research with integrity and responsible conduct and to disseminate their findings in a manner that presents a complete and accurate picture of the methods and results of their research. In potential violation of this responsibility, several manuscripts were noted by AM editors that appeared to be examples of “salami science.” While it is difficult to define salami science, it is generally identified as occurring when portions of a larger study are published in multiple different journals without referencing the related publications or justifying the contributions for multiple publications from the same data set.

For example, publishing several studies using the same data set by the same authors can be misconstrued as multiple investigations asking and answering similar questions with different data sets. A second repercussion of salami science is findings that are not as generalizable since the findings are generated from the same data set. The study, then out of context, does not give readers the full breadth of its impact.6 When this (i.e., the submission of an article that is part of a larger data set without informing the reviewers) happens, the article under consideration may appear to make a greater contribution than it would if the other article(s) produced as part of the same study were known. Concerns regarding the responsible conduct of research in HPE have been raised by others.7 , 10 Authors and editors would benefit from clearer guidelines regarding how to identify and address concerns regarding responsible conduct of research. Freely available resources on the responsible conduct of research are available from the National Institutes of Health.14

Back to Top | Article Outline

Limitations and implications for future research

We recognize that this study reflects one journal examining its own internal editor review process. Future studies need to be conducted to determine whether reasons for rejection may be generalized to other HPE journals or journals in other fields. As comments were written by a group of three specific editors, it is possible that a different group of editors or larger number of editors might have reached different conclusions. Manuscripts rejected solely by the editor-in-chief were not included in this study because he did not routinely write comments for such articles. Additional studies, including of the editor-in-chief’s comments, are needed. Additionally, although we reviewed a large number of comments, we analyzed only one year of data. There may have been other reasons why editors chose to reject a submission prior to external peer review—our data are based on their explicit comments. As such, the editors’ comments regarding the reason for rejection may be part of the decision, but may not fully explain all the reasons a manuscript was rejected. Also, manuscripts with the same flaws described in this study likely proceed to external peer review at times. The editors also judge the importance of the topic, relationship between comments, and the overall quality of the manuscript to determine whether a submission should be sent out for external peer review. Based on our study, we would hypothesize that there are combinations of reasons for rejection that result in a higher likelihood of being rejected, but further studies need to be conducted to confirm.

Back to Top | Article Outline

Conclusion

In sum, we have presented a list of nine reasons why manuscripts are rejected from AM in the internal editor review process. We hope that these findings can facilitate authors in successfully avoiding rejection at the internal review stage. As we seek to improve the science and scholarly communication in medical education, we anticipate a more global community of scholars from multiple disciplines. This investigation is a step forward in enhancing the clarity and resources to help improve the quality of HPE submissions.

Acknowledgments: The authors wish to thank Mary Beth DeVilbiss for her assistance in supplying the data for this project.

Back to Top | Article Outline

References

1. American Journal Experts. The AJE Team. Peer review: How we found 15 million hours of lost time. http://www.aje.com/en/arc/peer-review-process-15-million-hours-lost-time/. Accessed May 10, 2017.
2. Warne V. Rewarding reviewers—Sense or sensibility? A Wiley study explained. Learn Publ. 2016;29:41–50.
3. Ralph P. Practical suggestions for improving scholarly peer review quality and reducing cycle times. Commun Assoc Inf Syst. 2016;38:13.
4. Siler K, Lee K, Bero L. Measuring the effectiveness of scientific gatekeeping. Proc Natl Acad Sci U S A. 2015;112:360–365.
5. Bordage G. Reasons reviewers reject and accept manuscripts: The strengths and weaknesses in medical education reports. Acad Med. 2001;76:889–896.
6. Norman G. Data dredging, salami-slicing, and other successful strategies to ensure rejection: Twelve tips on how to not. Adv Health Sci Educ. 2014;19:1–5.
7. Brice J, Bligh J, Bordage G, et al. Publishing ethics in medical education journals. Acad Med. 2009;84(10 suppl):S132–S134.
8. Durning JS. Review Criteria for Research Manuscripts. 2015. 2nd ed. Washington, DC: Association of American Medical Colleges; https://members.aamc.org/eweb/upload/Review%20Criteria%20For%20Research%20Manuscripts.pdf. Accessed May 13, 2017.
9. Bowen GA. Grounded theory and sensitizing concepts. Int J Qual Methods. 2006;5:12–23.
10. Ringsted C, Hodges B, Scherpbier A. “The research compass”: An introduction to research in medical education: AMEE guide no. 56. Med Teach. 2011;33:695–709.
11. Morrison J. Developing research questions in medical education: The science and the art. Med Educ. 2002;36:596–597.
12. Prideaux D, Bligh J. Research in medical education: Asking the right questions. Med Educ. 2002;36:1114–1115.
13. Artino A. Conducting Research in Health Professions Education: From Idea to Publication. 2016. Washington, DC: Association of American Medical Colleges; http://journals.lww.com/academicmedicine/Documents/ACM.LastPage.e-book.FINAL.fromW-K.pdf. Accessed May 10, 2017.
14. National Institutes of Health. Office of Intramural Research. Responsible conduct of research training. https://oir.nih.gov/sourcebook/ethical-conduct/responsible-conduct-research-training. Accessed May 10, 2017.
© 2018 by the Association of American Medical Colleges