On the Use and Value of Reporting Guidelines in Health Professions Education Research

Conversations about the use and value of reporting guidelines in research publications often elicit polarized views. Such views signal a need for more information and thoughtful consideration. As an example from our own field of health professions education, the editors’ panel discussion at the 2019 meeting of the Association for Medical Education in Europe in Vienna, Austria, entitled Controversies and Challenges in Publishing Health Professions Education Research, prompted a lively discussion among editors and authors around the use of reporting guidelines in health professions education research publications. Some participants strongly endorsed use of such guidelines, while others staunchly opposed them. In this editorial, we aim to inform the Academic Medicine community about reporting guidelines, share our perspective on them as editors of the journal, and promote ongoing dialogue on these guidelines.

What Are Reporting Guidelines?

Reporting guidelines list and describe the key information that authors should include in a manuscript that describes a research study or literature review, to ensure that readers can understand, evaluate, and/or replicate the study or review.¹ Reporting guidelines complement a journal’s instructions for authors, which tend to focus on technical requirements such as word count and structure,² and differ from critical appraisal tools, which are used to assess the trustworthiness, relevance, and quality of evidence presented in a manuscript or publication.^3–5 Some maintain that evaluation of thorough information reporting should be kept distinct from the critical appraisal process, since incomplete reporting is often interpreted as a sign of low methodologic quality when, in fact, it may be a decision made by authors to abide by word limits or a simple oversight.⁶ Others consider thorough reporting of information to be a necessary precursor to critical appraisal and part of a process that fulfills high scientific standards.^7,8

A Brief History of Reporting Guidelines

Reporting guidelines evolved over time in response to concerns about the quality of evidence published in scientific journals and the risks associated with trusting what may turn out to be weak evidence in practical decision making. Across many fields of medicine, researchers’ efforts to verify evidence by replicating study findings have been confounded by insufficient information about study methods and results.⁹ Two reporting guidelines for randomized controlled trials in biomedical research were independently developed to address this problem; the 2 were ultimately combined to form CONSORT (Consolidated Standards of Reporting Trials).^9,10

The perceived value of reporting guidelines has increased, given current trends in scholarship such as the Open Science movement, which calls for greater transparency in research,¹¹ and the recent boom in systematic reviews, meta-analyses, and other forms of knowledge synthesis.^1,12 CONSORT laid the foundation for subsequent reporting guidelines spanning multiple genres and approaches to scholarship—for example, CARE (for case reports),¹³ PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses),^14,15 SQUIRE (Standards for Quality Improvement Reporting Excellence) 2.0,¹⁶ COREQ (Consolidated Criteria for Reporting Qualitative Research),¹⁷ and STROBE (Strengthening the Reporting of Observational Studies in Epidemiology).¹⁸

While many of the early guidelines originated in biomedical and health research, scholars in the social sciences and education adopted, adapted, and created their own versions of reporting guidelines. Guidelines developed for health professions education include SQUIRE-EDU (Standards for Quality Improvement Reporting Excellence in Education),¹⁹ GREET (Guideline for Reporting Evidence-Based Practice Educational Interventions and Teaching),²⁰ and SRQR (Standards for Reporting on Qualitative Research.²¹ Other guidelines frequently referenced in health professions education include CHERRIES (Checklist for Reporting Results of Internet E-Surveys),²² PRISMA,^14,15 RAMESES (Realist and Meta-Narrative Evidence Syntheses: Evolving Standards),²³ and STROBE.¹⁸Table 1 presents examples of guidelines that are applicable to research in health professions education.

Table 1: Examples of Guidelines Applicable to Research in Health Professions Education

The proliferation of reporting guidelines created a need for quality control and management to reduce redundancy and enhance dissemination. The EQUATOR (Enhancing the Quality and Transparency of Health Research) Network, established in 2008, has played a large role in shaping how members of the biomedical and social science communities conceptualize reporting guidelines.¹ The EQUATOR Network states that the purpose of reporting guidelines is “to remind researchers of what information to include in the manuscript, not to tell them how to do research.”^9(p73)

Conflicting Arguments on the Value of Reporting Guidelines

Use of reporting guidelines varies. Some journals require authors to complete reporting guideline checklists when submitting articles, other journals encourage reviewers to refer to reporting guidelines when completing reviews, and still other journals do not mention reporting guidelines at all.^24,25 Proponents tend to view reporting guidelines as a welcome tool for transparency. Guidelines make basic standards and expectations freely available to all, regardless of publication experience. Guidelines may also improve the quality of scholarship by reducing the omission of key information, a problem noted in several studies that have evaluated published articles for reporting quality.^7,26 Reporting key information about a study in a consistent way rewards authors who invest in rigorous methodology.²⁷

Those opposed to reporting guidelines raise concerns about their prescriptive and uncritical use by authors and reviewers.²⁸ The conversion of reporting guidelines into reporting “checklists” can inhibit creativity and imply that the approaches or techniques that conform to guidelines are more valuable than those that do not. The use of reporting guidelines may disadvantage researchers with fewer resources who report smaller-scale studies with visible limitations (e.g., design flaws, lower response rates).²⁹ Some also argue that guidelines are not necessary in a robust peer review system, as it is the duty of expert reviewers to evaluate the sufficiency of information provided by authors and request additional clarification as needed.²⁹

Editors and reviewers appreciate the need for nuance. While it is true that the quality standards for medical education research have been heavily influenced by those of biomedical research, the influx of robust social science research has diversified the reviewer pool, expanded the expertise of journals’ editorial teams, and encouraged many thoughtful discussions of how to ensure fair and accurate evaluation of manuscripts that reflect different philosophies of science and employ methodologies from various disciplines.³⁰

Our View

As the editorial team at Academic Medicine, we advocate judicious use of reporting guidelines. Authors, reviewers, and journal editors must apply good judgment when deciding what information must be reported in a given study to maximize clarity and coherence. In some circumstances, it may be acceptable not to include certain information (e.g., information that would compromise participant anonymity, that cannot be obtained, or that does not make sense in the context of the particular study design or methodology or available resources). After all, efforts to investigate many important educational research questions are limited by the amount and quality of the research that aims to answer them. In these cases, new studies, even with important methodological limitations of their own, can substantially and meaningfully add to the literature. This recognition means that we all have to be open to judging research in light of what it adds to our knowledge and not judging that research only by reporting standards that may be overly stringent given the state of a field.

Education research grows by accretion, and small gains can be valuable. Furthermore, authors may write excellent articles and reviewers may complete superb reviews without using reporting guidelines. We therefore do not mandate the use of reporting guidelines, yet we believe that reporting guidelines have value. In the sections that follow, we describe the ways that we envision authors, reviewers, and editors using reporting guidelines.

Authors

We encourage authors to become familiar with reporting guidelines endorsed by the EQUATOR Network as well as those commonly used in social sciences such as education^31–33 and psychology.^34,35 Judicious use means that authors review reporting guidelines carefully, selecting the guidelines most appropriate for their work, and confidently deciding when certain guideline items do not apply. Authors always have the option to explain their rationale to reviewers and editors if they are asked to provide information that they think is not appropriate for their study.

In short, we advise authors to refer to guidelines as a resource throughout the design, conduct, and write-up of a study.²⁸ We do not mandate the use of reporting guidelines for any submission to Academic Medicine. As previously indicated, reports involving novel scholarly questions, or questions that can only be addressed in lower-resources settings, may be more modest or less adherent to reporting guidelines and still represent truly important contributions to the literature. Authors must be careful when making claims or interpretations and should clearly state the limitations of their work. All research reports should be transparent in their descriptions of their methodologies.

The goal is to make research accessible to a diverse audience, not to overly standardize research reports. This ambitious goal requires authors, reviewers, and editors to negotiate a delicate balance between (1) a clear, comprehensive description of the researchers’ assumptions, approach, processes, and findings and (2) respect for researchers’ integrity, ingenuity, and judgment. Finding this balance is particularly important when researchers draw on approaches and methodologies that are well established outside of health professions education.

Reviewers

We rely heavily on the expertise of peer reviewers, who are a critical part of our scholarly community and process. We are committed to finding reviewers with the necessary expertise to fairly and critically judge the quality of manuscripts. At the same time, we recognize that the breadth of philosophical orientations and methodologies in a multidisciplinary field such as health professions education makes it challenging to identify reviewers with both content and methodological expertise for all studies, especially for studies introducing new ideas and approaches to health professions education research. We see reporting guidelines as a potentially helpful resource for reviewers who encounter unfamiliar methodology in an area where they may have content expertise. We also see reporting guidelines as useful reminders for experts who may know the methodology well and take for granted decisions, procedures, or terminology that warrant more explanation for nonexpert readers. In sum, we view guidelines as a tool to enhance the excellent insights that reviewers bring to each manuscript, not as an attempt to standardize, undermine, or de-skill the review process.

Of note, Academic Medicine provides review criteria for research manuscripts (available at https://store.aamc.org/review-criteria-for-research-manuscripts.html). Review criteria are meant to assist reviewers in evaluating manuscripts and providing helpful feedback to authors and editors.

Editors

As the journal’s editorial team, we are committed to ensuring that manuscripts accepted for publication provide readers with the information they need to judge the credibility of studies and make informed decisions about how to use study findings. Some manuscripts will require more words than the suggested limits to convey their messages clearly, and we are willing to consider this on a case-by-case basis. Every manuscript submitted to Academic Medicine is also assessed by at least 2 members of the editorial and staff team. We learn about new methodologies ourselves and discuss challenging decisions about manuscripts to ensure that we bring multiple perspectives to bear. While we do not require authors to use reporting guidelines as a checklist, we may recommend that authors review a particular guideline as a way to ensure that they have optimized the reporting of their work.

Summing Up

Our recommendation of judicious use of reporting guidelines will, we hope, unify different perspectives on the role of such guidelines in health professions education publications. We aim to increase awareness of the existence of reporting guidelines, encourage ongoing education about them, and stimulate thoughtful dialogue about the use of reporting guidelines in our community. Ongoing dialogue must include consideration of the utility and relevance of reporting guidelines for research in health professions education and, ideally, will lead to new opportunities for research on the value of reporting guidelines in fostering clear communication and furthering rigor in our field.