Share this article on:

The Statistical Checklist and Statistical Review: Two Essential Yet Challenging Deliverables

Mascha, Edward J. PhD; Vetter, Thomas R. MD, MPH

doi: 10.1213/ANE.0000000000001863
Editorials: Editorial

From the *Departments of Quantitative Health Sciences and Outcomes Research, Cleveland Clinic, Cleveland, Ohio; and Department of Surgery and Perioperative Care, Dell Medical School at the University of Texas at Austin, Austin, Texas.

Accepted for publication December 1, 2016.

Funding: None.

The authors declare no conflicts of interest.

Reprints will not be available from the authors.

Address correspondence to Thomas R. Vetter, MD, MPH, Department of Surgery and Perioperative Care, Dell Medical School at the University of Texas at Austin, Dell Pediatric Research Institute, Suite 1.114, 1400 Barbara Jordan Boulevard, Austin, TX 78723. Address e-mail to thomas.vetter@austin.utexas.edu.

“A little learning is a dangerous thing; drink deep, or taste not the Pierian spring: there shallow draughts intoxicate the brain, and drinking largely sobers us again.”

—from An Essay on Criticism by English poet Alexander Pope (1688–1744).

In this issue of Anesthesia & Analgesia, Dexter and Shafer1 describe the published experiences with the statistical review process. In their present narrative review, they specifically (1) examined scientific studies of several well-publicized efforts to improve statistical reporting; and (2) reviewed several retrospective assessments of the impact of these efforts. In this accompanying editorial, we offer our perspective on Dexter and Shafer’s 4 salient conclusions1 as well as our proposed solutions to the continued challenges and dilemma of how to achieve critically important statistical rigor in our literature in a consistent, efficient, and equitable manner.

Back to Top | Article Outline

CONCLUSION 1: “INSTRUCTIONS TO AUTHORS AND STATISTICAL CHECKLISTS ARE NOT SUFFICIENT; NO FINDINGS SUGGESTED THAT EITHER IMPROVES THE QUALITY OF STATISTICAL METHODS AND REPORTING.”1

We agree that it will not be sufficient for journal editors to create a statistical checklist and then simply suggest or recommend its use in their “Instructions for Authors.” Nevertheless, we strongly believe that a statistical checklist can play an important, helpful, and necessary role in the manuscript review process. Having a statistical checklist in place and calling authors’ attention to it allows authors to standardize their submissions and the journal to streamline its manuscript statistical review process. Practically speaking, when authors proactively address common deficiencies in their initial manuscript submission by using a statistical checklist, doing so can often save at least 1 round of peer review—a win for everyone.

One should take note of the results of the intriguing 2 × 2 randomized factorial design2 discussed by Dexter and Shafer1 in which having a statistical review substantially increased the quality of manuscripts as measured on the Goodman scale, but recommending statistical guidelines to authors did not. A key feature of this study’s design was that guidelines were only “recommended” to authors in the review process, not required, and there was no follow-up to verify or insist that they be used. On the contrary, we believe that a clearly defined statistical checklist or guideline needs to be enforced by the editorial board and the statistical editor/team to be effective.

One feasible approach would thus require authors to complete the checklist before a submitted manuscript undergoes initial peer review. The statistical editor or statistical team could then readily confirm whether all the specific items have been included or properly addressed. We furthermore suggest that to be most effective and impactful:

  • The statistical checklist is created and codified by a suitably qualified (“bona fide”) statistical editor and/or group of similarly qualified statistical reviewers.
  • The primary editors (executive editors who “handle manuscripts”) routinely refer authors to the statistical checklist and related manuscript guidelines adopted by the journal: this acting as the initial “screening” before a manuscript is sent to a statistician for more detailed review of the items.
  • The journal’s primary editors actively and consistently support the efforts of the statistical editor and/or statistical reviewers in holding authors accountable for any identified statistical checklist deficiencies.
  • The requirement for author adherence to the statistical checklist must ultimately be consistently enforced by the journal’s editorial leadership (its editor-in-chief).

Anesthesia & Analgesia currently has a recommended “Basic Statistical Methods and Reporting Mini-Checklist” within its “Instructions for Authors.”a The Journal will likely implement a more comprehensive and required manuscript statistical checklist in the near future.

Back to Top | Article Outline

CONCLUSION 2: “EVEN BASIC STATISTICS SUCH AS POWER ANALYSES ARE FREQUENTLY MISSING OR INCORRECTLY PERFORMED.”1

We agree that the fundamentals of inferential statistics are frequently performed incorrectly or missing altogether in the medical literature. This has been bluntly stated in the 2014 Statistical Analyses and Methods in the Published Literature (SAMPL) guidelines: “The truth is that the problem of poor statistical reporting is long-standing, widespread, potentially serious, concerns mostly basic statistics, and yet is largely unsuspected by most readers of the biomedical literature.”3

Closer to home, this predicament was highlighted in a 2011 editorial by Dexter4 that summarized the recurring fundamental statistical issues identified in a sample of 200 articles submitted to Anesthesia & Analgesia. Some of these were basic elements (eg, confidence intervals; multiple measurements of a primary outcome variable over time, without a correction of α to mitigate the risk of a Type I or “false-positive” error).

These issues have persisted in Anesthesia & Analgesia during 2016 with the current common problems being inappropriate or incorrectly applied statistical tests; no clearly predefined primary outcome(s); inadequate or nonexistent adjustment for confounding; missing/incomplete/inaccurate sample size justification; lack of reporting of treatment effect estimates and confidence intervals; and overstated conclusions based on the study design or reported results.

The advent and widespread availability of statistical software with a very approachable graphical user interface has obviated the need to master arcane programming code (“machine language”).5 This has likely empowered many well-intended and resource-constrained researchers to pursue a “do-it-yourself” (“DYI”) and “cookbook” approach to data analysis6—thus circumventing the additional expense but also the expertise of a consultant biostatistician or epidemiologist.

Back to Top | Article Outline

CONCLUSION 3: “STATISTICAL REVIEW IS NEEDED FOR ALL MANUSCRIPTS THAT INVOLVE DATA ANALYSIS.”1

We agree that a consistent statistical review is needed for all manuscripts submitted to Anesthesia & Analgesia that report data and undertake data analysis.

Based on our 1-year experience at the helm of Anesthesia & Analgesia, virtually every (>99%) submitted manuscript that is not rejected outright during the initial round of scientific review requires some improvement in its statistical methods and/or data reporting. Our statistical review includes all manuscripts with the goal of making inference from a sample to a population, including either (1) estimating an unknown parameter like the incidence of an event, prevalence of a condition, or correlation; or (2) making an inference about a treatment effect on outcomes of interest. This comprises clinical trials, observational studies, estimation studies, surveys, and laboratory medicine—basically all manuscripts including numbers—other than a true case report or limited case series.

However, a major logistic question is when should this statistical review occur? Anesthesia & Analgesia has chosen, at least for the time-being, to send all manuscripts for formal statistical review that are not rejected outright during the initial round of scientific peer review. Our current workflow, although more protracted, ostensibly lessens the likelihood of an additional, unexpected (and unwelcomed), later revision by the authors; however, it places greater demands on our statistical reviewers and requires greater resources (“bandwidth”).

Back to Top | Article Outline

CONCLUSION 4: “NONSTATISTICAL REVIEWERS (EG, SCIENTIFIC REVIEWERS) AND JOURNAL EDITORS GENERALLY PERFORM POORLY ASSESSING STATISTICAL QUALITY.”1

We agree that scientific reviewers and journal editors generally lack the necessary expertise to perform a consistently adequate statistical review.7 In fairness to these vital team members of a journal like Anesthesia & Analgesia—most of whom are practicing physicians and/or clinician scientists—traditional undergraduate medical school and postgraduate residency/fellowship curricula do not include basic inferential biostatistics, let alone more advanced data analysis methods.

Furthermore, there is an innate division of labor and hence required skill set with peer reviewers and journal editors expected to assess the rationale, novelty, scientific rigor, potential impact, and other merits and flaws of a research study8,9—a fundamentally different assessment than that of statistical reviewers.7 We believe that these respective content experts can and should naturally complement one another’s journal-related efforts and contributions.

Back to Top | Article Outline

Goals of the Statistical Review

The overarching goals of the statistical review are (1) to ensure that the study was properly conducted and the results were presented appropriately; and (2) to identify possible errors and omissions.7 Depending on the study design, the statistical review can combine statistical and epidemiologic principles.7 Specific goals of the statistical review include assuring that:

  • Basic study design and the study hypotheses are stated clearly and logically.
  • Primary (and secondary) outcomes are defined clearly and logically, flowing from the stated study aims and hypotheses.
  • Statistical methods and results are presented clearly and logically, flowing from the stated study hypotheses and including only the initially identified primary (and secondary) outcomes.
  • Sample size is always justified with either an estimate of the accompanying power to detect a given or minimally important difference or a statement of the available confidence interval width (ie, “precision”) for the primary analysis.
  • Risks of a Type I error and Type II error are minimized.
  • In addition to P values, estimates of treatment effect or observed associations along with confidence intervals are provided.
  • Appropriate—but not necessarily the arguably “best” available—statistical methods are used. Statistical methods have increased in complexity in parallel with the complexity of biomedical research and available data and databases. For example, data about confounders are now much more available and in greater number and, therefore, complex methods to adjust for them have been developed. Simulation studies, before-and-after designs, and genetic searches all require more complex statistical methods than previously used. For these reasons, we highly advise that a trained statistician or epidemiologist be included in study design and analysis for most studies.
Back to Top | Article Outline

Our Educational Plan

In addressing the process and performance improvement opportunities identified here by Dexter and Shafer,1 the adage that “a rising tide lifts all boats” is apropos. We will thus build on the efforts of our predecessors at Anesthesia & Analgesia by further educating our entire stakeholder audience about basic study design, statistical analysis, and data reporting. This additional education will coincide with a manuscript statistical checklist implemented by the Journal.

As thoughtfully proposed by one of our current Senior Editors, this will in part take the form of an ongoing series of short tutorials, each of which will accompany a selected article in that issue of the Journal, highlighting its exemplary study design, statistical analysis, and/or data reporting. These short tutorials will parallel the recommendations of the 2014 SAMPL guidelines.3 The goal of these short tutorials will be to increase awareness and basic understanding, thus fostering collegial dialogue. These short tutorials will be occasionally be supplemented by “Statistical Grand Rounds” papers that describe more specialized, innovative, and controversial topics.

Hopefully, we will thus achieve a better future state, acknowledging that as William Camden (1551–1623), English antiquarian, historian, topographer, and herald, opined: “All the proof of a pudding, is in the eating.”

Back to Top | Article Outline

DISCLOSURES

Name: Edward J. Mascha, PhD.

Contribution: This author helped write and revise the manuscript.

Name: Thomas R. Vetter, MD, MPH.

Contribution: This author helped write and revise the manuscript.

This manuscript was handled by: Jean-Francois Pittet, MD.

Back to Top | Article Outline

FOOTNOTE

a Anesthesia & Analgesia: “Instructions for Authors.” Available at: http://edmgr.ovid.com/aa/accounts/ifauth.htm. Accessed November 24, 2016.
Cited Here...

Back to Top | Article Outline

REFERENCES

1. Dexter F, Shafer SL. Narrative review of statistical reporting checklists, mandatory statistical editing, and rectifying common problems in the reporting of scientific articles. Anesth Analg. 2017;124:943–947.
2. Cobo E, Selva-O’Callagham A, Ribera JM, Cardellach F, Dominguez R, Vilardell M. Statistical reviewers improve reporting in biomedical articles: a randomized trial. PLoS One. 2007;2:e332
3. Lang TA, Altman DG. Moher D, Altman DG, Schulz KF, Simera I, Wager E. Statistical analyses and methods in the published literature: the SAMPL guidelines. Guidelines for Reporting Health Research: A User’s Manual. 2014;Vol 10Oxford, UK: John Wiley & Sons, Ltd1–10.
4. Dexter F. Checklist for statistical topics in Anesthesia & Analgesia reviews. Anesth Analg. 2011;113:216–219.
5. Comrey AL, Lee HB. Learning statistics. Elementary Statistics: A Problem Solving Approach. 20064th edMorrisville, NC: Lulu.com1–8.
6. Norman GR, Streiner DL. Biostatistics: The Bare Essentials. 20083rd edHamilton, Ontario, Canada: Decker.
7. Petrovečki M. The role of statistical reviewer in biomedical scientific journal. Biochemia Medica. 2009;19:223–230.
8. Del Mar C, Hoffmann TC. A guide to performing a peer review of randomised controlled trials. BMC Med. 2015;13:248
9. Lovejoy TI, Revenson TA, France CR. Reviewing manuscripts for peer-review journals: a primer for novice and seasoned reviewers. Ann Behav Med. 2011;42:1–13.
© 2017 International Anesthesia Research Society