Secondary Logo

Journal Logo

Invited Opinion

Improving Reproducibility and Transparency in Shock

the Arrive Guidelines Need Better Implementation and Enforcement

Reynolds, Penny S.

Author Information
doi: 10.1097/SHK.0000000000001388
  • Free
  • Editor's Choice

Abstract

The ARRIVE (Animals in Research: Reporting In Vivo Experiments) guidelines were developed to improve animal research reporting standards. Since their launch in 2010 (1), they have been endorsed by over 1,000 journals, numerous scientific societies, and funding agencies. The Shock Society officially endorsed ARRIVE in 2012 (2).

Unfortunately, the purpose and intent of ARRIVE seem to be widely misunderstood by both investigators and reviewers. The guidelines are not a statement of investigator compliance with ethical care and treatment standards. Instead, ARRIVE is a checklist of the most essential items of information to be reported to allow independent appraisal of the research for reproducibility, relevance, and conformity with best scientific practice (1).

Six years after endorsement by the Shock Society, reporting standards in Shock are still weak and have persisted with little sign of correction. I base this conclusion on a formal assessment of 100 randomly selected animal-based research articles published in Shock between 2014 and 2018, and evaluated for methodological quality and welfare items (ARRIVE items 1, 2, 5 through 16). A full description of research methods and conclusions will be submitted separately; findings are briefly summarized in this letter. Over half to two-thirds of surveyed papers omitted verifiable ethical oversight information, animal “demographic” information (age, sex, weight, caging, and housing), and baseline descriptive data. “Core” statistical methodology items recommended by both ARRIVE and the National Institute of Health (NIH) (including total and group sample sizes, sample size justification, bias minimization methods such as randomization, blinding, and experimental design, and appropriate statistical analyses) were rarely described in sufficient detail to meet best-practice requirements, if they were reported at all. Paradoxically, there was considerable overreliance on hypothesis testing with reporting of excessive numbers of “statistically significant” P values. Claims of effect were further weakened by major and persistent errors in basic statistical analyses. However, most troubling was the poor reporting of welfare-related information. Very few studies documented use of preemptive, concurrent, and follow-up use of appropriate pain relief medications (as required for major and/or multiple survival surgeries), unambiguous humane endpoint criteria, welfare monitoring procedures, or even how animals were euthanized. A number of studies described use of anesthetic agents or methods of euthanasia prohibited or discouraged by US institutions.

Lack of compliance results in part from lack of enforcement. As gatekeepers, reviewers must share this responsibility. When reviewers certify that the manuscript under review contains “sufficient information to comply with the ARRIVE guidelines,” this is a guarantee that publication in Shock denotes that animals have been treated ethically and humanely, and that the research is of the highest possible scientific quality. However, it is apparent that the relevant confirmatory information is consistently overlooked during the review process. The solution is that reviewers must actively use the checklist to assist in manuscript evaluations. Rather than representing an additional administrative burden, checklists can actually speed up the review process by identifying key items required to assess methodological quality, thus allowing the reviewer to perform rapid triage. Poorly reported manuscripts requiring substantial revision can be sent back that much sooner, and priority given to ethical, well-executed, and well-documented studies.

However, the primary responsibility for poor reporting compliance belongs to investigators and authors. If investigators are consistent in using the ARRIVE checklist during manuscript preparation, many reporting deficiencies could be eliminated. However, actual compliance begins in the planning phases of an experiment, long before submission of the manuscript for review. At the beginning of a study, the investigator should consider consulting general best-practice planning checklists, such as PREPARE: Planning Research and Experimental Procedures on Animals: Recommendations for Excellence(3). Familiarity with current best-practice recommendations should be indispensable for planning disease (4, 5) and injury (6) animal models to ensure model fidelity and clinical relevance, and reduce disparities between ‘competing’ models. Finally, consultation with a professionally trained statistician with expertise in experimental design is highly recommended, so that statistical experimental design and bias minimization strategies are built into the protocol before study implementation.

The problems outlined above are unfortunately not unique to Shock, but seem to be widespread and common to nearly all journals featuring animal-based research (7–9). The poor methodological quality of many animal studies means that evidence for efficacy and potential risk–benefit to humans is unreliable, potentially misleading, and a major obstacle to translation (10, 11). Investigators must understand that poorly designed experiments, misuse of statistics, and poor reporting of methods and results are negligent and unethical (12, 13). Improvement in standards and greater rigor in reporting will lead to improvement in experimental quality overall, and single out Shock and the Shock Society as leaders and innovators in shock-allied basic research.

REFERENCES

1. Kilkenny C, Browne WJ, Cuthill IC, Emerson M, Altman DG. Improving bioscience research reporting: the ARRIVE guidelines for reporting animal research. PLoS Biol 8 (6):e1000412, 2010.
2. Reynolds PS, Wall P, van Griensven M, McConnell K, Lang C, Buchman T. Shock supports the use of animal research reporting guidelines. Shock 38 (1):1–3, 2012.
3. Smith AJ, Clutton RE, Lilley E, Hansen KEA, Brattelid T. PREPARE: guidelines for planning animal research and testing. Lab Animal 52 (2):135–141, 2018.
4. Osuchowski MF, Ayala A, Bahrami S, Bauer M, Boros M, Cavaillon JM, Chaudry IH, Coopersmith CM, Deutschman CS, Drechsler S, et al. Minimum Quality Threshold in Pre-Clinical Sepsis Studies (MQTiPSS): an international expert consensus initiative for improvement of animal modeling in sepsis. Shock 50 (4):377–380, 2018.
5. Zingarelli B, Coopersmith CM, Drechsler S, Efron P, Marshall JC, Moldawer LL, Wiersinga WJ, Xiao X, Osuchowski MF, Thiemermann C. Part I: Minimum Quality Threshold in Preclinical Sepsis Studies (MQTiPSS) for study design and humane modeling endpoints. Shock 51 (1):10–22, 2019.
6. Tremoleda JL, Watts SA, Reynolds PS, Thiemermann C, Brohi K. Modeling acute traumatic hemorrhagic shock injury: challenges and guidelines for preclinical studies. Shock 48 (6):610–623, 2017.
7. Avey MT, Moher D, Sullivan KJ, Fergusson D, Griffin G, Grimshaw JM, Hutton B, Lalu MM, Macleod MR, Marshall J, et al. The devil is in the details: Incomplete reporting in preclinical animal research. PLoS One 11 (11):e0166733, 2016.
8. Kilkenny C, Parsons N, Kadyszewski E, Festing MF, Cuthill IC, Fry D, Hutton J, Altman DG. Survey of the quality of experimental design, statistical analysis and reporting of research using animals. PLoS One 4 (11):e0007824, 2009.
9. Macleod MR, Lawson McLean A, Kyriakopoulou A, Serghiou S, de Wilde A, Sherratt N, Hirst T, Hemblade R, Bahor Z, Nunes-Fonseca C, et al. Risk of bias in reports of in vivo research: a focus for improvement. PLoS Biol 13 (11):e1002301, 2015.
10. Henderson VC, Kimmelman J, Fergusson D, Grimshaw JM, Hackam DG. Threats to validity in the design and conduct of preclinical efficacy studies: a systematic review of guidelines for in vivo animal experiments. PLoS Med 10:e1001489, 2013.
11. Wieschowski S, Chin WWL, Federico C, Sievers S, Kimmelman J, Strech D. Preclinical efficacy studies in investigator brochures: do they enable risk-benefit assessment? PLoS Biol 16 (4):e2004879, 2018.
12. Altman DG. Statistics and ethics in medical research. Misuse of statistics is unethical. BMJ 281 (6249):1182–1184, 1980.
13. MacCallum CJ. Reporting animal studies: good science and a duty of care. PLoS Biol 8 (6):e1000413, 2010.
Keywords:

Animal Research; ARRIVE; checklist; editorial; reporting guidelines

© 2020 by the Shock Society