Secondary Logo

Journal Logo

Institutional members access full text with Ovid®

The Fragility of Statistically Significant Results in Pediatric Orthopaedic Randomized Controlled Trials as Quantified by the Fragility Index

A Systematic Review

Khormaee, Sariah MD, PhD*; Choe, Judy DO; Ruzbarsky, Joseph J. MD*; Agarwal, Kunal N. MS*; Blanco, John S. MD*; Doyle, Shevaun M. MD*; Dodwell, Emily R. MD, MPH, FRCSC*

Journal of Pediatric Orthopaedics: September 2018 - Volume 38 - Issue 8 - p e418–e423
doi: 10.1097/BPO.0000000000001201
Patient Evaluation

Background: The randomized controlled trial (RCT) is the gold standard study design allowing critical comparison of clinical outcomes while minimizing bias. Traditionally clinical trials are evaluated through statistical significance, expressed by P-values and confidence intervals. However, until recently, the robustness of a study’s conclusions has been given little attention. A new metric, the fragility index, quantifies the number of patients theoretically required to switch outcomes in order to reverse the study conclusions. The primary aim of our work was to determine the fragility index of RCTs in the pediatric orthopaedic literature. The secondary aim was to determine study factors associated with lower fragility index.

Methods: Pubmed and Embase were systematically searched for pediatric orthopaedic RCTs published September 1, 2006 to September 1, 2016. Two independent reviewers screened titles, abstracts, and manuscripts to identify studies published in English involving 2 treatment arms. Trials without dichotomous primary or secondary outcomes or with patients >18 years were excluded. Data were extracted from each eligible article in duplicate and the fragility index was determined using Fisher exact test, with previously published methods. Univariate analysis was used to determine factors associated with lower fragility index.

Results: Seventeen trials were eligible for inclusion. The median treatment arm size was 58 and overall sample size was 116 patients. The median fragility index was 3 (range, 0 to 18). A fragility index of 3 means that just 3 patients would need to switch treatment outcomes in order for the trial results to become statistically nonsignificant. In 1 study, the number of patients lost to follow-up exceeded the fragility index, such that the study conclusions could be completely reversed purely depending on the outcomes of the patients lost to follow-up. Lower fragility index was associated with smaller patient sample sizes and greater P-values.

Conclusions: The fragility index is a useful adjunct metric to the P-value and confidence intervals, allowing analysis of the robustness of study conclusions. RCTs in pediatric orthopaedics often have small sample sizes, many with low fragility indices. Future efforts could focus on encouraging institutional collaboration and patient recruitment with the ultimate goal of improving RCT sample sizes, and potentially improving the robustness of RCT results.

Level of Evidence: Level I.

*Hospital for Special Surgery

New York Presbyterian, University Hospitals of Columbia and Cornell, New York, NY

S.K., J.S.B., S.M.D.: contributed to study design, interpretation of results, and drafting of the manuscript. J.C., J.J.R., K.N.A.: contributed to data acquisition and analysis and drafting of the manuscript. E.R.D.: conceptualized the study, contributed to study design, interpretation of results, drafting of the manuscript, and has approved of the final version to be published, and agrees to be accountable for all aspects of the work.

Reprints: Emily R. Dodwell, MD, MPH, FRCSC, Hospital for Special Surgery, 535 East 70th Street, New York, NY 10021. E-mail:

Copyright © 2018 Wolters Kluwer Health, Inc. All rights reserved.