We analyzed 5 high-frequency ventilation studies with a total number of 2152 randomized patients. Those 5 trials compared a high-frequency oscillatory ventilator with conventional mechanical ventilation. Table 1 provides numbers of the outcomes of interest. Taken together, those 5 studies showed that high-frequency oscillatory ventilation had an OR of 0.92 (95% confidence interval [CI] = 0.77–1.09) for death or chronic lung disease, 0.98 (0.80–1.21) for chronic lung disease in survivors, 1.01 (0.79–1.29) for intraventricular hemorrhage grade III and IV, and 0.90 (0.62–1.33) for periventricular leukomalacia.
Treatment with high-frequency oscillatory ventilation was comparable among trials (Appendix Table available with the online version of this article). The ventilation strategy in the conventional mechanical ventilation groups also did not differ much among trials. Inconsistency in primary outcome assessed by I2 was 7.5% indicating a low percentage of total variation across studies due to heterogeneity.
Sequential meta-analysis showed that the first of the 5 trials was enough to rule out a 15% reduction in death or chronic lung disease (Fig. 2A). In a sensitivity analysis that decreased the effect to a reduction of 10%, it took only 2 trials before the boundary for “no reduction” was crossed (OR = 0.97; 95% CI = 0.68–1.41) (Fig. 2B). Sensitivity analysis excluding the studies by Thome et al11 and Moriette et al13 resulted in an OR of 0.98 (0.68–1.39) (data not shown). The same result was found with chronic lung disease alone as the outcome, with an estimated effect of 15% reduction (Fig. 2C). After one trial (by Thome et al11), the boundary for “no reduction” was crossed (0.89; 0.50–1.58). Sequential analyses were also applied with intraventricular hemorrhage grade III and IV and periventricular leukomalacia as outcome measures. For both outcomes, there was not enough evidence in the 5 trials to draw a definitive conclusion (data not shown).
To be of value, a new RCT must add useful information. Assessing whether clinical equipoise was present at the start of a new RCT should be general research practice.22 As Chalmers has observed, “Science is meant to be cumulative, but many scientists are not cumulating scientifically.”23 Cumulative meta-analysis is a recognized technique for systematic review. Various authors have performed cumulative meta-analyses of RCTs.22,24 The usual approach is to analyze the available studies, testing the null hypothesis that the 2 treatments are equally effective. If the test is not statistically significant, a new trial is added (when its results become available) and the analysis is repeated. This approach continues until a statistically significant result is found, ie, until the null hypothesis is rejected. Berkey et al25 noticed that this general approach does not adjust for the multiple testing and lacks either a formal stopping rule or a way to quantify the power of the conclusion. We performed a sequential meta-analysis according to the approach of Whitehead.26 Using this approach, the overall significance level α (the type I error) is preserved, thus preventing the increase of the cumulative α by multiple testing. A prespecified power to detect a clinically relevant treatment difference is guaranteed. Furthermore, this approach permits stopping when there is enough evidence either to reject the null hypothesis of treatment equivalence or to not reject the null hypothesis.
This is a second report that discusses the relevance of new trials using sequential meta-analysis.27 We conducted this analysis retroactively and found that the first of 5 recent trials was enough to demonstrate lack of benefit. Four more studies were performed powered to show the same amount of effect.13–16 All of these trials occurred at approximately the same time, which would have limited the application of sequential meta-analysis in real time for this specific example. Nonetheless, this example demonstrates the potential use of the approach.
To compare trials, it is important that there is sufficient similarity of treatment. For example, ventilation strategies in high-frequency oscillatory ventilation and conventional mechanical ventilation have changed in recent years.18 In the previous cumulative meta-analysis, ventilation strategies were an important source of heterogeneity between trials.18 In the last 5 trials, however, ventilation strategies were comparable and results were homogeneous among trials. Only a small amount of variation among trials was due to heterogeneity of treatment.
The most important differences among the earlier trials were the use of surfactant therapy and the application of a lung protective strategy in patients on conventional mechanical ventilation.18,28 Both modalities were applied in the last 5 trials. In the only trial that showed a reduction in chronic lung disease, the conventional mechanical ventilation therapy was most rigidly controlled.14 Therefore, it seems unlikely that in daily practice, the same difference between high-frequency oscillatory ventilation and conventional mechanical ventilation will occur.29
In general, the size of a trial is estimated by a power analysis that is based on a clinically relevant effect size and chosen probabilities for type I and II errors. However, this does not answer the question whether this new trial will be able to adjust the available cumulative evidence sufficiently to conclude that a clinically relevant effect can be refuted or accepted. By performing a sequential analysis (ie, a sequential meta-analysis of earlier comparable trials), it can be decided whether enough cumulative evidence has already been gathered to render another trial uninformative. In this report, we apply sequential meta-analysis to a series of randomized trials to demonstrate its use in assessing the accumulating evidence in a series of trials.
Sequential meta-analysis of earlier comparable studies should be an integral part in the planning and design of new randomized trials. As we have shown, sequential meta-analyses can provide useful information that could affect the design of further trials or even affect the decision as to whether further trials are necessary.
1. Whitehead A, Whitehead J. A general parametric approach to the meta-analysis of randomized clinical trials. Stat Med
2. Young C, Horton R. Putting clinical trials into context. Lancet
3. MacIntyre NR. Current issues in mechanical ventilation for respiratory failure. Chest
4. Froese AB, Butler PO, Fletcher WA, et al. High-frequency oscillatory ventilation in premature infants with respiratory failure: a preliminary report. Anesth Analg
5. High-frequency oscillatory ventilation compared with conventional mechanical ventilation in the treatment of respiratory failure in preterm infants. The HIFI Study Group. N Engl J Med
6. Carlo WA, Siner B, Chatburn RL, et al. Early randomized intervention with high-frequency jet ventilation in respiratory distress syndrome. J Pediatr
7. Ogawa Y, Miyasaka K, Kawano T, et al. A multicenter randomized trial of high-frequency oscillatory ventilation as compared with conventional mechanical ventilation in preterm infants with respiratory failure. Early Hum Dev
8. Wiswell TE, Graziani LJ, Kornhauser MS, et al. High-frequency jet ventilation in the early management of respiratory distress syndrome is associated with a greater risk for adverse outcomes. Pediatrics
9. Keszler M, Modanlou HD, Brudno DS, et al. Multicenter controlled clinical trial of high-frequency jet ventilation in preterm infants with uncomplicated respiratory distress syndrome. Pediatrics
10. Rettwitz-Volk W, Veldman A, Roth B, et al. A prospective, randomized, multicenter trial of high-frequency oscillatory ventilation compared with conventional ventilation in preterm infants with respiratory distress syndrome receiving surfactant. J Pediatr
11. Thome U, Kossel H, Lipowsky G, et al. Randomized comparison of high-frequency ventilation with high-rate intermittent positive pressure ventilation in preterm infants with respiratory failure. J Pediatr
12. Plavka R, Kopecky P, Sebron V, et al. A prospective randomized comparison of conventional mechanical ventilation and very early high-frequency oscillatory ventilation in extremely premature newborns with respiratory distress syndrome. Intensive Care Med
13. Moriette G, Paris-Llado J, Walti H, et al. Prospective randomized multicenter comparison of high-frequency oscillatory ventilation and conventional ventilation in preterm infants of less than 30 weeks with respiratory distress syndrome. Pediatrics
14. Courtney SE, Durand DJ, Asselin JM, et al. High-frequency oscillatory ventilation versus conventional mechanical ventilation for very-low-birth-weight infants. N Engl J Med
15. Johnson AH, Peacock JL, Greenough A, et al. High-frequency oscillatory ventilation for the prevention of chronic lung disease of prematurity. N Engl J Med
16. Van Reempts P, Borstlap C, Laroche S, et al. Early use of high-frequency ventilation in the premature neonate. Eur J Pediatr
17. Henderson-Smart DJ, Bhuta T, Cools F, et al. Elective high-frequency oscillatory ventilation versus conventional ventilation for acute pulmonary dysfunction in preterm infants. Cochrane Database Syst Rev
18. Bollen CW, Uiterwaal CS, van Vught AJ. Cumulative metaanalysis of high-frequency versus conventional ventilation in premature neonates. Am J Respir Crit Care Med
19. Higgins JP, Thompson SG, Deeks JJ, et al. Measuring inconsistency in meta-analyses. BMJ
20. PEST 4: Operating Manual. MPS Research Unit
. The University of Reading; 2000.
21. Whitehead J. The Design and Analysis of Sequential Clinical Trials
(revised 2nd ed). Chichester: John Wiley & Sons Ltd; 1997.
22. Fergusson D, Glass KC, Hutton B, et al. Randomized controlled trials of aprotinin in cardiac surgery: could clinical equipoise have stopped the bleeding? Clin Trials
23. Chalmers I. The scandalous failure of science to cumulate evidence scientifically. Clin Trials
24. Lau J, Antman EM, Jimenez-Silva J, et al. Cumulative meta-analysis of therapeutic trials for myocardial infarction. N Engl J Med
25. Berkey CS, Mosteller F, Lau J, et al. Uncertainty of the time of first significance in random effects cumulative meta-analysis. Control Clin Trials
26. Whitehead A. A prospectively planned cumulative meta-analysis applied to a series of concurrent clinical trials. Stat Med
27. Pogue JM, Yusuf S. Cumulating evidence from randomized trials: utilizing sequential monitoring boundaries for cumulative meta-analysis. Control Clin Trials
28. Froese AB, Kinsella JP. High-frequency oscillatory ventilation: lessons from the neonatal/pediatric experience. Crit Care Med
29. Stark AR. High-frequency oscillatory ventilation to prevent bronchopulmonary dysplasia—are we there yet? N Engl J Med
Suppose k RCTs are available for a sequential meta-analysis. All RCTs compare the same experimental treatment E with a control treatment C and all have the same dichotomous outcome (event or no event). Results from the ith RCT (I = 1, …, k) can be summarized as follows:
The proportions of events with the experimental and with the control treatment are PEi = SEI/NEI and PCi = SCi/NCi, respectively.
The logarithm of the odds ratio, as a measure for association between treatment and outcome, is defined as
The test statistic Zi is expressed as the difference between the observed number of events with E in the ith RCT (SEi) and the expected number under the null hypothesis of treatment equivalence.
The statistic Vi, the variance of Zi, is defined as
The pooled estimate for the overall θ is equal to
as the estimated log(OR) for the ith RCT and the weighting factor wi = Vi.
An approximate 95% confidence interval for θ can be estimated by
(For further details, see references 1, 20, and 21.)