Secondary Logo

Journal Logo


Simulation Comparison of the Quality Effects and Random Effects Methods of Meta-analysis

Doi, Suhail A. R.; Barendregt, Jan J.; Khan, Shahjahan; Thalib, Lukman; Williams, Gail M.

Author Information
doi: 10.1097/EDE.0000000000000289

To the Editor:

The problem with the conventional random effects model of meta-analysis is that as heterogeneity increases, the coverage probability of the model derived confidence interval (CI) drops well below the nominal level, substantially underestimates the statistical error, produces overconfident conclusions and yet has a greater width than the fixed effect model derived CI.1,2 In addition, the random effects point estimate may even be less conservative than that derived from the fixed effect model estimate3 because the former (random effects) model modifies the study inverse variance weights in a way that is unjustified (tending to only equalize the weights).4

As a remedy, we introduced the quality effects estimator in 2008 (modified in 2009) that made use of quality ranks as added input.5,6 The quality effects estimator has undergone two important updates since 2009. First, overdispersion (true variance of the estimator underestimated through the model) has been corrected using an intra-class correlation-based multiplicative scale parameter. Second, the quality ranking is now made scale independent by a further modification, making it relative to the best study. These updates are described in the Online Appendix in the Supplementary material (

We now evaluate the quality effects and random effects estimators using a series of simulations considering the ln(OR) as the effect size. To remove suspicion that the quality input is extremely or arguably subjective, quality effects estimator values are obtained after three different quality inputs within the simulation study: uncertain input (the standard model that mimics the usual variability in quality assessment), random input, and no input (IVhet model in our software). Detailed information regarding the codes (,, and the simulation protocol ( is in the Supplementary material. Performance measures7 are reported in Table for OR = 0.7, though the results from the simulations of the remaining ORs (0.5 to 2.3 in increments of 0.2) were in agreement and are reported in the Supplementary material (

Simulation Results for θ = ln(0.7)

The first quality effects estimator (the standard model simulating quality rank uncertainty through a beta distribution) had a lower mean squared error (MSE) than the random effects estimator. Though there was more bias compared with the random effects estimator, this was completely compensated by the drop in estimator variance due to weighting. The CI width and coverage probabilities were also correct (at the nominal level) when computed using the quality effects model, whereas the CI width based on the random effects model was too narrow, and thus prone to falsely significant results.

The second quality effects estimator (no quality input; IVhet model in MetaXL) was more biased than the standard quality effects estimator, yet it outperformed the random effects estimator and CI coverage was again maintained at the nominal level of 95%. However, without quality input, bias increases much more with increasing magnitude of the effect size or increasing heterogeneity. The number of studies in a meta-analysis has no effect on estimator bias but it does decrease estimator variance. Thus, when there is a very large number of studies showing extreme effects, the bias in this model (IVhet) may exceed the variance and this estimator (without quality input) can then have a MSE exceeding that of the random effects estimator by up to the magnitude of the square of the quality effects estimator bias. Nevertheless, by this point, the random effects estimator has defaulted to the arithmetic mean and we no longer need a weighted estimator to reduce variance, as this reduction is achieved by the large number of studies. Contrary to Shuster,8 this is the only situation where an unbiased (unweighted) estimator performs better than a biased (weighted) estimator.

The third quality effects estimator (quality rank randomly input into the quality effects model without regard to the actual simulated quality) had an identical MSE to the random effects estimator. Despite the performance estimates equalizing, the coverage probability of the CI under the quality effects model was much closer to the nominal level than that under the random effects model, even given the constraint of random quality ranks.

We conclude that the quality effects method of meta-analysis is a clear improvement over the random effects method for handling heterogeneity with or without (IVhet model in MetaXL) quality input as both outperform the random effects estimator and avoid the pitfall of spurious significance. To facilitate the computations under competing methods, our software, MetaXL (available for free download at, has been updated to version 2.0. This package runs both the quality effects and its IVhet variant as well as other conventional methods for comparison. Although this research letter reports simulation results for binary outcomes, conclusions are equally valid for continuous outcomes.

Suhail A. R. Doi

Research School of Population Health

Australian National University

Canberra, Australia

[email protected]

Jan J. Barendregt

Department of Epidemiology and Biostatistics

School of Population Health

University of Queensland

Brisbane, Australia

Shahjahan Khan

School of Agricultural, Computational and Environmental Sciences

University of Southern Queensland

Toowoomba, Australia

Lukman Thalib

Department of Community Medicine

Kuwait University


Gail M. Williams

Department of Epidemiology and Biostatistics

School of Population Health

University of Queensland

Brisbane, Australia


1. Brockwell SE, Gordon IR.. A comparison of statistical methods for meta-analysis. Stat Med. 2001;20:825–840
2. Noma H.. Confidence intervals for a random-effects meta-analysis based on Bartlett-type corrections. Stat Med. 2011;30:3304–3312
3. Poole C, Greenland S.. Random-effects meta-analyses are not always conservative. Am J Epidemiol. 1999;150:469–475
4. Overton RC.. A comparison of fixed-effects and mixed (random-effects) models for meta-analysis tests of moderator variable effects. Psychological Methods. 1998;3:354–379
5. Doi SA, Thalib L.. A quality-effects model for meta-analysis. Epidemiology. 2008;19:94–100
6. Doi SA, Thalib L.. An alternative quality adjustor for the quality effects model for meta-analysis. Epidemiology. 2009;20:314
7. Burton A, Altman DG, Royston P, Holder RL.. The design of simulation studies in medical statistics. Stat Med. 2006;25:4279–4292
8. Shuster JJ.. Empirical vs natural weighting in random effects meta-analysis. Stat Med. 2010;29:1259–1265

Supplemental Digital Content

Copyright © 2015 Wolters Kluwer Health, Inc. All rights reserved.