ArticleAn evaluation of homogeneity tests in meta-analyses in pain using simulations of individual patient dataGavaghan, David J.a,*; Moore, Andrew R.b; McQuay, Henry J.bAuthor Information aOxford University Computing Laboratory, Wolfson Building, Parks Road, Oxford OX1 3QD, UK bPain Research, Nuffield Department of Anaesthetics, The Churchill, Oxford Radcliffe Hospital, Oxford OX3 7LJ, UK *Corresponding author. Tel.: +44-1865-283574; fax: +44-1865-273839 E-mail: [email protected] Received 22 January 1999; received in revised form 1 October 1999; accepted 23 November 1999. Pain: April 2000 - Volume 85 - Issue 3 - p 415-424 doi: 10.1016/S0304-3959(99)00302-4 Buy Metrics Abstract In this paper we consider the validity and power of some commonly used statistics for assessing the degree of homogeneity between trials in a meta-analysis. We show, using simulated individual patient data typical of that occurring in randomized controlled trials in pain, that the most commonly used statistics do not give the expected levels of statistical significance (i.e. the proportion of trials giving a significant result is not equal to the proportion expected due to random chance) when used with truly homogeneous data. In addition, all such statistics are shown to have extremely low power to detect true heterogeneity even when that heterogeneity is very large. Since, in most practical situations, failure to detect heterogeneity does not allow us to say with any helpful degree of certainty that the data is truly homogeneous, we advocate the quantitative combination of results only where the trials contained in a meta-analysis can be shown to be clinically homogeneous. We propose as a definition of clinical homogeneity that all trials have (i) fixed and clearly defined inclusion criteria and (ii) fixed and clearly defined outcomes or outcome measures. In pain relief, for example, the first of these would be satisfied by all patients having moderate or severe pain, whilst the second would be satisfied by using at least 50% pain relief as the successful outcome measure. © 2000 Lippincott Williams & Wilkins, Inc.