Size is everything – large amounts of information are needed to overcome random effects in estimating direction and magnitude of treatment effects

Abstract

Variability in patients' response to interventions in pain and other clinical settings is large. Many explanations such as trial methods, environment or culture have been proposed, but this paper sets out to show that the main cause of the variability may be random chance, and that if trials are small their estimate of magnitude of effect may be incorrect, simply because of the random play of chance. This is highly relevant to the questions of ‘How large do trials have to be for statistical accuracy?’ and ‘How large do trials have to be for their results to be clinically valid?’ The true underlying control event rate (CER) and experimental event rate (EER) were determined from single-dose acute pain analgesic trials in over 5000 patients. Trial group size required to obtain statistically significant and clinically relevant (0.95 probability of number-needed-to-treat within ±0.5 of its true value) results were computed using these values. Ten thousand trials using these CER and EER values were simulated using varying group sizes to investigate the variation due to random chance alone. Most common analgesics have EERs in the range 0.4–0.6 and CER of about 0.19. With such efficacy, to have a 90% chance of obtaining a statistically significant result in the correct direction requires group sizes in the range 30–60. For clinical relevance nearly 500 patients are required in each group. Only with an extremely effective drug (EER>0.8) will we be reasonably sure of obtaining a clinically relevant NNT with commonly used group sizes of around 40 patients per treatment arm. The simulated trials showed substantial variation in CER and EER, with the probability of obtaining the correct values improving as group size increased. We contend that much of the variability in control and experimental event rates is due to random chance alone. Single small trials are unlikely to be correct. If we want to be sure of getting correct (clinically relevant) results in clinical trials we must study more patients. Credible estimates of clinical efficacy are only likely to come from large trials or from pooling multiple trials of conventional (small) size.