Postlaunch data on medical treatments can be analyzed to explore adverse events or relative effectiveness in real-life settings. These analyses are often complicated by the number of potential confounders and the possibility of model misspecification.
We conducted a simulation study to compare the performance of logistic regression, propensity score, disease risk score, and stabilized inverse probability weighting methods to adjust for confounding. Model misspecification was induced in the independent derivation dataset. We evaluated performance using relative bias confidence interval coverage of the true effect, among other metrics.
At low events per coefficient (1.0 and 0.5), the logistic regression estimates had a large relative bias (greater than −100%). Bias of the disease risk score estimates was at most 13.48% and 18.83%. For the propensity score model, this was 8.74% and >100%, respectively. At events per coefficient of 1.0 and 0.5, inverse probability weighting frequently failed or reduced to a crude regression, resulting in biases of −8.49% and 24.55%. Coverage of logistic regression estimates became less than the nominal level at events per coefficient ≤5. For the disease risk score, inverse probability weighting, and propensity score, coverage became less than nominal at events per coefficient ≤2.5, ≤1.0, and ≤1.0, respectively. Bias of misspecified disease risk score models was 16.55%.
In settings with low events/exposed subjects per coefficient, disease risk score methods can be useful alternatives to logistic regression models, especially when propensity score models cannot be used. Despite better performance of disease risk score methods than logistic regression and propensity score models in small events per coefficient settings, bias, and coverage still deviated from nominal.
Supplemental Digital Content is available in the text.
From the aJulius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht, The Netherlands; bDivision of Pharmacoepidemiology and Clinical Pharmacology, Utrecht Institute for Pharmaceutical Sciences, Utrecht, The Netherlands; cDepartment of Farm Animal Health, Faculty of Veterinary Medicine, Utrecht University, Utrecht, The Netherlands; and dInstitute of Cardiovascular Science, Faculty of Population Health, University College London, London, United Kingdom.
Submitted 28 November 2014; accepted 25 August 2015.
Supported by the Innovative Medicines Initiative Joint Undertaking under Grant Agreement No. 115546, resources of which are composed of financial contribution from the European Union’s Seventh Framework Programme (FP7/2007‐2013) and EFPIA companies’ in kind contribution.
A.F.S., R.H.H.G., and O.H.K. contributed to the idea and design of the study. A.F.S. performed the analyses and drafted the manuscript. O.H.K. and R.H.H.G. provided guidance during initial planning of the paper and during critical revision. A.F.S. had full access to all of the data and takes responsibility for the integrity of the data presented.
The research leading to these results was conducted as part of the GetReal consortium. For further information, please refer to www.imi-getreal.eu.
The authors report no conflicts of interest.
Supplemental digital content is available through direct URL citations in the HTML and PDF versions of this article (www.epidem.com).
Correspondence: Amand F. Schmidt, Institute of Cardiovascular Science, Faculty of Population Health, University College London, London, United Kingdom. E-mail: firstname.lastname@example.org.