Comparing Two-way Fixed Effects and New Estimators for Difference-in-Differences: A Simulation Study and Empirical Example

Abstract

Background: 

Two-way fixed effects methods have been used to estimate effects of policies adopted in different places over time, but they can provide misleading results when effects are heterogeneous or dynamic, and alternate methods have been proposed.

Methods: 

We compared methods for estimating the average treatment effect on the treated (ATT) under staggered adoption of policies, including two-way fixed effects, group-time ATT, cohort ATT, and target-trial approaches. We applied each method to assess the impact of Medicaid expansion on preterm birth using the National Center for Health Statistics’ birth records. We compared each estimator’s performance in a simulation parameterized to mimic the empirical example. We generated constant, heterogeneous, and dynamic effects and calculated bias, mean squared error, and confidence interval coverage of each estimator across 1000 iterations.

Results: 

Two-way fixed effects estimated that Medicaid expansion increased the risk of preterm birth (risk difference [RD], 0.12; 95% CI = 0.02, 0.22), while the group-time ATT, cohort ATT, and target-trial approaches estimated protective or null effects (group-time RD, −0.16; 95% CI = −0.58, 0.26; cohort RD, −0.02; 95% CI = −0.46, 0.41; target trial RD, −0.16; 95% CI = −0.59, 0.26). In simulations, two-way fixed effects performed well when treatment effects were constant and less well under heterogeneous and dynamic effects.

Conclusions: 

We demonstrated why new approaches perform better than two-way fixed effects when treatment effects are heterogeneous or dynamic under a staggered policy adoption design, and created simulation and analysis code to promote understanding and wider use of these methods in the epidemiologic literature.