With the publication of “Repeal of Comprehensive Background Check Policies and Firearm Homicide and Suicide,”1 the synthetic control method joins a group of useful statistical methods for the analysis of observational data imported to the field of epidemiology from economics and political science. These approaches include fixed effects,2 difference in differences,2 regression discontinuity,3 and instrumental variables.4 This expansion of methodological tools has been critical for the development of epidemiology as the discipline has expanded to consider the analysis of exposures that are not easily subjected to true random assignment, and for which unmeasured confounders plausibly exist. Furthermore, for social epidemiology, these methods help to facilitate a consequential approach,5,6 as the subfield moves to evaluating the impacts of social and economic policies on health. The importance of considering the synthetic control method for epidemiologic questions is that when randomization is difficult or impossible, and there are no available instrumental variables, the tools available for analysis are currently limited. This is particularly true when only 1 or a small number of units are treated.
The synthetic control method is most closely related to a difference in differences approach. The difference in differences comparison is based on defining a treated unit (e.g., the population of a state that is exposed to a policy) and a control unit (e.g., a single state or multiple states), and comparing pre/post change between these units. The novel contribution of this method is that, rather than aggregating all comparison groups into 1 control unit, potential control groups (e.g., the “donor pool” of unaffected states) are considered separately, and a weighted combination of states that are the best pretreatment match with the treatment unit is used as the controls. Inference is then based on a posttreatment difference between the treatment and the weighted combination of control states. Thus, while difference in differences requires a parallel trend assumption for the pre/post comparison, the synthetic control method relies on there being a close pretreatment match with the donor pool. The “synthetic” term refers to the fact that the control group is a weighted average of the available control states, such that typically some states (because they diverge too much from the treatment state in their pretreatment characteristics) are not used at all, and other states contribute different amounts to the synthetic control population. For example, as shown in Table 2 of Kagawa et al,1 when analyzing firearm homicide in Indiana, the synthetic control is a combination of Illinois (weighted as 56% of the synthetic control group) and Iowa (the remaining proportion).
Epidemiologists, perhaps more than most disciplines given the field’s development of case-control methods, are acutely aware of the importance of appropriately selecting controls, and this is a critical decision to be made with the synthetic control method. The user selects the pool of eligible control regions. Kagawa et al1 appropriately select only eligible states, and because the treatment is repeal of a policy, they are limited to only the 8 other states that currently have the policy. This approach is most consistent with the selection of the control population in a case-control study with respect to the outcome of interest, selecting randomly from among those states at risk for the outcome.7 Recent synthetic control method applications used as few as 168 but as many as 1309 regions in the donor pool. In addition, based on the matching shown in Table 2 of Kagawa et al,1 half of the 8 states are not very good matches, with virtually all of the control states across outcomes made up of Illinois, Iowa, Michigan, and North Carolina. The primary advantage in the synthetic control is taking a data-driven approach to getting a close pretreatment match between treatment and control regions, but this advantage is more challenging to leverage when the sample of donor pool states is small. In scenarios where only a small number of regions match, robustness checks with the synthetic control method have been performed by excluding the best matches and showing that results were consistent with the second best selection of control regions.10
The importance of the donor pool is magnified with the synthetic control method because inference about whether the results are due to random error also depends on the choice of the donor pool. This is perhaps the most novel and challenging aspect of the synthetic control method; there is not a validated, parametric approach to establishing confidence intervals on the matched posttreatment difference between the treatment and synthetic control as there is with difference in differences. Kagawa et al1 take the generally recommended approach of a placebo test where each of the control states is used to establish the range of the posttreatment level of the outcome that could be expected in the absence of treatment, since the control states did not receive the intervention. While there is a rough analogy here to simulated confidence intervals from approaches such as the bootstrap, the additional challenge for synthetic control method is that the standard assumptions for estimating an effect parameter are also required assumptions for valid inference on whether the effect parameter estimated is within the range of natural variation observed in the placebo controls. That is, the following assumptions must be met for valid inference about the precision of the estimates: (1) stable unit treatment value assumption; (2) assumption of random assignment to the treatment; and (3) assumption that the potential outcomes for regions are fixed but a priori unknown.11 Imagine, for example, that New York and Massachusetts are regions that differ substantially from Indiana and Tennessee (which is consistent with the fact that they don’t serve as best matched control regions). These regions are then used to establish the natural variation that would be expected for Indiana and Tennessee if no intervention had occurred. The conclusion that there is no impact of the repeal of comprehensive background checks is dependent on the assumption that Indiana and Tennessee are randomly selected for treatment from the sample of control states. This is a particular challenge when study conclusions are that there was no impact of a policy change. We are thus concerned about the potential power for Kagawa et al1 to detect an effect, particularly on a rare outcome, using the synthetic control method approach. This limitation may be able to be addressed in the future by modifications to the method that produce confidence intervals.11–13 A further recommended approach is to assess placebo treatments based on time.8
In addition to selecting the potential control units, another set of critical decisions involves what variables should be used to match units, and an evaluation of how close the synthetic match is. Since pretreatment levels of the outcome are almost always the strongest predictor of posttreatment levels of the outcome, critical decisions revolve around how to include these in the model. Kagawa et al1 fit several alternatives with respect to pretreatment outcome levels, deciding on using a single year measure of the outcome directly prior to treatment based on matching fit, an important consideration to avoid false-positive findings.14 However, the difficulty of finding a good match across all covariates is in part driven by the fact that there are very few matching units available, so in several cases the best matched synthetic controls are ones that differ substantially in their pretreatment levels of the outcome. For example, the rate of firearm suicide pretreatment for the synthetic control group for Tennessee is 39% lower in 1977 than the pretreatment level in actual Tennessee (shown in eTable 3). This creates a situation where firearm suicide is likely to appear higher in Tennessee after the treatment. The difficulty in pretreatment matching is made more difficult by the instability in the outcome and the sharply declining trend in the firearm homicide rate just prior to treatment (Kagawa Figure 1A). Prior applications of the synthetic control method have shown less volatility in the outcome and a generally higher pretreatment match,8–10 for example, a 1% difference in the primary models and a range of 0–3% differences in alternative specifications.10
Abadie et al,8 who developed the method, present the synthetic control method as a bridge between qualitative and quantitative research in that it provides more rigor for the comparison groups chosen for comparative case studies. This assessment emphasizes that when there are only 1 or 2 treated units and a small number of control units, the power for inference will always be limited. Without robust confidence intervals and a clear understanding of statistical power, the estimates from the method should thus be considered similar to estimates from a case study when considering the strength of evidence from this approach. We do not use the term “case study” pejoratively here, but rather to frame how we consider the contribution that estimates from synthetic control method contribute to causal questions. The synthetic control method provides a useful tool for epidemiology when there are only a few treated units. It is a tool that, as demonstrated by Kagawa et al,1 increases the transparency of how closely a control unit matches pretreatment, a worthwhile advance over analyses using single unit or aggregated unit control groups.
1. Kagawa RMC, Castillo-Carniglia A, Vernick JS, et alRepeal of comprehensive background check policies and firearm homicide and suicide. Epidemiology. 2018;29:494502.
2. Strumpf EC, Harper S, Kaufman JSFixed Effects and Difference in Differences. Methods in Social Epidemiology. 2017.San Francisco CA: Jossey-Bass;
3. Bor J, Moscoe E, Mutevedzi P, Newell ML, Bärnighausen TRegression discontinuity designs in epidemiology: causal inference without randomized trials. Epidemiology. 2014;25:729–737.
4. Greenland SAn introduction to instrumental variables for epidemiologists. Int J Epidemiol. 2000;29:722–729.
5. Galea SAn argument for a consequentialist epidemiology. Am J Epidemiol. 2013:kwt172.
6. Nandi A, Harper SHow consequential is social epidemiology? A review of recent evidence. Curr Epidemiol Rep. 2014;2:61–70.
7. Robins JM, Gail MH, Lubin JHMore on “Biased selection of controls for case-control analyses of cohort studies.” Biometrics. 1986;42:293–299.
8. Abadie A, Diamond A, Hainmueller JComparative politics and the synthetic control method. Am J Political Sci. 2015;59:495–510.
9. Gobillon L, Magnac TRegional policy evaluation: interactive fixed effects and synthetic controls. Rev Econ Stat. 2016;98:535–551.
10. Pinotti PThe economic costs of organised crime: evidence from Southern Italy. Econ J. 2015;125:F203–F232.
11. Firpo S, Possebom VSynthetic control method: inference, sensitivity analysis and confidence sets. 2017.
12. Xu YGeneralized synthetic control method: causal inference with interactive fixed effects models. Political Anal. 2017;25:57–76.
13. Athey S, Bayati M, Doudchenko N, Imbens G, Khosravi KMatrix completion methods for causal panel data models. arXiv preprint arXiv:2017.171010251.
14. Ferman B, Pinto C, Possebom VCherry Picking with Synthetic Controls. March 2018. Germany MPRA Paper 85138, University Library of Munich, https://mpra.ub.uni-muenchen.de/85138/