Commentary

How to Report Instrumental Variable Analyses (Suggestions Welcome)

Instrumental variable (IV) methods are becoming mainstream in comparative effectiveness research,¹ but IV methods are radically different from traditional epidemiologic methods. The goal of IV methods is to eliminate confounding without ever measuring the confounders. This apparent miracle can be achieved only when four conditions are met (see below).^2,3 Here we suggest a checklist for investigators who use IV methods. Like others before,⁴ we hope this step-by-step guide will improve the reporting of IV estimates and increase the transparency of the underlying assumptions.

Our discussion focuses on reports of causal effects of medical interventions and is informed by two papers by Davies et al^5,6 that appear in this issue of EPIDEMIOLOGY: an application that estimates the causal effects of cyclooxygenase-2 (COX-2) selective versus nonselective nonsteroidal anti-inflammatory drugs (NSAIDs), and a literature review of IV papers that describes their use, and perhaps misuse, in epidemiology. We supplement this review with additional information from our own review of IV analyses of observational studies with a relatively well-defined medical intervention. Details of our review can be found in the online supplement (https://links.lww.com/EDE/A664).

IV methods require a variable—the “instrument”—that meets the three so-called instrumental conditions: (1) the instrument is associated with the treatment, (2) the instrument does not affect the outcome except through treatment (also known as the exclusion restriction assumption), and (3) the instrument does not share any causes with the outcome. An example of such an instrument is the randomization indicator in double-blind randomized experiments.² Davies et al⁶ summarize instruments that have been proposed in epidemiologic studies, including the physician preference type they use themselves.⁵ Unfortunately, no variable can be proved to be an instrument in observational studies because only condition (1) can be empirically verified. We outline some steps for reporting IV analyses based on the variables proposed as instruments (Figure) and discuss how these steps have been reported in previous studies (Table). A detailed specification of how IV methods should be implemented is beyond the scope of this commentary.

FIGURE: Flowchart for reporting of IV analyses with a single, non–time-varying, dichotomous instrument.

TABLE: Proportions Reporting According to Suggested Guidelines in a Review of IV Publications Assessing Effects of Medical Interventions (n = 81)

The first step in our reporting flowchart is to empirically verify condition (1). When the association between the proposed instrument and the treatment is weak, the proposed “weak” instrument may amplify biases due to small violations of conditions (2) or (3), producing very biased effect estimates.^7,8 Alternatively, if the proposed instrument is very strong, it may be more likely to violate conditions (2) or (3); in the extreme, a perfect correlation between the proposed instrument and the observational treatment implies the proposed instrument is associated with the same set of (possibly unmeasured) confounders as the treatment.⁹ Most prior studies have evaluated the strength of their proposed instrument via risk differences, odds ratios, or partial R² or F statistics like Davies et al⁵ did and several authors have recommended.^6,8 (In contrast, the use of R² or F statistics as effect measures is subject to many study-specific artifacts and should generally be avoided.^10,11)

Unlike condition (1), conditions (2) and (3) are not empirically verifiable: they are assumptions. Therefore, the next step is to use subject-matter knowledge to build a case for why the proposed instrument may be reasonably assumed to meet each of these two conditions; this is similar to how one would use subject-matter knowledge to justify the no unmeasured confounding assumption of traditional methods. When dichotomizing a polytomous or continuous treatment, condition (2) may hold for the original treatment, but not for its dichotomous version. Investigators can provide some support for condition (3) by showing the proposed instrument is not associated with measured confounders.⁶ Although any associations found with measured covariates could be adjusted for by modeling (see below), associations with measured confounders may raise concerns over associations with unmeasured confounders. Davies et al⁵ provide arguments to support conditions (2) and (3) when physician’s preference is proposed as an instrument; other authors have provided counterarguments.²

In our review, we notice a tendency for conditions (2) and (3) to be treated as a single condition. Although these conditions are statistically similar, it is important to consider them separately in order to incorporate subject-matter knowledge in discussions about their validity and in decisions on adjustment. For example, Davies et al⁵ found some evidence that the proposed instrument is associated with prior gastrointestinal (GI) complications, suggesting that different types of patients are seen at different practices, thus violating condition (3) but not condition (2). They also found an association with gastro-protective drug use; this evidence suggests that physicians who prefer one type of NSAID may also prefer prescribing other medications concomitantly, which would violate condition (2) but not condition (3). On the other hand, if gastro-protective medications were prescribed because the NSAID induced GI symptoms, neither condition is violated and no adjustment for gastro-protective medications would be necessary.

Though conditions (2) and (3) cannot be proven true, it is sometimes possible to find empirical evidence against them, that is, conditions (2) and (3) can be falsified. The next step in the flowchart is to perform these falsification tests, which are based on leveraging prior causal assumptions, assessing inequalities that can detect extreme violations, making use of effect modifiers, or comparing estimates utilizing several potential instruments. Examples of each of these have been presented elsewhere.¹² Only 20% of studies explicitly reported a falsification test; none reported two or more tests. Bear in mind that falsification tests may fail to reject a proposed instrument even when conditions (2) and (3) are violated.

Having an instrument is insufficient to obtain a point estimate for the causal effect. Conditions (1)–(3) only identify bounds,^13,14 that is, lower and upper limits for the effect that are consistent with the data. One more untestable condition, often not given explicitly, is required to obtain a point estimate. Therefore, we suggest investigators estimate such bounds, either for the average treatment effect in the population^13,14 or within subgroups.^15,16 Bounds are typically very wide (even wider if one estimates 95% confidence intervals around them): for example, Davies et al⁵ report bounds that range from a strong protective effect to a strong harmful effect of COX-2–selective NSAIDs. However, these bounds are important because they convey the uncertainty about the causal effect when one combines the data with conditions (1)–(3). That is, the bounds show how much “information” needs to be provided by a fourth condition to fill in the blanks left by the data and the instrument.¹⁷ No studies in our review besides Davies et al⁵ reported bounds.

Whether estimating bounds or effects (see below), one needs to decide the causal effect of interest. Two options are the average treatment effect in the population and the local average treatment effect (LATE) in the subpopulation of “compliers.” For Davies et al,⁵ “compliers” are patients who would be prescribed a selective NSAID had they seen a physician who preferred selective NSAIDs, but would be prescribed a nonselective NSAID had they seen a physician who preferred nonselective NSAIDs. For nondichotomous instruments, LATE-like effects are weighted averages of the effect in multiple subgroups.²

The relative relevance of the average treatment effect and the LATE for understanding etiology and informing policy is debatable.^17,18 Because “compliers” cannot be identified, the LATE is a causal effect in an unknown subset of the population, for example, if one found a beneficial effect of treatment, it is unclear who the “compliers” would be who would have this benefit. Because the average treatment effect and the LATE will generally differ, investigators should be explicit about their reasons for choosing one over the other; only 61% of studies in our review were explicit.

If interest is in the average treatment effect, the untestable fourth condition for point estimation may be either condition (4c) of identical (ie, constant) treatment effect for all individuals in the population—generally impossible for dichotomous outcomes and highly unlikely for others—or the weaker homogeneity condition (4h) of no additive effect modification across levels of the instrument within the treated and the untreated (no effect modification only in the treated would allow valid estimation of the effect in the treated).^2,19 The next step in the flowchart is to assess this homogeneity condition (4h) using subject-matter knowledge. Because assessing effect modification by the instrument conditional on treatment is not straightforward, Hernán and Robins² presented a more natural (but still untestable) sufficient condition in terms of effect modification by confounders: for example, for Davies et al,⁵ condition (4h) would be violated if the causal effect of NSAIDs varies by underlying risk of GI complications, an unmeasured variable that likely informs treatment decisions.

If interest is in the LATE, investigators need to assess a different untestable condition (4m): monotonicity or no “defiers.” For Davies et al,⁵ “defiers” are patients who would be prescribed a nonselective NSAID had they seen a physician who preferred selective NSAIDs, but would be prescribed a selective NSAID had they seen a physician who preferred nonselective NSAIDs.²⁰ Under some study designs monotonicity is plausible,^20,21 but this may not be the case for many IV applications in observational studies.²² Monotonicity is generally implausible for the commonly proposed preference-based instruments, which are dichotomous, because such potential instruments are surrogates for a likely continuous preference variable.²

Although one cannot identify who is a “complier,” under conditions (1)–(3) and (4m) one can estimate the proportion of the study population that is composed of “compliers” when the instrument is dichotomous.²⁰ For example, Davies et al’s⁵ LATE estimate is pertinent for 27% of the study population; the magnitude of the effect, if any, in the other 73% is unknown. Further, one can characterize the “compliers” in terms of their distribution of observed covariates relative to the study population.²³ The next steps in our flowchart are to report the proportion of “compliers” and their relevant characteristics. In our review, 70% of studies with dichotomous instruments reported enough information to estimate the proportion of “compliers,” but only 5% (two studies) stated it explicitly; 10% of studies reported information to understand “complier” characteristics. The proportion of “compliers” ranged from 1% to 62% (median = 11%).

If conditions (1)–(3) and either (4h) or (4m) appear justifiable, it may be appropriate to estimate effect estimates using IV methods. The average treatment effect and LATE can be estimated nonparametrically using the standard^19,20 or other² IV estimators. Alternatively, one can use models that estimate these effects within levels of measured covariates, which means the requisite conditions need to hold conditional on these covariates. The last step in the flowchart calls for a detailed description of the modeling approach; Davies et al⁶ summarize previously used modeling approaches like the common two-stage least squares methods, and structural mean models (used by Davies et al⁵) that rely on condition (4h). As always when adjusting for covariates, one should be aware that adjustment for some covariates (ie, colliders) may introduce new bias.^24–26

Our reporting flowchart does not cover more advanced IV analyses. For example, IV analyses based on structural nested models can incorporate time-varying instruments and treatments.^27,28 Other models can incorporate multiple instruments to help with condition (1), although then the other three conditions must be jointly satisfied by all instruments. As for all causal inference methods, sensitivity analyses can illuminate how robust estimates are to violations of the untestable conditions. Some sensitivity analyses have been suggested for IV methods,^7,29–31 but further development and implementation of such approaches are warranted.

Causal inference relies on transparency of assumptions and on triangulation of results from methods that depend on different sets of assumptions. Because of the wide 95% confidence intervals typical of IV estimates, the value added by using this approach will often be small. More importantly, IV methods present special challenges as relatively minor violations of the four conditions may result in large biases of unpredictable or counterintuitive direction. Further efforts are needed to understand the direction and magnitude of bias when the conditions do not hold perfectly. Until then, IV methods are particularly dangerous, which is why our flowchart conservatively suggests avoiding IV methods when the necessary conditions are unlikely to hold. At the very least, we hope that our flowchart will help investigators report IV analyses in such a way that colleagues can better evaluate the estimates.

ACKNOWLEDGMENTS

We thank Alan Brookhart, James Robins, Sander Greenland, Sonia Hernández-Díaz, Debbie Lawlor, and Dylan Small for helpful comments.