Secondary Logo

Journal Logo

Methods

Biases in Randomized Trials

A Conversation Between Trialists and Epidemiologists

Mansournia, Mohammad Ali; Higgins, Julian P. T.; Sterne, Jonathan A. C.; Hernán, Miguel A.

Author Information
doi: 10.1097/EDE.0000000000000564

Abstract

Erratum

Figure 4c in our recent Epidemiology paper1 is a repetition of Figure 4b and is incorrect. As explained in the text, Z is a common cause of A and C in the corrected Figure 4c; the correct Figure appears here.

Figure.
5+ images

Epidemiology. 29(5):e49, September 2018.

Randomized controlled trials (RCTs) and observational studies are used to assess the causal effects of medical interventions.1 By definition, treatment strategies are randomly assigned in RCTs but not in observational studies. Randomization, which prevents bias due to noncomparability between groups, is exploited in full when the data analysis follows the “intention-to-treat” principle.

Another difference between some RCTs and observational studies is masking (blinding) of trial participants and personnel, which can be achieved by using a placebo that is indistinguishable from the active treatment. Masking prevents differential care during follow-up, accounts for nonspecific effects associated with receiving an intervention (placebo effects), may facilitate blinding of outcome assessors, and may improve adherence.

Widespread use of masking and of intention-to-treat analyses became established by regulatory requirements, which privileged intention-to-treat analyses of double-blind placebo-controlled RCTs to assess the efficacy of drugs before licensing. However, masking is sometimes not feasible (e.g., in surgical trials), and may not even be desirable (e.g., in pragmatic trials whose goal is estimating effects in real-world conditions). An intention-to-treat analysis is not feasible if trial participants are lost to follow-up and has disadvantages in safety and noninferiority trials.2

Discussions about the differences between RCTs and observational studies can be complicated by the different terminology employed by trialists and epidemiologists.3 Trialists often use the taxonomy of bias typified by the Cochrane tool for assessing risk of bias in RCTs: selection bias, performance bias, detection bias, attrition bias, reporting bias, and other bias.4,5 Epidemiologists, on the other hand, tend to use the categories confounding, selection bias, and measurement (or information) bias.1,6,7

Causal diagrams have been used extensively to represent biases in epidemiologic studies.8–14 These diagrams, represented as directed acyclic graphs, comprise variables (nodes) and arrows (directed edges). The absence of an arrow pointing from variable A to variable B indicates that variable A does not have a direct causal effect on B. A key advantage of causal diagrams is that they provide a mathematically rigorous yet intuitive tool for deducing the statistical independencies implied by the lack of causal arrows.1,8,9

Here, we use causal diagrams to represent the biases described in the Cochrane Risk of Bias Tool, and provide a translation to the epidemiologic terms of confounding, selection bias, and measurement bias. For simplicity, we focus on individually randomized (not cluster randomized), parallel group (not crossover) trials that compare two time-fixed treatment strategies. We start by reviewing the main types of causal effect that are of interest in RCTs.

INTENTION-TO-TREAT EFFECT AND PER-PROTOCOL EFFECT

The intention-to-treat effect is the effect of treatment assignment (or allocation).15 Consider an RCT in which HIV-positive individuals are assigned to either initiating a new treatment Z = 1 or to continuing on their existing treatment Z = 0, and are followed until death or the end of follow-up at 5 years. The outcome of interest is 5-year mortality Y (1: yes, 0: no). The intention-to-treat effect is unbiasedly estimated by an intention-to-treat analysis that compares the mean outcome between those assigned to Z = 1 and Z = 0. For example, the intention-to-treat effect on the causal risk difference scale is unbiasedly estimated by the difference of the risks in the groups Z = 1 and Z = 0, which are readily computed from the study data.

The magnitude of the intention-to-treat effect in a particular study depends on the magnitude and type of adherence to the assigned treatment strategies. To see this, consider two RCTs with identical eligibility criteria and that compare the same two strategies. In the first RCT, only half of the patients assigned to the new treatment (Z = 1) end up actually taking it (A = 1); the other half do not take it (A = 0). In the second RCT, all patients assigned to treatment take it (i.e., patients with Z = 1 also have A = 1). In both studies, all patients assigned to Z = 0 cannot take the new treatment because it is not available outside the study (i.e., patients with Z = 0 also have A = 0). Even if the effect of the new treatment is identical in both studies, the intention-to-treat effect will generally differ between the two studies. For example, the intention-to-treat effect will be closer to the null in the first RCT than in the second one if the effect of treatment goes in the same direction (beneficial or harmful) for all patients, and more beneficial in the first RCT than in the second one if adherers tend to be those for whom treatment has a beneficial effect and non adherers tend to be those for whom treatment has a harmful effect.1 If the above RCTs were head-to-head trials that assigned participants to two active treatments, then the intention-to-treat effect in the first RCT might also be either closer or further from the null than that in the second RCT.2

An alternative to the intention-to-treat effect that is not affected by the study-specific adherence to treatment is the per-protocol effect, that is, the causal effect that would have been observed if all patients had adhered to the protocol of the RCT. Unfortunately, valid estimation of the per-protocol effect in the presence of imperfect adherence generally requires untestable assumptions.16

Two common approaches to estimate the per-protocol effect are (i) comparing the outcomes of those who took treatment A = 1 and treatment A = 0 (regardless of the treatment they were assigned to), for example, Pr[Y = 1|A = 1] − Pr[Y = 1|A = 0], and (ii) comparing the outcomes of those who took treatment A = 1 among those assigned to Z = 1 and treatment A = 0 among those assigned to Z = 0, for example, Pr[Y = 1|A = 1, Z = 1] − Pr[Y = 1|A = 0, Z = 0].) Approach (i) is often referred to as an “as-treated” analysis and approach (ii) as a “per-protocol” analysis.2,8 Neither approach is generally valid to estimate the per-protocol effect, as we discuss below. (G-estimation and instrumental variable methods can sometimes be used to estimate some form of per-protocol effects even in the presence of unmeasured confounders in Figure 1C, D.16,17)

F1-9
FIGURE 1:
Cochrane selection bias. A, Epidemiologic confounding in an intention-to-treat analysis. B, Epidemiologic selection bias in an intention-to-treat analysis. C, Epidemiologic selection bias in a per-protocol analysis. D, Epidemiologic confounding in an as-treated analysis.

Although “as-treated” and “per-protocol” analyses are potentially biased, the per-protocol effect may be of greater interest to patients and their clinicians than the intention-to-treat effect. We now discuss how the potential for bias in effects estimated from RCTs depends on whether the goal is to estimate the per-protocol or the intention-to-treat effect.

COCHRANE BIAS DOMAINS AND CAUSAL DIAGRAMS

The Cochrane Risk of Bias Tool for randomized trials covers six domains of bias.4,5 In the next sections, we use causal diagrams to show the structure of most of these biases, and discuss their correspondence to the epidemiologic terms of confounding, selection bias, and measurement bias. Because all these biases can occur under the null, we draw the causal diagrams under the causal null hypothesis, unless otherwise specified. (Any causal structure that results in bias under the null will also cause bias under the alternative that treatment has an effect on the outcome, but the converse is not true.) For each bias, we explain whether it affects the intention-to-treat effect or the per-protocol effect. Our definition of bias is the same as in Chapter 10 of Reference [1] under either a randomization model or a correct population model.18,19

SELECTION BIAS

In its risk of bias tool, Cochrane defines selection bias as the result of “systematic differences between baseline characteristics of the groups that are compared.”4 The presence of “systematic differences between baseline characteristics” means that the distribution of prognostic factors varies between the groups being compared. The bias may affect the estimate of the intention-to-treat effect and/or the estimate of the per-protocol effect, depending on the definition of “groups that are compared.”

Let us first consider the case in which the “groups that are compared” are the randomized groups Z = 1 and Z = 0. There are at least three reasons why differences in the distribution of risk factors may arise.

(i) The assignment of patients to a group is influenced by knowledge of which treatment they will receive.

This bias can occur if the assignment that was not properly randomized or the randomized assignment was not sufficiently concealed, and so the person enrolling participants was aware of allocation sequence and influenced which patients were assigned to each group based on their prognostic factors. This situation is depicted by the causal diagram in Figure 1A that includes the prognostic factors L (e.g., CD4 count, viral load) as common causes of the outcome Y and the assignment Z. The arrow from L to Z may be due to the improperly randomized or insufficiently concealed allocation sequence. There are other causal diagrams that represent common causes of Z and Y (see, for example, Chapter 7 of Reference [1]); we chose the simplest.

Epidemiologists refer to biases that arise from the presence of common causes as confounding. The existence of common causes L of assignment Z and outcome Y introduces confounding bias for the intention-to-treat effect in an intention-to-treat analysis that compares individuals in groups Z = 1 and Z = 0, and for the per-protocol effect in a per-protocol analysis that compares individuals in groups Z = 1 and Z = 0 with A = Z. In both cases, the Cochrane Risk of Bias Tool refers to this bias as selection bias (Table).

T1-9
TABLE:
Translation of Cochrane Bias in Randomized Trials Domains into Common Epidemiologic Terms

Appropriate randomization, generation, and concealment of the allocation sequence, or adjustment for the prognostic factors L removes the LZ arrow and therefore the confounding bias.

Even under perfect randomization procedures, random imbalances in prognostic factors may bias intention-to-treat effect estimates. This so-called chance confounding20 (sometimes referred to as allocation bias19 or accidental bias18,21,22) is quantitatively addressed by frequentist confidence intervals and is mitigated by adjusting for measured prognostic factors that are imbalanced.1,10 Unlike the structural confounding depicted in Figure 1A, chance confounding is expected to become smaller as sample size increases.

(ii) The decision to recruit a patient is influenced by knowledge of which treatment the patient will receive.19,21

This bias can occur if an investigator is aware of the random sequence and decides to enroll patients with certain prognostic factors only if they are known to be assigned to a particular treatment strategy. The Cochrane Risk of Bias Tool describes this problem: “Knowledge of the next assignment […] can cause selective enrolment of participants on the basis of prognostic factors. Participants who would have been assigned to an intervention deemed to be “inappropriate” may be rejected.”4

The causal diagram in Figure 1B represents this scenario. The node S is the selection into the trial (1: yes, 0: no), which depends on the values of assignment Z and prognostic factors L. The box around S indicates that the analysis is restricted to those with S = 1. This bias arises from the selection of a subset of the potential study population into the analysis and, because S is a common effect of assignment and prognostic factors, both intention-to-treat and per-protocol analyses may be biased even if both effects are truly null. Epidemiologists10 and Cochrane refer to this bias as selection bias (Table).

The elimination of this selection bias requires removing the ZS arrow through appropriate concealment of the allocation sequence, or adjustment for the prognostic factors L.

(iii) The decision to adhere to the assigned treatment is influenced by prognostic factors.

This may result in an imbalance between the groups A = 1 and A = 0, but not between the groups Z = 1 and Z = 0. Therefore, this imbalance will not bias the intention-to-treat estimate, but will generally bias the per-protocol estimate of a naïve per-protocol analysis. This third case is not addressed by the Cochrane Risk of Bias Tool.

The causal diagram in Figure 1C represents this scenario. The node UA stands for common causes (e.g., symptoms resulting from severe immunosuppression) of adherence to treatment A and outcome Y.2 The node S is the variable selection into the per-protocol population (1: yes, 0: no), which depends on the values of Z and A (S = 1 if Z = A, S = 0 if ZA), and the analysis is restricted to those with S = 1.

Epidemiologists may refer to this bias as selection bias because it arises from the selection of a subset (the per-protocol population) of the study population into an analysis that compares Z = 1 versus Z = 0. However, note that this selection bias for the per-protocol effect only arises when there is confounding of the effect of A due to common causes UA of A and Y.

Regardless of whether we refer to this bias as confounding or selection bias, reducing it requires either a masked design or a non-naïve, more realistic per-protocol analysis that adjusts for the variables UA or their proxies. Because a per-protocol analysis compares groups not entirely defined by randomization, the analysis is subject to the biases usually associated with observational studies.

Finally, let us consider the case in which the “groups that are compared” are the non randomized groups A = 1 and A = 0, that is, an as-treated analysis. Because as-treated analyses are effectively observational analyses, their estimates of the per-protocol effect will be confounded in the presence of common causes of treatment A and the outcome Y. The structure of the bias is shown by the causal diagram in Figure 1D. In subsequent diagrams, we will omit the common causes of A and Y to avoid clutter and to focus the attention on the other sources of bias.

PERFORMANCE BIAS

The Cochrane Risk of Bias Tool defines performance bias as the result of “systematic differences between groups in the care that is provided, or in exposure to factors other than the interventions of interest.”3 These differences may occur due to knowledge of the assigned treatment Z by study participants and thus will be less likely in masked trials.

Again let us first consider the case in which the “groups that are compared” are the randomized groups Z = 1 and Z = 0. Consider the causal diagram in Figure 2A, where O represents medical interventions that are forbidden by the study protocol (e.g., an intensive monitoring and treatment of cardiovascular risk factors) and UO represents unmeasured common causes of O and Y (e.g., risk factors for cardiovascular disease). The arrow from Z to O indicates that awareness of the assigned treatment might lead to changes in the behavior of study participants or their doctors, which in turn may affect the outcome, hence an arrow from O to Y. The interventions O are a result of assignment Z itself, and therefore just mediators of the effect of Z.

F2-9
FIGURE 2:
Cochrane performance bias. A, Not a bias in intention-to-treat analyses. B, Biased direct effect in a per-protocol or as-treated analysis. C, Epidemiologic confounding a in per-protocol or as-treated analysis.

Because the intention-to-treat effect is the effect of assignment and part of the effect of assignment is mediated through O, then O cannot be viewed as a source of bias. The intention-to-treat effect naturally incorporates the effects of deviations from protocol, including the interventions O. That is, in an intention-to-treat analysis whose goal is to estimate the intention-to-treat effect, performance bias cannot occur. In epidemiologic terms, there is no confounding or selection bias.

However, the Cochrane literature appears to suggest that performance bias can occur even in intention-to-treat analyses. To explore this issue, consider two types of departures from intended interventions

(i) Departures from intended interventions that might happen in real life.

When trial participants receive interventions O that are prohibited by the protocol but that they would have also received outside of the trial, we believe that most people would agree with the conclusion that no performance bias exists in intention-to-treat analyses.

(ii) Departures from intended interventions that arise only because of the randomized trial context.

When trial participants receive interventions O that are prohibited by the protocol and that they would have not received outside of the trial, the intention-to-treat effect estimated from the trial is relatively unhelpful for patients outside the trial. This may be a reason that Cochrane uses the performance bias label for randomized trials. However, the use of the word “bias” in this context needs to be carefully qualified.

In the absence of any of the other biases discussed here, an intention-to-treat analysis of a trial in which interventions O occur is an unbiased estimator of the intention-to-treat effect in that particular trial context and population. The presence of interventions O, like other trial-specific characteristics (e.g., eligibility criteria, monitoring) do not affect the estimates’ internal validity but may affect their external validity (e.g., if the estimate cannot be transported to clinical contexts outside of the trial in which the interventions O are less frequent). In this case, it might be more appropriate to say that the intention-to-treat effect from the trial is not generalizable or transportable to other settings rather than saying that it is “biased.”

Another reason that Cochrane uses the performance bias label for intention-to-treat analysis of randomized trials is that the implicit research question may not be about the pure intention-to-treat effect, but rather an intention-to-treat effect where the only deviation in protocol is non-adherence to the assigned treatment, that is, the research question is neither intention-to-treat effect nor per-protocol effect. Thus, different research questions might lead to different categorizations of bias.

Performance bias may occur when estimating the per-protocol effect via either a per-protocol or an as-treated analysis. There are at least two distinct reasons for the bias to arise.

Figure 2B depicts a setting in which deviations from protocol O are affected by the received treatment (e.g., because the use of certain therapies prompts doctors conduct tests for cardiovascular risk factors that were forbidden by the protocol). The per-protocol effect is then the direct effect of treatment in the absence of those deviations from protocol O (e.g., if no tests for cardiovascular factors had been conducted). Unfortunately, per-protocol and as-treated analyses will yield a biased direct effect (per-protocol) estimate whether they do or do not adjust for O (Table). Lack of adjustment will result in bias because the effect estimate will include the indirect effect as well; adjustment for O but not also for all confounders of the effect of O on the outcome (e.g., UO),11 will generally result in selection bias (in graph-theoretic terms, O is a collider).

Performance bias for the per-protocol effect may also have the same structure as confounding. Figure 2C represents a setting in which O operates as a confounder of the effect of A on Y. The arrow from O to A indicates that interventions not specified in the protocol (e.g., intensive monitoring of cardiovascular risk factors) may alter treatment received during the follow-up (e.g., the presence of cardiovascular risk factors leads doctors to prescribe a different type of antiretroviral therapy). With time-varying variables A and O, the structures represented in Figure 2B, C can occur simultaneously.

DETECTION BIAS

The Cochrane Risk of Bias Tool defines detection bias as the result of “systematic differences between groups in how outcomes are determined.”4 This bias (also called observer, ascertainment, or assessment bias) occurs if knowledge of a patient’s assigned strategy influences outcome assessment. Figure 3A represents detection bias for the intention-to-treat effect. In this graph, the true outcome Y remains unmeasured and Y* represents the mismeasured outcome. The arrows from Z and Y to Y* represent that outcome measurement depends on both the true outcome Y and the treatment assignment Z. An intention-to-treat estimate of the effect of Z on Y* from Figure 3A will be biased for the intention-to-treat effect of Z on Y; the bias is a consequence of mismeasurement of Y, and is commonly referred to as “measurement bias” or “information bias” in epidemiology (Table).1,6,12 The type of measurement error represented in Figure 3A is differential with respect to treatment assignment1,5 and therefore, like all other biases discussed previously, leads to bias even if Z has no effect on Y.

F3-9
FIGURE 4:
Cochrane attrition bias. A, Selection bias due to differential loss to follow-up in an intention-to-treat analysis. B, Selection bias due to differential loss to follow-up in a per-protocol or as-treated analysis. C, Selection bias due to differential loss to follow-up in a per-protocol or as-treated analysis.

Detection bias may affect per-protocol effect estimates either directly if A affects Y*, as in Figure 3B, or indirectly, as in Figure 3C, if Z is a common cause of A and Y*. Measurement bias in Figure 3A–C can be avoided by masking of outcome assessors, because it removes the ZY* arrow or the AY* arrow.

ATTRITION BIAS

The Cochrane Risk of Bias Tool defines attrition bias as the result of “systematic differences between groups in withdrawals from a study.”4 The source of bias is differential loss-to-follow-up (e.g., drop out) or other forms of exclusions from the analysis. Figure 4A includes the censoring indicator C, which takes value 1 for individuals excluded from the analysis. The box around C = 0 indicates that the analysis is restricted to those who were not excluded from the analysis. The arrow from Z to C indicates that withdrawal from the analysis is influenced by knowledge of the participant’s group assignment, for example, patients assigned to less potent combination antiretroviral therapy are more likely to not attend future visits if they were aware of their assigned treatment. The arrow from L to C indicates that individuals with worse prognosis (L = 1) are more likely to be excluded than the others (with L = 0), because the severity of their disease prevents them from attending future study visits. In graph-theoretic terms, the intention-to-treat effect estimate is biased because, even under the null, the path ZCLY is open when conditioning on the collider C. This bias is another example of what epidemiologists refer to as selection bias (Table).1,10

F4-9
FIGURE 3:
Cochrane detection bias. A, Measurement bias due to outcome misclassification in an intention-to-treat analysis. B, Measurement bias due to outcome misclassification in a per-protocol or as-treated analysis. C, Measurement bias due to outcome misclassification in a per-protocol or as-treated analysis.

The per-protocol effect estimate is also subject to attrition bias. The bias may arise directly if A affects censoring C (e.g., subjects receiving A = 1 are at a greater risk of experiencing side effects, which could lead them to dropout), as in Figure 4B, or indirectly, as in Figure 4C, if Z is a common cause of A and C. In Figure 4A, C, masking of participants and doctors providing care can prevent attrition bias for intention-to-treat and per-protocol effect estimates by removing the ZC arrow, and thus blocking the biasing paths ZCLY and AZCLY, respectively. Adjustment for L also in Figure 4A–C also adjusts for selection bias.

REPORTING BIAS

The Cochrane Risk of Bias Tool defines reporting bias as the result of “systematic differences between reported and unreported findings.”4 Outcome reporting bias may occur because so-called statistically significant effect estimates are more likely to be reported than nonsignificant effect estimates. Then the average published result will be farther from the null than the true average result, which will bias meta-analyses and systematic reviews.23 A similar bias, “stepwise selection,” results in inflated estimates for weak effects, sometimes known as testimation (estimation after testing) bias.24–26 Reporting bias, which applies to both intention-to-treat and per-protocol effects, is negligible when treatment has a strong effect on the outcome or the trial size is very large.24,25

Epidemiologists have long warned against the problems resulting from abuse of significance testing and selective reporting after multiple comparisons.27,28 Because reporting bias of individual studies is not a structural bias, it cannot be generally represented using the causal diagrams.

DISCUSSION

We described how the terminology used to describe similar biases differs between trialists and epidemiologists, and why an explicit specification of the causal target of each randomized trial is beneficial when discussing the risk of bias. For example, making the intention-to-treat effect the target allows trialists to stop worrying about some forms of “performance bias.” On the other hand, making the per-protocol effect the target makes it clear that adjustment for pre- and post randomization confounding is needed, which has implications for the design and analysis of RCTs.29

We encourage trialists and epidemiologists to be more explicit about their inferential goals. In particular, an open question is whether trialists conducting intention-to-treat analyses are really interested in the intention-to-treat effect. The per-protocol effect, which is not affected by differential implementation of the treatment strategies being compared, may often be the ultimate target.

Causal diagrams help reduce confusion created by ambiguous terminology.30 For example, the term selection bias is used with different meanings by trialists and epidemiologists. Drawing the corresponding causal diagram helps resolve these confusions. The structural approach to bias using causal diagrams also shows that some biases that are described using different terms in the RCT literature have the same structure. For example, Figures 1B and 4A are essentially the same apart from the time and reason for confounding.

Our simplistic graphical presentations of the Cochrane selection, performance, detection, and attrition biases cannot possibly cover all possibilities. Specifically, in trials with time-varying treatments and attrition, Robins’s g-methods (g-formula, inverse-probability weighting, g-estimation) are generally needed to properly adjust for time-varying confounding and selection bias when estimating intention-to-treat and per-protocol effects.16,31

REFERENCES

1. Hernán MA, Robins JM. Causal Inference. 2017.London: Chapman & Hall/CRC.
2. Hernán MA, Hernández-Díaz S. Beyond the intention-to-treat in comparative effectiveness research. Clin Trials. 2012;9:48–55.
3. Heitjan DF. Ignorability and bias in clinical trials. Stat Med. 1999;18:2421–2434.
4. Higgins JPT, Green S. Cochrane Handbook for Systematic Reviews of Interventions. 2008.Chichester: John Wiley & Sons.
5. Higgins JPT, Altman DG, Gøtzsche PC, et al. The Cochrane Collaboration’s tool for assessing risk of bias in randomised trials. BMJ. 2011;343:d5928.
6. Rothman KJ, Greenland S, Lash TL. Rothman KJ, Greenland S, Lash T. Validity in epidemiologic studies. In Modern Epidemiology. 2008:3rd ed. Philadelphia, PA: Lippincott Williams &Wilkins; 128–147.
7. Miettinen OS. Theoretical Epidemiology. 1985.New York, NY: Wiley.
8. Pearl J. Causality. 2009.2nd ed. New York, NY: Cambridge University Press.
9. Greenland S, Pearl J, Robins JM. Causal diagrams for epidemiologic research. Epidemiology. 1999;10:37–48.
10. Hernán MA, Hernández-Díaz S, Robins JM. A structural approach to selection bias. Epidemiology. 2004;15:615–625.
11. Cole SR, Hernán MA. Fallibility in estimating direct effects. Int J Epidemiol. 2002;31:163–165.
12. Hernán MA, Cole SR. Invited commentary: causal diagrams and measurement bias. Am J Epidemiol. 2009;170:959–962; discussion 963.
13. Hernán MA, Hernández-Díaz S, Werler MM, Mitchell AA. Causal knowledge as a prerequisite for confounding evaluation: an application to birth defects epidemiology. Am J Epidemiol. 2002;155:176–184.
14. Mansournia MA, Hernán MA, Greenland S. Matched designs and causal diagrams. Int J Epidemiol. 2013;42:860–869.
15. Food and Drug Administration. International Conference on Harmonisation; Guidance on Statistical Principles for Clinical Trials. Federal Register. 1998;63:49583–49598.
16. Toh S, Hernán MA. Causal inference from longitudinal studies with baseline randomization. Int J Biostat. 2008;4:Article 22.
17. Hernán MA, Robins JM. Instruments for causal inference: an epidemiologist’s dream? Epidemiology. 2006;17:360–372.
18. Rosenberger WF, Lachin JM. Randomization in Clinical Trials: Theory and Practice. 2002.New York, NY: Wiley.
19. Matthews JNS. An Introduction to Randomised Controlled Clinical Trials. 2006.2nd ed. Boca Raton, FL: Chapman & Hall/CRC.
20. Greenland S, Mansournia MA. Limitations of individual causal models, causal graphs, and ignorability assumptions, as illustrated by random confounding and design unfaithfulness. Eur J Epidemiol. 2015;30:1101–1110.
21. Friedman LM, Furberg CD, DeMets DL. Fundamentals of Clinical Trials. 2010.4th ed. New York, NY: Springer.
22. Chow SC, Liu JP. Design and Analysis of Clinical Trials. 2004.2nd ed. Hoboken, NJ: Wiley-Interscience.
23. Greenland S, O’Rourke K. Rothman KJ, Greenland S, Lash T. Meta-analysis. In Modern Epidemiology. 2008:3rd ed. Philadelphia, PA: Lippincott Williams & Wilkins; 652–682.
24. Steyerberg EW. Clinical Prediction Models: A Practical Approach to Development, Validation, and Updating. 2008.New York, NY: Springer.
25. Steyerberg EW, Eijkemans MJ, Habbema JD. Stepwise selection in small data sets: a simulation study of bias in logistic regression analysis. J Clin Epidemiol. 1999;52:935–942.
26. Maldonado G, Greenland S. Simulation study of confounder-selection strategies. Am J Epidemiol. 1993;138:923–936.
27. Greenland S, Rothman KJ. Rothman KJ, Greenland S, Lash T. Fundamentals of epidemiologic data analysis. In Modern Epidemiology. 2008:3rd ed. Philadelphia, PA: Lippincott Williams & Wilkins; 213–237.
28. Ioannidis JPA. Why most discovered true associations are inflated. Epidemiology. 2008;1:640–648.
29. Hernán MA, Hernández-Díaz S, Robins JM. Randomized trials analyzed as observational studies. Ann Intern Med. 2013;159:560–562.
30. Shrier I. Structural approach to bias in meta-analyses. Res Synth Methods. 2011;2:223–237.
31. Mansournia MA, Altman DG. Inverse probability weighting. BMJ. 2016;352:i189.
    Copyright © 2017 Wolters Kluwer Health, Inc. All rights reserved.