Secondary Logo

Journal Logo

Methods

Regression Discontinuity Design

Simulation and Application in Two Cardiovascular Trials with Continuous Outcomes

van Leeuwen, Nikki; Lingsma, Hester F.; de Craen, Anton J.M.; Nieboer, Daan; Mooijaart, Simon P.; Richard, Edo; Steyerberg, Ewout W.

Author Information
doi: 10.1097/EDE.0000000000000486

Abstract

Randomized controlled trials (RCTs) are the reference standard to demonstrate the efficacy of medical interventions. However, recruitment of sufficient numbers of patients is a challenge in RCTs, and becomes increasingly difficult due to regulatory requirements. Including participants is particularly challenging when the effect of an intervention on an outcome is of interest, but withholding treatment is considered unethical. Also, patients may not want to be randomized1,2 or physicians are not willing to include patients.3 It is estimated that between only 3% and 5% of all eligible adult cancer patients in the United Stated and the United Kingdom enroll in RCTs, partly due to dislike of the concept of trials and the idea of randomization of both patient and clinician.4 Low recruitment rates in trials are also common in other fields, especially in surgery5,6 and elderly7,8 research. Failure to achieve recruitment goals limits statistical precision, leads to an increase of costs, and decreases the efficiency of a RCT.9 Even when investigators enroll an adequate number of participants, they rarely do so on schedule.3,10,11

Second, low recruitment rates threaten the generalizability of the findings in RCTs. Patients enrolled in trials may poorly represent the population of interest.8,12 Mostly women and elderly are underrepresented in RCTs.8,13

An alternative epidemiologic design to assess effectiveness of medical treatment is the quasi-experimental “regression discontinuity” design. This design is common in the social sciences, and was introduced in public health and medicine in 1996.14 Although in other fields regression discontinuity has been evaluated,15–20 recently Vandenbroucke and le Cessie,21 Bor et al.,22 and O’Keeffe et al.23 noted the importance of studying the feasibility and robustness of this design in clinical settings. In the regression discontinuity design, treatment is not allocated randomly, but is assigned to a subset of patients, based on a threshold of a baseline characteristic. The control group consists of a subset of patients below the threshold, not receiving treatment. The threshold variable does not necessarily have to be prognostic for the outcome measure assessed in the study, e.g., all patients with a baseline blood pressure (BP) over 140 mmHg may receive treatment (intervention group) and patients with a baseline BP below 140 mmHg do not receive treatment (control group). Such treatment allocation closely resembles clinical practice and may thus facilitate easier recruitment of participants into a prospective, comparative study. Due to the assignment rule, RDs can achieve balance on unobserved factors, just like in an RCT. When estimating the treatment effect, a regression analysis compares treated to control patients, while adjusting for the assignment variable, in this example baseline BP. Regression discontinuity provides a possible opportunity for obtaining unbiased causal effect estimates, when experiments are not feasible or when we want to evaluate interventions under “real-life” circumstances.24

The regression discontinuity as a prospective quasi-experimental study might be attractive because the challenges of the randomization process are avoided. However, the estimates from a quasi-experimental regression discontinuity might be different and substantially less efficient compared with an RCT. We aimed to assess validity of this design as a prospective quasi-experimental design compared with a traditional RCT, because this has not been discussed in detail in the epidemiologic literature. In this study, we perform simulation studies and analyze data from two large cardiovascular RCTs as validation studies.

METHODS

Monte Carlo Simulations

We used Monte Carlo simulations to compare regression discontinuity and RCT. One hundred patients were simulated with a hypothetical mean prognostic measurement of 10 and a SD of 2. The mean outcome was 90 (SD, 20) and was normally distributed. A treatment effect of −10 was simulated. Linear correlations of the prognostic measurement with outcome were varied (R2: 0.0, 0.2, 0.5, and 0.8) to assess the importance of the prognostic strength of the adjustment model in the regression discontinuity.

For the RCT, treatment was allocated at random, and sample size was 100 patients in total (50 per arm). In the regression discontinuity setting, treatment was assigned to 50 patients with a prognostic measurement above 10; 50 patients with a prognostic measurement below 10 were used as controls. For both the RCT and regression discontinuity settings, linear regression models were used to estimate the treatment effect, adjusted for the baseline variable both in a linear term and a restricted cubic spline term. The regression discontinuity setting was also analyzed with local linear regression analysis. In local linear regression, only patients around the threshold are used in the analysis to estimate a local treatment effect, while normal regression uses all patients, resulting in a global, or average, treatment effect estimate. Treatment effect estimates were compared in terms of bias (expressed as mean estimated treatment effect) and precision (expressed as mean squared error of the treatment effect estimate). The simulation code is provided in the eAppendix (http://links.lww.com/EDE/B44).

Validation Study

Two cardiovascular trials were used to validate the regression discontinuity in empirical data. The “Prevention of Dementia by Intensive Vascular Care” study (preDIVA) is an ongoing cluster-randomized trial to assess the efficacy of a multicomponent, nurse-led intervention targeting all cardiovascular risk factors in an elderly population (70–78 years). The study protocol has been approved by the medical ethical committee of the Academic Medical Centre.25 The primary outcome of this RCT is incident dementia during 6 years of follow-up. Of 3,533 patients enrolled, 1,894 are in the intervention and 1,639 in the control group. As this RCT is ongoing, we randomly sampled 3,000 patients from the enrolled patients.

The second RCT was the “PROspective Study of Pravastatin in elderly (70–82 years) individuals at risk of vascular disease” (PROSPER). It was conducted between December 1997 and May 1999 and enrolled 5,804 patients, who were assigned to pravastatin (n = 2,891) or placebo (n = 2,913) to reduce the risk of coronary disease in elderly individuals.26

To validate the regression discontinuity, we used continuous measures collected during follow-up, which were not the primary endpoints of the trials, as the outcome variable. To evaluate the influence of the choice of the assignment variable on the estimates, we did multiple analyses using two different baseline measures. For preDIVA, both age and the BP at baseline were used as the assignment variable and the BP at 2-year intermediate follow-up as outcome. Both BP measures were calculated as the mean of two systolic BP measurements during a visit (expressed in mmHg). BP data at 2 years were unavailable for part of the 3,000 randomly selected patients, mostly because they reached a clinical endpoint before 2 years, or missed one study visit. We could hence include 2,346 patients for analysis. For PROSPER, we considered total cholesterol level measured 3 months after inclusion (expressed in mmol/L) as the outcome and both age and total cholesterol at baseline as the treatment assignment variables. After exclusion of patients with missing 3-month total cholesterol, we were able to analyze 5,581 patients from PROSPER.

Statistical Analysis

Baseline characteristics were described with standard descriptive statistics; median and interquartile range for continuous variables and frequencies and percentages for categorical variables.

To analyze the data as an regression discontinuity design, we selected those patients with a value of the assignment variable above a certain threshold treated as the intervention group, and those with a value below that threshold and not treated as control group. This thus led to a sample of approximately half the trial population. In both trials, we used a threshold of the assignment variable known to be used in clinical practice27,28 (e.g., BP > 140 mmHg), or, if not available, a hypothetical threshold (e.g., age > 72 years). Histograms of the assignment variables in both preDIVA and PROSPER are shown in the eAppendix 1 (http://links.lww.com/EDE/B44). In this patient selection, the treatment effect was estimated using a linear regression model, with adjustment for the assignment variable (Y ~ Tx + baseline assignment variable). We further analyzed a hypothetical different cutoff to assess the robustness of the treatment effect estimate to the chosen cutoff. In other words, whether the treatment effect was global or local. Usually this would not be possible in regression discontinuity as the cutoff is determined in advance.

Analysis on these sets of patients was compared with a random sample (repeated 5,000 times) of half of the of the RCT patients. To compare relative efficiency in terms of required sample size between the different approaches we used standard errors (SEs) of the estimated treatment effects in the following formula: (SEregression discontinuity/SERCT)2. This is the ratio of variances in both designs.

As the validity of the regression discontinuity is dependent on proper adjustment for the assignment variable, we explored the relation between the assignment variable and outcome in detail. We assessed nonlinearity with nonlinear restricted cubic spline functions and interaction between the baseline assignment variable and the treatment effect. Both were presented graphically (Figure A, B). A restricted cubic spline function is a smooth function that consists of pieced-together cubic splines that are restricted to be linear in the tails.29 We used the default setting for flexibility of the model with five knots.30 Consequently we used the restricted cubic spline of the assignment variable in the adjustment model to obtain the optimal model fit for adjustment. This method was compared with local linear regression models for the adjustment of the baseline variable. All analyses (RCT and regression discontinuity) were adjusted for the assignment measurement that was used to assign treatment; both age and baseline BP in preDIVA and both age and baseline cholesterol in the PROSPER trial. R2 statistics were calculated to indicate the explained variance of the assignment model.

FIGURE
FIGURE:
Nonlinear restricted cubic spline functions of the interaction of the intervention effects over the assignment variables in both the preDIVA study (n = 2,346) and the PROSPER trial (n = 5,581). A, Nonlinear rcs function (the function fitted is 2 year blood pressure ~ intervention * rcs [baseline blood pressure]) of the interaction of the intervention effect over baseline blood pressure in the preDIVA study. B, Nonlinear rcs function (the function fitted is 3-month cholesterol ~ treatment * rcs [baseline cholesterol]) of the interaction of the treatment effect over baseline total cholesterol in the PROSPER trial. C, Nonlinear rcs function (the function fitted is 2-year blood pressure ~ intervention * rcs [baseline age]) of the interaction of the intervention effect over baseline age in the preDIVA study. D, Nonlinear rcs function (the function fitted is 3-month cholesterol ~ treatment * rcs [baseline age]) of the interaction of the treatment effect over baseline age in the PROSPER trial. RCS indicates restricted cubic spline.

We further explored different assumptions on interaction between the assignment variable and treatment. We assumed no interaction between age and treatment in both studies. Therefore, we considered the treatment effect estimates in the regression discontinuity studies in which treatment was assigned on age, global treatment effects. We compared these treatment effect estimates to the global effect estimated in the RCT. The treatment effect estimates from the regression discontinuity studies, where treatment was assigned on baseline BP and baseline total cholesterol level, could be considered as local treatment effects, because we assumed interaction between treatment over both baseline BP and baseline cholesterol level. Therefore, we compared these estimates to the local effects in the RCT, estimated with restricted cubic spline adjustment. These estimates are the differences between the treated and untreated lines in Figure A, B. Treatment effects were estimated using linear regression and expressed as regression coefficients with 95% confidence intervals (95% CIs) for BP or cholesterol level in the treatment group compared with the controls.

All statistical analyses were performed in R statistical software version 2.15.3 (R Foundation for Statistical Computation, Vienna, Austria) using the rdd package and Harrell’s rms package.

RESULTS

Simulations

Simulations showed that under the ideal situation of a linear relation between the assignment variable and the outcome, no unmeasured confounders, and no treatment effect interaction, regression discontinuity provided unbiased treatment effect estimates all scenarios (Table 1). However, regression discontinuity with linear adjustment resulted in substantially less precise effect estimates compared with the RCT. For instance, in the hypothetical setting with an R2 of 20% for the correlation of the assignment measurement with outcome, the mean squared error of the estimated treatment effect estimate in an RCT was 3.2 compared with 9.0 in the regression discontinuity design. In this example, this means that when the regression discontinuity is used and the analysis matches the underlying true model, triple the number of patients are required to obtain the same statistical precision as when using an RCT. This magnitude of (in)efficiency was consistent for all simulated correlations between the assignment measurement and outcome. Regression discontinuity estimates adjusted with restricted cubic splines were seven times less efficient than the RCT estimates analyzed with restricted cubic splines. When analyzing regression discontinuity with local linear regression, the estimated were on average 1.4 times less efficient than analyzing regression discontinuity with restricted cub spline adjustment (Table 1).

TABLE 1
TABLE 1:
Simulations Treatment Effect Over Baseline in Randomized Control Trial and Regression Discontinuity Design, Analyzed with Linear Regression, Restricted Cubic Spline Functions and Local Linear Regression

Validation Study

In the validation study, we assessed BP in 2,346 patients from the preDIVA trial. The median age was 74 years (interquartile range [IQR]: 72, 76) and the median BP at baseline was 153 mmHg (IQR: 140, 168). We analyzed 5,581 patients from the PROSPER trial, who had a median age of 74 years (IQR: 72–77 years) and a median total cholesterol level of 5.6 mmol/L (IQR: 5.0, 6.3) at baseline (Table 2).

TABLE 2
TABLE 2:
Patient Characteristics of preDIVA (n = 2,346) and PROSPER (n = 5,581)

In the RCT design, the treatment effect estimate on BP at 2 years adjusted for baseline BP was −4.0 mmHg (95% CI: −5.4, −2.6; Table 3).

TABLE 3
TABLE 3:
RCT and Regression Discontinuity Analyses in the preDIVA Study (n = 2,346)

The adjusted estimated effect was −5.9 mmHg (95% CI: −10.8, −1.0) with the hypothetical regression discontinuity, adjusted for BP as restricted cubic spline variable. Here, patients with a BP over a baseline BP of 140 mmHg would receive treatment and patients with a lower BP would not receive treatment. An additional analysis of a hypothetical regression discontinuity design in which patients with a BP of 160 mmHg would receive treatment and the higher baseline BP patients not adjusted with a restricted cubic spline showed a treatment effect estimate of −5.9 mmHg (95% CI: −11.4, −0.3; Table 3).

In the preDIVA trial, there appeared to be a different treatment effect for patients with relative low baseline BP compared with high baseline BP, indicating statistical interaction (Figure A). This explains the different effect estimates in the regression discontinuity setting with treated high-risk patients and low-risk controls compared with the RCT estimate. When the treatment effect was analyzed with local linear regression, the intervention effect estimates in the two regression discontinuitys were more different than the RCT estimate: −9.3 mmHg (95% CI: −18.5, −0.1) for the setting with the cutoff at 140 and −10.2 mmHg (95% CI: −21.0,0.6) for the setting with the cutoff at 160 mmHg (Table 3).

Figure C shows the interaction between treatment and age. A global effect of treatment over the age range was assumed. The estimates from the regression discontinuity analyzed with restricted cubic splines showed more similar estimates for the different cutoffs and these were closer to the estimate from the RCT (−0.66 [−6.44, 5.12], −2.71 [−7.16, 1.74]).

In the PROSPER trial, the estimated treatment effect on total cholesterol level at 3 months in the RCT, adjusted for baseline total cholesterol level was −1.31 mmol/L (95% CI: −1.35, −1.27). In a hypothetical regression discontinuity, we used a threshold of 5.0 mmol/L for treatment assignment. The treatment effect estimate in PROSPER was beneficial over the whole range of baseline total cholesterol level, but differed in magnitude (Figure B). Analysis with local linear regression showed different treatment effect estimates: −1.04 mmol/L (95% CI: −1.16, −0.93) with the cutoff at baseline BP of 5.0 and −1.29 mmol/L (95% CI: −1.40, −1.18) with the cutoff at baseline BP of 5.5 mmol/L (Table 4). Analysis with restricted cubic spline adjustment in regression discontinuity in PROSPER showed only slightly different treatment effect estimates compared with the results from the regression discontinuity setting analyzed with local linear regression (Table 4).

TABLE 4
TABLE 4:
RCT and Regression Discontinuity Analyses in the PROSPER Trial (n = 5,581)

In the hypothetical regression discontinuity in PROSPER with assignment on the age variable, both analyses with local linear regression and normal linear regression with adjustment for age as a restricted cubic spline variable showed similar treatment effect estimates compared with the estimates from the RCT (Table 4). This is consistent with Figure D, which shows a constant effect of treatment over the whole range of age.

In terms of statistical precision, the regression discontinuity with restricted cubic spline adjustment was one to four times less efficient than the adjusted RCT for the local effects estimated, and six to 12 times less efficient for the global effects estimated in regression discontinuity compared to the RCT (eAppendix; http://links.lww.com/EDE/B44).

DISCUSSION

In this study, we performed simulation studies and analyzed data from two large cardiovascular RCTs with the aim to assess the validity of the regression discontinuity as a prospective quasi-experimental design compared with a traditional RCT. In the simulations studies, we found unbiased but substantially less precise treatment effect estimates from an regression discontinuity compared with an RCT design.

In the two validation studies, we found somewhat different results. In the case of the treatment assignment on baseline BP and baseline cholesterol level, we assumed treatment effect heterogeneity over the baseline assignment variable. Therefore, we assumed the treatment effect estimates in these RDs to be local. The estimates from the analyses with local linear regression are not comparable with the RCT estimates in terms of bias because these effects are local, in contrast to the global RCT estimates. In terms of statistical precision, the regression discontinuity with restriction cubic spline adjustment was one to four times less efficient than the adjusted RCT for the local effects estimated.

When treatment was assigned on the age variable, we assumed no interaction between age and treatment. Therefore, the treatment effect estimate for the regression discontinuity analysis was assumed to be a global treatment effect. Estimates from the analyses with flexible functions (restricted cubic splines) to obtain an optimal fit of the adjustment model were comparable with the RCT estimate. In PROSPER, the treatment effect estimates were consistent over the different cutoffs, which confirms the assumption of no interaction. In preDIVA, the estimates were quite different over the different cutoffs. This gives the impression that there is interaction and that the treatment effects are local over the baseline age range. Furthermore, we found that sample size needs to be at least six times larger to make regression discontinuity as precise as an RCT to estimate global treatment effect estimates.

Position of Regression Discontinuity Design in Epidemiology

The main goal of our study was to assess the regression discontinuity as a prospective design, and compare it with an RCT design. The regression discontinuity could be implemented as an observational study design, but we focused on the situation of prospective enrollment of patients, with a predefined cutoff. While estimation of treatment effects with retrospectively collected data could be hampered by selection bias and confounding by indication, which is difficult to fully control because unmeasured confounding cannot be accounted for (“residual confounding”), in the prospective application of the regression discontinuity the confounding variable is measured and controlled by design.

Local or Global Treatment Effect and Model Specification

A very important question in designing a regression discontinuity study is whether a global or a local treatment effect should be assumed. The assignment variable in an RCT is random allocation and therefore would not interact with treatment in an RCT. This results in a global or average treatment effect. In contrast, the assignment variable in regression discontinuity could interact with treatment. Therefore, the estimated effect is not always a global effect, but a local treatment effect estimate. Only when the assumption of no interaction between the baseline assignment variable and treatment can be made, the estimated effect of regression discontinuity can be interpreted as a global treatment effect. Our analyses with normal linear regression models with restricted cubic spline adjustment assume such a global treatment effect. In our RCT data, we could test the assumption of no interaction as we had treatment and control patients across the complete range of the assignment variable. In practice, however, this assumption cannot be tested in RDs, as treatment and control patients have no overlap in the assignment variable.

When the aim is to estimate the treatment effect on a certain threshold value of the assignment variable, local linear regression should be used. This estimate will be unbiased for that certain cutoff, but it is only valid for a small subset of patient around the cutoff and is not generalizable for the whole population.

Thus, if one aims to estimate an overall treatment effect from regression discontinuity, using a normal regression model is preferred over local linear regression, as supported by our finding that the treatment effect estimates from regression discontinuity with restricted cubic splines more were more similar to the RCT estimate than the local linear regression estimates. However, this could be expected because local linear regression uses only data around the cutoff, while normal regression uses all data, as happens in the RCT. In fact the local linear regression and restricted cubic splines estimates cannot be judged as more or less biased. They estimate a different effect (global or local) and their validity is dependent on the assumptions made.

In our case of BP, the assumption of no interaction was not met: graphical inspection showed qualitative treatment effect heterogeneity (Figure A). The intervention is a multicomponent intervention tailored according to patients’ risk profile, and the intensity of the intervention (both medical and lifestyle) is thus higher for participants with a higher risk profile.25 This difference in intensity might explain the interaction of treatment with baseline risk for outcome, i.e., a stronger effect in patients with higher baseline risk. This led to a different treatment effect when we misspecified the adjustment model.

Optimal modeling of the effect of the assignment variable is extremely important in regression discontinuity. We used restricted cubic spline functions and local linear regression. Restricted cubic spline functions are attractive for their flexibility with low degrees of freedom. They are not driven by extreme values in both ends of the fit, which is an advantage over ordinary cubic spline functions.29 We suggest that flexible functions should be used for optimal adjustment because this function accounts for differential effects of the baseline variable that is used as the assignment variable, on outcome.

Choice of Regression Discontinuity Versus Randomized Clinical Trials

Requirements for informed consent for clinical trials are often more stringent than for treatment outside of the setting of an RCT.31 Patient enrollment may hence be easier due to avoidance of the randomization process. More lenient inclusion criteria and easier enrollment when using the regression discontinuity design most likely will result in more representative cohorts for analysis.

A regression discontinuity is attractive when random assignment of treatment is problematic or not possible. This may occur when a medical intervention is already standard practice for a part of the patients in clinic but the effectiveness has not yet been assessed. For instance, the effect of BP and cholesterol lowering on incident dementia has only been studied in RCTs using dementia as a secondary outcome and with inconclusive results.32 At present performing such RCTs is no longer considered ethical, as there is a clinical imperative to treat those with high BP and cholesterol. However, there is circumstantial evidence that there is a beneficial effect of BP reduction on dementia risk, which is not translated to clinical guidelines in the absence of evidence to substantiate this claim. In this situation, regression discontinuity could be a solution to assess a treatment effect using less affected patients for whom an intervention is not deemed indicated as control patients. In fact, because this strategy already closely resembles clinical practice, it may well be feasible to include a number of participants in such a trial that is six- to 12-fold higher than in a classical RCT. Adherence to assignment of treatment according to the threshold value is crucial, and both participating clinicians and patients should therefore be well aware that they are participating in a comparative study. A possible threat when using a regression discontinuity is selection bias near the threshold value. When physicians selectively treat patients just below the threshold value and vice versa, selection bias occurs due to confounding by indication. It is thus very important to avoid such protocol violations in a prospective regression discontinuity study.

Strengths and Limitations

A limitation of this study is that in the simulation study, we assumed the ideal condition of a linear treatment effect and no residual confounding, which may not reflect real life practice. However, we showed that even in such an ideal setting regression discontinuity is less efficient than RCT. Furthermore, we only studied continuous outcomes and therefore cannot draw conclusions on the performance of the regression discontinuity for dichotomous outcome parameters. The relative inefficiency may be different for such settings. Furthermore, we assessed the regression discontinuity in RCT data, in which we artificially set the threshold. This results in perfect adherence to the defined threshold, which is unlikely to occur in real life.

On the other hand, the possibility to change the cutoff was a major strength in our study. We were able to study interaction by varying the cutoff, which would be impossible in a “real” regression discontinuity. Moreover, with the RCT data, we were able to study the effect of different hypothetical variables to assign treatment, which is a unique feature of this study.

There might be measurement error in the assignment variable. Many claim that regression to the mean caused by such measurement error is a possible threat to the validity of regression discontinuity.14 A high baseline measurement will on expectation regress down to a lower value and a low baseline measurement will on expectation regress up to a higher value. However, as this will occur equally on both sides of the cutoff in the assignment variable, the measurement error in the end will be irrelevant for the correct estimation of the treatment effect.21

CONCLUSION

We conclude that the regression discontinuity has perfect theoretical validity and may have reasonable validity in real life situations compared with RCT. Regression discontinuity may provide similar estimates of treatment effects to RCT estimates, but requires the assumption of a global treatment effect over the range of the assignment variable. Controlling for the assignment variable is essential and may be achieved by an optimal fit of the adjustment model, with for example restricted cubic splines when the assumption of a global treatment effect over the range of the assignment variable can be made. Regression discontinuity is, however inefficient, requiring sample sizes which are over six times higher than for conventional RCTs to obtain the same statistical precision for a global treatment effect estimate. When considering a study with regression discontinuity versus RCT design, in addition to a risk of bias due to wrong assumptions, researchers need to weigh better recruitment against the substantial loss in precision.

ACKNOWLEDGMENTS

We would like to thank two reviewers, including Dr. Jacob Bor, for many comments and suggestions for improvement of the manuscript.

REFERENCES

1. Wheeler JL, Segal AD, Ledoux WR, et al. Patient willingness to enroll in a randomized surgical trial. Presented at the 25th Annual Summer Meeting of the American Orthopaedic Foot and Ankle Society. July 15–18, 2009. British Columbia.Vancouver.
2. Jenkins V, Fallowfield L. Reasons for accepting or declining to participate in randomized clinical trials for cancer therapy. Br J Cancer. 2000;82:1783–1788.
3. Gotay CC. Accrual to cancer clinical trials: directions from the research literature. Soc Sci Med. 1991;33:569–577.
4. Harrison JD, Solomon MJ, Young JM, et al. Surgical and oncology trials for rectal cancer: who will participate? Surgery. 2007;142:94–101.
5. Abraham NS, Young JM, Solomon MJ. A systematic review of reasons for nonentry of eligible patients into surgical randomized controlled trials. Surgery. 2006;139:469–483.
6. Creel AH, Losina E, Mandl LA, et al. An assessment of willingness to participate in a randomized trial of arthroscopic knee surgery in patients with osteoarthritis. Contemp Clin Trials. 2005;26:169–178.
7. Chang BH, Hendricks AM, Slawsky MT, Locastro JS. Patient recruitment to a randomized clinical trial of behavioral therapy for chronic heart failure. BMC Med Res Methodol. 2004;4:8.
8. Martinson BC, Crain AL, Sherwood NE, Hayes MG, Pronk NP, O’Connor PJ. Population reach and recruitment bias in a maintenance RCT in physically active older adults. J Phys Act Health. 2010;7:127–135.
9. Vozdolska R, Sano M, Aisen P, Edland SD. The net effect of alternative allocation ratios on recruitment time and trial cost. Clin Trials. 2009;6:126–132.
10. Halpern SD. Prospective preference assessment: a method to enhance the ethics and efficiency of randomized controlled trials. Control Clin Trials. 2002;23:274–288.
11. Institute of Medicine Forum on Drug Discovery D, Translation. Opportunities: Workshop Summary.2010.
12. Gross CP, Mallory R, Heiat A, Krumholz HM. Reporting the recruitment process in clinical trials: who are these patients and how did they get there? Ann Intern Med. 2002;137:10–16.
13. Beckie TM, Mendonca MA, Fletcher GF, Schocken DD, Evans ME, Banks SM. Examining the challenges of recruiting women into a cardiac rehabilitation clinical trial. J Cardiopulm Rehabil Prev. 2009;29:13–21quiz 22.
14. Shadish WR, Cook TD, Campbell DT. Experimental and Quasi-Experimental Designs for Generalized Causal Inference. 2002:Boston, MA: Houghton-Mifflin; 229.
15. Cook TD, Shadish WR, Wong VC. Three conditions under which experiments and observational studies produce comparable causal estimates: new findings from within-study comparisons. J Policy Anal Manag. 2008;27:724–750.
16. Wong VC, Cook TD, Barnett WS, Jung K. An effectiveness-based evaluation of five state pre-kindergarten programs. J Policy Anal Manag. 2008;27:122–154.
17. Aiken LS, West SG, Schwalm DE, Carroll JL, Hsiung S. Comparison of a randomized and two quasi-experimental designs in a single outcome evaluation - Efficacy of a university-level remedial writing program. Eval Rev. 1998;22:207–244.
18. Goldberger AS. Selection bias in evaluating treatment effects: some formal illustrations. 1972.Working Paper, Economics Department, University of Wisconsin.
19. Buddelmeyer H, Skoufias E. An evaluation of the performance of regression discontinuity design on PROGRESA. World Bank Policy Research. Aug 2004.
20. Black D, Galdo J, Smith J. Estimating the selection bias of the regression discontinuity design using experimental data. 2007.Working paper, University of Chicago.
21. Vandenbroucke JP, le Cessie S. Commentary: regression discontinuity design: let’s give it a try to evaluate medical and public health interventions. Epidemiology. 2014;25:738–741.
22. Bor J, Moscoe E, Mutevedzi P, Newell ML, Bärnighausen T. Regression discontinuity designs in epidemiology: causal inference without randomized trials. Epidemiology. 2014;25:729–737.
23. O’Keeffe AG, Geneletti S, Baio G, Sharples LD, Nazareth I, Petersen I. Regression discontinuity designs: an approach to the evaluation of treatment efficacy in primary care using observational data. BMJ. 2014;349:g5293.
24. Moscoe E, Bor J, Bärnighausen T. Regression discontinuity designs are underutilized in medicine, epidemiology, and public health: a review of current and best practice. J Clin Epidemiol. 2015;68:122–133.
25. Richard E, Van den Heuvel E, Moll van Charante EP, et al. Prevention of dementia by intensive vascular care (PreDIVA): a cluster-randomized trial in progress. Alzheimer Dis Assoc Disord. 2009;23:198–204.
26. Shepherd J, Blauw GJ, Murphy MB, et al.; PROSPER study group. PROspective Study of Pravastatin in the Elderly at Risk. Pravastatin in elderly individuals at risk of vascular disease (PROSPER): a randomised controlled trial. Lancet. 2002;360:1623–1630.
27. White HD, Simes RJ, Anderson NE, et al. Pravastatin therapy and the risk of stroke. N Engl J Med. 2000;343:317–326.
28. Port S, Demer L, Jennrich R, Walter D, Garfinkel A. Systolic blood pressure and mortality. Lancet. 2000;355:175–180.
29. Steyerberg EW. Clinical Prediction Models. A Practical Approach to Development, Validation, and Updating. 2009.New York, NY: Springer.
30. Harrell FE Jr, Lee KL, Pollock BG. Regression models in clinical studies: determining relationships between predictors and response. J Natl Cancer Inst. 1988;80:1198–1202.
31. Ellis PM, Butow PN, Tattersall MH. Informing breast cancer patients about clinical trials: a randomized clinical trial of an educational booklet. Ann Oncol. 2002;13:1414–1423.
32. Peters R, Beckett N, Forette F, et al.; HYVET investigators. Incident dementia and blood pressure lowering in the hypertension in the very elderly trial cognitive function assessment (HYVET-COG): a double-blind, placebo controlled trial. Lancet Neurol. 2008;7:683–689.

Supplemental Digital Content

Copyright © 2016 Wolters Kluwer Health, Inc. All rights reserved.