Background: The need for evidence about the effectiveness of therapeutics and other medical practices has triggered new interest in methods for comparative effectiveness research.
Objective: Describe an approach to comparative effectiveness research involving cluster randomized trials in networks of hospitals, health plans, or medical practices with centralized administrative and informatics capabilities.
Research Design: We discuss the example of an ongoing cluster randomized trial to prevent methicillin-resistant Staphylococcus aureus (MRSA) infection in intensive care units (ICUs). The trial randomizes 45 hospitals to: (a) screening cultures of ICU admissions, followed by Contact Precautions if MRSA-positive, (b) screening cultures of ICU admissions followed by decolonization if MRSA-positive, or (c) universal decolonization of ICU admissions without screening.
Subjects: All admissions to adult ICUs.
Measures: The primary outcome is MRSA-positive clinical cultures occurring ≥2 days following ICU admission. Secondary outcomes include blood and urine infection caused by MRSA (and, separately, all pathogens), as well as the development of resistance to decolonizing agents.
Results: Recruitment of hospitals is complete. Data collection will end in Summer 2011.
Conclusions: This trial takes advantage of existing personnel, procedures, infrastructure, and information systems in a large integrated hospital network to conduct a low-cost evaluation of prevention strategies under usual practice conditions. This approach is applicable to many comparative effectiveness topics in both inpatient and ambulatory settings.
From the *Department of Population Medicine, Harvard Medical School and Harvard Pilgrim Health Care Institute, Boston, MA; †Hospital Corporation of America, Nashville, TN; ‡Division of Healthcare Quality Promotion, Centers for Disease Control and Prevention, Atlanta, GA; and §Division of Infectious Diseases and Health Policy Research Institute, University of California Irvine School of Medicine, Irvine, CA.
Supported by the Agency for Healthcare Research and Quality (59 HMN HHSA290-2005-0033=TO11), with supplemental funding from the Centers for Disease Control and Prevention (CDC Prevention Epicenters Program, U01CI000344).
R.P. has received research grants from Sanofi-Pasteur, GlaxoSmithKline, Pfizer, and TAP Pharmaceuticals in the past 2 years.
The findings and conclusions in this report are those of the authors, and do not necessarily represent the official position of the Centers for Disease Control and Prevention.
Reprints: Richard Platt, MD, MS, Department of Population Medicine, 133 Brookline Avenue, 6th Floor, Boston, MA 02215. E-mail: firstname.lastname@example.org.
In 2007, the Institute of Medicine's (IOM) Roundtable on Evidence Based Medicine described a “learning healthcare system” as one that both generates and uses evidence to guide clinical decision-making.1 For evidence generation, it called for a diverse array of methodologies to find out what works, under what circumstances, and for whom. Since then, interest in methods for conducting comparative effective research has grown substantially—most recently with the legislative mandate to create a national Patient Centered Outcomes Research Institute. Some have argued that as much as 10% of this new funding should be directed toward methodological guidance and innovation.2 Moreover, with the recent publication of the list of national priorities for comparative effectiveness research, there is heightened interest in matching priority research topics with appropriate methodologies.3
Although methodological advances have improved the validity of nonexperimental methods—including systematic reviews, and quasiexperimental observational studies—experimental methods, in particular randomized clinical trials (RCTs), are still considered the most robust method for generating comparative effectiveness evidence and will undoubtedly remain an important component of an advanced comparative effectiveness research framework.4 The IOM corroborated this view in its national priorities list for comparative effectiveness research by recommending the use of RCTs for 49 of 100 research topics, by far the most frequently recommended methodology.3
Despite their strengths, conventional RCTs have important limitations. Although they are excellent tools for judging efficacy (performance under ideal conditions), they often fail to judge effectiveness (performance under conditions of actual use). This is because most RCTs require more standardization and a higher level of medical care than occurs in practice. In addition, generalizability may be lost because participants in RCTs are often not fully representative of the eventual target group. Furthermore, RCTs are often very costly and time-consuming to implement. For example, the ALLHAT study of initial treatment of hypertension, sometimes cited as an example of a pragmatic clinical trial,2 cost over $80 million and took 8 years to complete.5
Cluster randomized trials are RCTs which randomize groups (clusters) rather than individuals. Cluster randomization is the only feasible method for randomization when an intervention must be applied to an entire group, such as a community-based health promotion initiative. They are also the only method for evaluating interventions for which the status of individuals is linked, for instance when herd immunity for contagious illness is an important consideration. Cluster randomization may also be desirable in some instances in which it would be possible to randomize individuals.6,7 For instance, many organizations use formularies to guide prescribing choices when there is more than one therapeutically equivalent agent. Specific agents may be included or excluded for reasons unrelated to therapeutic effect; examples include cost and streamlined contracting. Randomization of formularies to include one or another preferred drug among those believed to be equivalent can yield balanced groups of individuals exposed to the different agents, allowing evaluation of comparative effectiveness under conditions of actual use. Statistical techniques that use information from all participating individuals while accounting for the tendency of individuals in a cluster to respond similarly (intracluster correlation) minimize the loss of statistical power associated with randomizing groups rather than individuals.7
Cluster randomized trials have several advantages in comparative effectiveness studies, including those evaluating therapeutic interventions and policies.8,9 First, by applying interventions at the hospital, practice, or health plan level, cluster randomized trials can more readily study interventions under conditions of actual use. For instance, a cluster randomized trial that uses existing clinical and administrative mechanisms incorporates the impact of group dynamics (advocacy, peer pressure, reminders) among healthcare providers. Second, cluster randomized trials are often intended to be applied to an entire hospital, Intensive Care Unit (ICU) or clinic population without exclusion, which enhances generalizability. Third, cluster randomized trials are able to harness the health care delivery system's existing administrative capacities, including quality improvement programs and data collection systems, simplifying the logistics of implementation and reducing study costs.10 The increasing availability of electronic health information facilitates the implementation of cluster randomized trials, as routinely collected electronic health information can be used to assess baseline status, monitor implementation, and measure outcomes.
We illustrate some of the potential of cluster randomization in comparative effectiveness research through a current trial which compares strategies to prevent methicillin-resistant Staphylococcus aureus (MRSA) infections in hospital ICUs—one of the high priority topics identified by the IOM Committee on Comparative Effectiveness Research Prioritization. This example demonstrates several of the design strengths of cluster randomized trials that make them likely to generate comparative effectiveness evidence in an efficient and timely manner, thus enabling swift policy action.
Randomized Evaluation of Decolonization vs Universal Clearance to Eliminate Methicillin Resistant S. Aureus (REDUCE-MRSA) Trial
MRSA has become the most common cause of hospital-acquired infection in many US hospitals. Recent national surveys estimate that 5% to 7% of inpatients harbor MRSA,11–13 with higher prevalence in ICUs.14 Because MRSA infections are associated with increased mortality, costs, and length of stay,15,16 hospitals have implemented a variety of quality improvement strategies to reduce hospital-associated MRSA disease, usually targeting high-risk patients (eg, ICU or surgical patients). Three dominant prevention strategies have emerged:
1. Active screening and isolation. This involves obtaining anterior nares surveillance cultures from all patients at the time of ICU admission to identify carriers.14,17,18 Contact Precautions (private room, if available, and gloves and gowns for patient care) are instituted for patients with positive surveillance cultures and for those previously identified as MRSA-positive during prior admissions. Several states mandate MRSA screening, although most do not require subsequent contact isolation.19,20 A resource-intensive alternative, practiced more widely in Europe, uses Contact Precautions for all patients deemed at high risk (eg, recently hospitalized in a high MRSA-prevalence country) until they are demonstrated to be culture-negative.21,22
2. Active screening and decolonization of MRSA carriers. Carriers identified through surveillance cultures are placed in Contact Precautions and decolonized through repeated applications of topical antimicrobials, most commonly chlorhexidine on the skin and intranasal mupirocin. This strategy has been shown to be highly effective in eliminating MRSA carriage, at least in the short-term.23–26 It has also been associated with lower rates of hospital-associated MRSA transmission and infection.25–28
3. Universal decolonization without regard to MRSA status. Universal decolonization is achieved by applying to all ICU patients the same chlorhexidine and mupirocin agents that are used for selective decolonization. The rationale for this strategy includes both operational simplicity and the potential to affect a broad range of nosocomial pathogens, not just MRSA. Decolonization is applied to all patients on the day of admission to the ICU and thus is not susceptible to the usual delay in identification of carriers or false-negative surveillance cultures. It avoids the need for a routine microbiologic surveillance program and allows ICUs to treat all patients (and pathogens) alike. Because universal decolonization without initial MRSA screening of all patients also reduces the use of Contact Precautions, it may reduce overall costs and lead to fewer negative consequences associated with isolation precautions such as decreased visitation by hospital staff, increased fall risk, and potential depression among patients who feel isolated.29–31
Although some evidence supports each of these approaches to reduce nosocomial MRSA infection, it remains unclear which is most effective. This is especially important in light of the increasing number of legislative directives that require a specific approach,20 even in the absence of consistent and generalizable evidence of benefit. A cluster randomized trial is an efficient, relatively low-cost method to compare the effectiveness of these approaches.
The REDUCE-MRSA trial (available at: www.clinicaltrials.gov ID#NCT00980980) is a cluster randomized trial evaluating 3 interventions intended to reduce MRSA disease in hospital ICUs. Hospitals were randomized to 1 of the 3 MRSA infection prevention strategies described above, with all adult ICUs in participating hospitals assigned to the same intervention. Cardiac surgery ICUs were excluded if they were implementing routine preoperative nares screening and decolonization of S. aureus carriers.
The REDUCE-MRSA trial is being conducted in 45 Hospital Corporation of America (HCA) hospitals with an average of 1400 annual ICU admissions per hospital. HCA has a strong central administrative structure, including system-wide quality improvement and infection control and prevention programs, and centralized informatics capabilities. HCA’s standard practice since 2007 included active surveillance for MRSA and Contact Precautions for carriers.
Of the 45 participating hospitals, 39 were randomized to the 3 intervention arms and 6 were separately randomized to only arm 1 (screen and isolate) or arm 2 (screen with targeted decolonization if MRSA-positive) because they were located in states with mandatory ICU MRSA screening laws. We assigned the 39 hospitals into 6 groups of 6 hospitals and 1 group of 3 hospitals by rank order of total annual ICU admissions. Within these groups, hospitals were ranked by the MRSA prevalence (from admission screening and clinical culture data) across all adult ICUs. Within this new rank order, each group of 3 consecutive hospitals was then randomized with 1 hospital being randomly assigned to each of the 3 arms. This approach offered a balance between potential large differences in the number of ICU admissions and MRSA prevalence across study arms, relative to randomization that controlled solely for one of these factors.
The interventions use the hospitals’ existing in-service education, compliance, and adherence monitoring programs. Each study arm is implemented and monitored by ICU Directors, Nurse Educators, and Infection Control and Prevention program managers at participating hospitals, rather than specially trained onsite study teams. Leadership from system-wide departments of Quality Improvement, Infection Control and Prevention, and Critical Care enabled standardized communication and rapid dissemination of study materials to all participants via usual HCA channels. Thus, the trial provides a realistic evaluation of the performance of these interventions as they would occur in HCA’s regular practice.
A Steering Committee composed of members of the CDC Prevention Epicenters Program and coinvestigators at HCA provides oversight.
The primary outcome is the number of ICU patients who have MRSA-positive clinical cultures occurring at least 2 days after ICU admission through 2 days after ICU discharge. Secondary outcomes include ICU and post-ICU bloodstream and urinary tract infections caused by MRSA, bloodstream and urinary tract infections caused by all pathogens, and an increase in the prevalence of resistance to the decolonizing agents among MRSA clinical isolates.
HCAs informatics systems are the source of information on number and duration of ICU and post-ICU stays, discharge diagnoses, use of decolonizing agents, microbiology results, and costs.
Statistical Analysis and Power
For bivariate analyses, we will assess whether the main and secondary outcomes in any of the 3 study arms are significantly different by using a generalized linear mixed model32,33 with adjustment only for clustering. This is the appropriate cluster-randomized equivalent of the ordinary (unadjusted) χ2 test used in a conventional RCT. We will then perform comprehensive multivariate analyses adjusting for covariates using generalized linear mixed models that account for clustering of outcomes within participating ICUs, as well as adjust for case mix, event rates in the year before the intervention period, and the amount of MRSA imported into participating ICUs.32 If the true probabilities of MRSA-positive nosocomial clinical cultures are 0.016 versus 0.005 per patient, and intraclass correlation (ICC) is assumed to be 0.001, then we have >99% power to observe a significant difference between the selective and nonselective decolonization arms (PASS software, NCSS, Kayem, UT).34 Probability estimates were guided by published literature from tertiary care academic centers,13,35,36 but additionally account for the secular trend toward reduced MRSA infections and the anticipated lower incidence in community hospitals. The ICC choice followed evidence that ICC for dichotomous outcomes should be based upon prevalence.37,38 Even if the ICC were as large as 0.01, power for the primary outcome would be 81%.
The Harvard Medical School Department of Population Medicine at the Harvard Pilgrim Health Care Institute leads one of the Centers for Disease Control and Prevention (CDC) Prevention Epicenters39 and is the lead institution for this study. Oversight is provided by the HPHC Institutional Review Board (IRB).
The IRBs of 41 hospitals delegated primary review responsibility to the HPHC IRB, as did both the corporate HCA IRB and the CDC IRB through IRB Authorization Agreements to streamline the IRB review process. Human protection training requirements were established by the lead IRB for all sites. Agreements were obtained from each hospital's Infection Control and Prevention program to inform the study team of unusual trends in the rate or type of ICU infections. In addition, these programs were required to notify study investigators of any limitations in their program that might affect their ability to monitor infection trends during the course of the study.
The IRB process, although centralized, required a significant amount of information relating to the local research context for each relying institution. It also required a Subpart C review and an Epidemiological waiver from the Office for Human Research Protections (OHRP) for the possible inclusion of prisoners admitted to ICUs at each site.
The IRB waived documentation of informed consent, finding that the study involved minimal risk (evaluation of standard hospital decolonization practices) and met the regulatory criteria under 45 CFR 46.116 (d), 117 (c) (2), and 21 CFR 56.109 (c) (1).40 Certification of infection control and surveillance, a limited study timeline and interim IRB reporting further mitigates potential risk.
The incremental costs of research (compared with implementation without randomization), formal evaluation of outcomes, assessment of impact on resistance, and monitoring of costs are <$2 million. These include investigator effort, informational materials, research supplies, informatics-related costs, and the costs of collecting and testing clinical MRSA isolates for susceptibility to topical decontamination agents. Active surveillance cultures, Contact Precautions, and decolonizing regimens were considered to be routine care and were not included in the costs of research, nor were the salaries of HCA personnel who participated in study design and oversight.
The REDUCE-MRSA study is ongoing. Fifty-five hospitals volunteered to participate. Ten were ineligible because of extensive routine use of chlorhexidine bathing, leaving 45 participating institutions. All institutions have launched their assigned intervention. The intervention is expected to end in Summer 2011.
This study illustrates the efficiency of cluster randomization in a network environment to evaluate comparative effectiveness. Salient features of the network that minimize the burden of recruitment, implementation, and evaluation include advocacy by system leaders, streamlined implementation of policies in multiple sites, use of existing quality improvement personnel and methods, use of information about processes of care and outcomes obtained as part of routine medical care, and the availability of informatics capabilities that allow centralized access to these data. The use of existing infrastructure was maximized by randomizing entire hospitals, so that all ICUs in each hospital used the same intervention protocol, supply chain, and compliance and reporting procedures.
An additional important factor in the development of the REDUCE-MRSA trial was the extensive, long-term collaboration between the CDCs public health experts, academic investigators, and HCAs clinical leaders. The existing infrastructure of the CDC-led Prevention Epicenters allowed each group's perspectives and expertise to contribute to shared understandings and capabilities that enabled very rapid design and implementation.
Although our example describes the use of a network of hospitals, a similar approach is applicable to the ambulatory setting, for instance in practices that are part of a single health plan or a practice-based research network.41–43 In any case, it will be essential to have access to sufficiently long-term follow-up data and to ensure that outcome data are available from all care venues. This information may be available to delivery systems, health plans, and insurers.
Cluster randomization does not allow every question of interest to be addressed. It typically results in more misclassification of exposure than RCTs do, either because individual providers choose to use a nonrecommended regimen for some patients, or they fail to adhere fully to the assigned regimen. To the extent that the frequency of failure is consistent with the level that occurs in usual clinical practice, these failures are part of the overall effectiveness measure. Thus, cluster randomization in inpatient or ambulatory systems may offer relatively low-cost and broadly generalizable means to examine many of the other 99 IOM priority topics for comparative effectiveness research.
For the US healthcare system to adopt cluster randomization as a common method for studying comparative effectiveness, several conditions will need to be satisfied. First, the concept of group randomization, often without requiring individual consent, raises important ethical issues about individual choice and participation in research. There will need to be agreement that it is ethical to perform cluster randomization.44 Second, all segments of the healthcare and lay communities need to understand the importance of acquiring information during delivery of care.42,43 Third, the burden of creating infrastructure to implement and evaluate such interventions needs to be reduced through the use of networks and collaboratives; and existing capabilities need to be used to streamline IRB reviews by multiple institutions. Most importantly, the healthcare system needs to recognize both the opportunity and need for research represented by the large amount of medical care that is not evaluated or supported by existing evidence-based research. Variation in care is a particularly important marker of this opportunity and need. It will require relatively little additional resources, compared with the cost of conventional RCTs or the cost of routine care, for clinicians, practices, hospitals, and health plans to form learning collaboratives that introduce systematic variation into regular practice, allowing rapid, efficient, low-cost assessments of comparative effectiveness that address the current gaps in the evidence.
The authors thank the investigators of the CDC Prevention Epicenters for their contributions to the design of the REDUCE MRSA trial.
1.Institute of Medicine. The Learning Healthcare System: Workshop Summary
. Washington, DC: The National Academies Press; 2007.
3.Institute of Medicine. Initial National Priorities for Comparative Effectiveness Research
. Washington, DC: The National Academies Press; 2009.
4.Luce BR, Kramer JM, Goodman SN, et al. Rethinking randomized clinical trials for comparative effectiveness research: the need for transformational change. Ann Intern Med
5.ALLHAT Officers and Coordinators for the ALLHAT Collaborative Research Group. Major outcomes in high-risk hypertensive patients randomized to angiotensin-converting enzyme inhibitor or calcium channel blocker vs diuretic: the Antihypertensive and Lipid-Lowering Treatment to Prevent Heart Attack Trial (ALLHAT). JAMA
6.Campbell MK, Elbourne DR, Altman DG. CONSORT statement: extension to cluster randomised trials. BMJ
7.Murray DM, Varnell SP, Blitstein JL. Design and analysis of group-randomized trials: a review of recent methodological developments. Am J Public Health
8.Jick H, Jick SS, Walker AM, et al. A comparison of wax matrix and microencapsulated potassium chloride in relation to upper gastrointestinal illness requiring hospitalization. Pharmacotherapy
9.Schneeweiss S, Maclure M, Carleton B, et al. Clinical and economic consequences of a reimbursement restriction of nebulised respiratory therapy in adults: direct comparison of randomised and observational evaluations. BMJ
10.Smith DH, Kramer JM, Perrin N, et al. A randomized trial of direct-to-patient communication to enhance adherence to beta-blocker therapy following myocardial infarction. Arch Intern Med
. 2008;168:477–483; discussion 483; quiz 447.
11.Hidron AI, Kourbatova EV, Halvosa JS, et al. Risk factors for colonization with methicillin-resistant Staphylococcus aureus
(MRSA) in patients admitted to an urban hospital: emergence of community-associated MRSA nasal carriage. Clin Infect Dis
12.Jarvis WR, Schlosser J, Chinn RY, et al. National prevalence of methicillin-resistant Staphylococcus aureus
in inpatients at US health care facilities, 2006. Am J Infect Control
13.Robicsek A, Beaumont JL, Paule SM, et al. Universal surveillance for methicillin-resistant Staphylococcus aureus
in 3 affiliated hospitals. Ann Intern Med
14.Huang SS, Rifas-Shiman SL, Warren DK, et al. Improving methicillin-resistant Staphylococcus aureus
surveillance and reporting in intensive care units. J Infect Dis
15.Cosgrove SE, Sakoulas G, Perencevich EN, et al. Comparison of mortality associated with methicillin-resistant and methicillin-susceptible Staphylococcus aureus
bacteremia: a meta-analysis. Clin Infect Dis
16.Engemann JJ, Carmeli Y, Cosgrove SE, et al. Adverse clinical and economic outcomes attributable to methicillin resistance among patients with Staphylococcus aureus
surgical site infection. Clin Infect Dis
17.Muto CA, Jernigan JA, Ostrowsky BE, et al. SHEA guideline for preventing nosocomial transmission of multidrug-resistant strains of Staphylococcus aureus
and enterococcus. Infect Control Hosp Epidemiol
18.Lucet JC, Grenet K, Armand-Lefevre L, et al. High prevalence of carriage of methicillin-resistant Staphylococcus aureus
at hospital admission in elderly patients: implications for infection control strategies. Infect Control Hosp Epidemiol
19.Weber SG, Huang SS, Oriola S, et al. Legislative mandates for use of active surveillance cultures to screen for methicillin-resistant Staphylococcus aureus
and vancomycin-resistant enterococci: position statement from the Joint SHEA and APIC Task Force. Am J Infect Control
21.Harbarth S, Masuet-Aumatell C, Schrenzel J, et al. Evaluation of rapid screening and pre-emptive contact isolation for detecting and controlling methicillin-resistant Staphylococcus aureus
in critical care: an interventional cohort study. Crit Care
22.van Trijp MJ, Melles DC, Hendriks WD, et al. Successful control of widespread methicillin-resistant Staphylococcus aureus
colonization and infection in a large teaching hospital in the Netherlands. Infect Control Hosp Epidemiol
23.Wendt C, Schinke S, Wurttemberger M, et al. Value of whole-body washing with chlorhexidine for the eradication of methicillin-resistant Staphylococcus aureus
: a randomized, placebo-controlled, double-blind clinical trial. Infect Control Hosp Epidemiol
24.Simor AE, Phillips E, McGeer A, et al. Randomized controlled trial of chlorhexidine gluconate for washing, intranasal mupirocin, and rifampin and doxycycline versus no treatment for the eradication of methicillin-resistant Staphylococcus aureus
colonization. Clin Infect Dis
25.Ridenour G, Lampen R, Federspiel J, et al. Selective use of intranasal mupirocin and chlorhexidine bathing and the incidence of methicillin-resistant Staphylococcus aureus
colonization and infection among intensive care unit patients. Infect Control Hosp Epidemiol
26.Climo MW, Sepkowitz KA, Zuccotti G, et al. The effect of daily bathing with chlorhexidine on the acquisition of methicillin-resistant Staphylococcus aureus
, vancomycin-resistant Enterococcus, and healthcare-associated bloodstream infections: results of a quasi-experimental multicenter trial. Crit Care Med
27.Girou E, Pujade G, Legrand P, et al. Selective screening of carriers for control of methicillin-resistant Staphylococcus aureus
(MRSA) in high-risk hospital areas with a high level of endemic MRSA. Clin Infect Dis
28.Sandri AM, Dalarosa MG, Ruschel de Alcantara L, et al. Reduction in incidence of nosocomial methicillin-resistant Staphylococcus aureus
(MRSA) infection in an intensive care unit: role of treatment with mupirocin ointment and chlorhexidine baths for nasal carriers of MRSA. Infect Control Hosp Epidemiol
29.Saint S, Higgins LA, Nallamothu BK, et al. Do physicians examine patients in contact isolation less frequently? A brief report. Am J Infect Control
30.Stelfox HT, Bates DW, Redelmeier DA. Safety of patients isolated for infection control. JAMA
31.Kirkland KB, Weinstein JM. Adverse effects of contact isolation. Lancet
32.Breslow NE, Clayton DG. Approximate inference in generalized linear mixed models. J Am Stat Assoc
33.Murray DM. Design and Analysis of Group-Randomized Trials.
New York, New York: Oxford University Press, Inc.; 1998.
34.Donner A, Klar N. Design and Analysis of Cluster Randomization Trials in Health Research
. London: Arnold; 2000.
35.Harbarth S, Fankhauser C, Schrenzel J, et al. Universal screening for methicillin-resistant Staphylococcus aureus
at hospital admission and nosocomial infection in surgical patients. JAMA
36.Weber DJ, Sickbert-Bennett EE, Brown V, et al. Comparison of hospitalwide surveillance and targeted intensive care unit surveillance of healthcare-associated infections. Infect Control Hosp Epidemiol
37.Gulliford MC, Adams G, Ukoumunne OC, et al. Intraclass correlation coefficient and outcome prevalence are associated in clustered binary data. J Clin Epidemiol
38.Gulliford MC, Ukoumunne OC, Chinn S. Components of variance and intraclass correlations for the design of community-based surveys and intervention studies: data from the Health Survey for England 1994. Am J Epidemiol
41.Brufsky JW, Ross-Degnan D, Calabrese D, et al. Shifting physician prescribing to a preferred histamine-2-receptor antagonist. Effects of a multifactorial intervention in a mixed-model health maintenance organization. Med Care
42.Mazor KM, Sabin JE, Boudreau D, et al. Cluster randomized trials: opportunities and barriers identified by leaders of eight health plans. Med Care
43.Mazor KM, Sabin JE, Goff SL, et al. Cluster randomized trials to study the comparative effectiveness of therapeutics: stakeholders’ concerns and recommendations. Pharmacoepidemiol Drug Saf
44.Sabin JE, Mazor K, Meterko V, et al. Comparing drug effectiveness at health plans: the ethics of cluster randomized trials. Hastings Cent Rep