Making Observational Studies Count

Shaping the Future of Comparative Effectiveness Research

There is substantial interest in using observational epidemiologic research in combination with randomized clinical trials (RCTs) and meta-analyses to support decision-making by regulators, payers, and physicians. Nevertheless, even well-designed and well-conducted observational studies are often viewed with skepticism. This lingering distrust comes, in part, from the well-accepted use of clinical trials compared with a widespread lack of familiarity with the principles for good conduct in observational research. The solution lies in sound guidelines for evaluating observational studies, especially studies that may prove useful for evaluating clinical effectiveness. Such guidance would help to focus on the quality and relevance of the evidence, whatever the study design.

With ever more medical interventions available, the need for hard evidence about which treatments work best, for whom and when, is spurring big changes. These changes are embodied in 3 new laws in the United States, starting with the Food and Drug Administration Amendment Act of 2007,¹ followed by the American Recovery and Reimbursement Act of 2009,² and the Patient Protection and Affordable Care Act of 2010.³ All 3 laws encourage evidence-based research to inform therapeutic decision-making, and ensure that comparative effectiveness research, defined as “the conduct and synthesis of research comparing the benefits and harms of different interventions and strategies to prevent, diagnose, treat and monitor health conditions in ‘real world’ settings,”⁴ will be driving decisions about patient care and health insurance coverage.

Noninterventional studies provide information that is infeasible or impossible to obtain elsewhere.⁵ Interest may lie in risks and benefits that extend far into the future, or in patients and real-world practices that are not well characterized, such as patient subgroups and health practitioner specialties. It may be impractical, too expensive, or politically unacceptable to do a randomized trial or other type of intervention study. Preparing for seasonal influenza is an example. With the threat of a global pandemic of potentially serious swine flu, it would have been politically unacceptable to deny vaccination to some, making observational studies the preferred approach for characterizing the effectiveness of this type of vaccine. For the highly lethal H5N1 (avian) strain of influenza, little is known about what treatments work since the disease is rare and outbreaks are unpredictable. A multicountry patient registry provided data that revealed a 51% reduction in the fatality rate for patients with laboratory-confirmed H5N1 infection who were treated promptly with oseltamivir. This reduction in mortality was smaller but still sizable when treatment was initiated up to 1 week after symptom onset, showing that even delayed treatment was helpful.⁶

Large benefits and risks are difficult to ignore, even when discovered outside the experimental setting. More commonly, though, the true risks and benefits of medical interventions fall in the range where unmeasured, confounding could explain the apparent effect. Epidemiologists regularly engage in discussions about channeling and other types of selection bias that cause people preferentially to receive or avoid a preventive or therapeutic intervention, or to be more or less adherent. Less attention has been paid to factors that affect the availability and use of treatments, such as health insurance coverage, and how these factors may affect the interpretation of findings from observational studies. The lack of attention to these factors is in part because information about what is covered by health insurance is difficult to access, varies by insurer, employer, and country, and changes over time.⁷ Furthermore, this information is not available in commercial or national data sets.

If decision-makers cannot distinguish quality work, meaningful observational work will be overlooked, no matter how well a study is designed, analyzed, and reported. Or worse—all observational research studies, regardless of quality, could be lumped together as if they provided comparable evidence. Some decision makers are skeptical that guidelines can improve the conduct, reporting, and interpretation of research,^8,9 but many others find them useful aids to recognize quality in research. Although there are widely accepted benchmarks for study quality (eg, Good Clinical Practice¹⁰ and CONSORT¹¹ for clinical trials, and PRISMA¹² for meta-analysis), there is a dearth of widely accepted good-practice guidance documents for observational studies, particularly with regard to the challenges that relate specifically to clinical effectiveness.

This situation is starting to change. There is a host of relatively new offerings, including GRACE (Good ReseArch for Comparative Effectiveness) principles for conducting and evaluating observational comparative effectiveness studies,¹³ the ENcEPP (European Network of Centres for Pharmacoepidemiology and Pharmacovigilance) Methodological Standards,¹⁴ some analytic recommendations from the International Society for Pharmacoeconomics and Outcomes Research,^15–17 and the STROBE (Strengthening The Reporting of OBservational studies in Epidemiology) reporting guidelines.¹⁸ With the exception of STROBE, which provides a reporting framework for all types of epidemiologic studies and is not tailored for clinical epidemiology, none of these guidance documents is widely known or broadly accepted yet (beyond the groups that created them). This leaves decision-makers without the support they need to use noninterventional studies with confidence.

The recent good-practice documents developed by the GRACE Initiative were designed specifically to guide decision-makers in the evaluation of observational studies of comparative effectiveness.¹³ These principles describe the value of a good study plan; the usefulness of conducting and analyzing studies in a manner consistent with established good practice at that point in time, and reporting with sufficient detail for evaluation and replication; and they discuss some issues to consider in evaluating the validity of the interpretation. The principles are presented as a “living document”¹⁹ and further development is expected. Guided questions may be useful to decision-makers, but the ultimate value of the GRACE principles will depend on building broad support for this conceptual guidance, especially as it offers neither a checklist nor scoring system.

In contrast, the ENcEPPMethodological Standards for Study Protocols, the newest entry in this quality-improvement field, provides a checklist of epidemiologic principles that should be considered in a protocol. ENcEPP also provides a Code of Conduct that affirms many good practices, such as the right of the principal investigator to independently prepare study publications, and a Guide on Methodological Standards for Pharmacoepidemiology that addresses many topics relating to study design, analysis, and data sources.²⁰ ENcEPP takes the position that post hoc subgroup analyses (ones that were conceived after data analysis has begun) may not be used to verify or reject a hypothesis of a causal association, but any safety signals identified through serendipitous analysis should be evaluated appropriately. Aside from the debatable stance about the value of prespecified hypotheses,^21–23 these guidance documents help to assure that investigators think through the principles of good practice, especially as they relate to safety.

Careful examination and analysis of both interventional and noninterventional studies that are of high quality will improve decisions affecting diagnosis and treatment, and delivery of medical care. The dramatic increased need for reasoned payment decisions may provide the strategic opportunity to advance understanding of the value of observational data for comparative effectiveness research. Guidelines appropriate to observational comparative effectiveness research will promote the acceptance, appreciation, and use of such research to benefit public health.

ABOUT THE AUTHOR

NANCY DREYER is Chief of Scientific Affairs at Outcome Sciences, Inc., where she leads a broad array of studies on the safety and effectiveness of medical interventions. She is a Sr. Editor of the AHRQ's User's Guide, Registries for Evaluating Patient Outcomes, and works with the European Medicines Agency on PROTECT, a project funded by the European Innovative Medicines Initiative.