Secondary Logo

Journal Logo



You Can Observe a Lot (About Medical Products) by Watching (Those Who Use Them)

Gagne, Joshua J.

Author Information
doi: 10.1097/EDE.0b013e31829f642d
  • Free

Over the last three decades, 26 prescription drugs (3.5% of all those approved in this period) have been withdrawn from the US market for safety reasons.1 On average, these drugs were withdrawn 6 years after their approval and, often, after many millions of people had been exposed. Terfenadine, an antihistamine linked to ventricular arrhythmias, was withdrawn from the US market in 1990 after an estimated 100 million people were exposed worldwide.2 In 1996, >18 million prescriptions were dispensed in the United States for the weight-loss drug fenfluramine. The next year, fenfluramine was withdrawn from the market for causing heart valve damage.3

Today, data on hundreds of millions of patients who use prescription medications and other medical products are routinely collected in electronic healthcare databases. These data typically include adjudicated insurance claims for prescription drugs dispensed at pharmacies and transactional records of diagnoses made and procedures performed in various care settings.4 However, as we learned with rofecoxib, despite the ubiquity of these data, their use has not been optimized to quickly identify and quantify potential harms associated with new medical products. A meta-analysis of 11 observational studies demonstrated the ability of well-conducted pharmacoepidemiologic studies to provide valid documentation of the adverse cardiovascular effects of rofecoxib.5 Yet, only two of these 11 studies were completed and published before rofecoxib’s market withdrawal, a withdrawal that followed exposure to rofecoxib by 80 million people worldwide.6

Yogi Berra7 once said, “You can observe a lot by watching.” With rofecoxib, we could have observed its association with myocardial infarction much sooner had we been actively watching patients’ experiences with the drug as they were accruing.8

Using routinely collected electronic healthcare data to more quickly characterize serious adverse effects of medical products requires three fundamental changes to the ways in which we currently assess the safety of these products once they reach the market. It requires analyses that are distributed, sequential, and semiautomated. Distributed analyses allow investigators to leverage multiple databases without the need for pooling the data into a central repository. Analyses in large networks of distributed databases are increasingly common in epidemiology, particularly for evaluating rare outcomes related to medical product exposure.9 Existing and emerging networks, including the Canadian Network for Observational Drug Effect Studies,10 the HMO Research Network,11 and the US Food and Drug Administration’s Mini-Sentinel system,12 provide an unprecedented opportunity to evaluate medical product use and associated outcomes in up to hundreds of millions of patients.

Once a new medical product is authorized for marketing, each of the databases within these networks prospectively captures data describing the use of the product. To characterize associated outcomes as quickly as possible, analyses must also be performed prospectively, in a sequential manner, as the data accrue. The US CDC’s Vaccine Safety Datalink serves as a nice model for distributed, prospective safety surveillance, from which much can be learned.13 Extending the application of active monitoring from vaccines, which relies mainly on self-controlled approaches, to drugs, devices, and other medical products, requires additional methods for addressing confounding in cohort-based design approaches and raises interesting questions about implementing these methods in both distributed and sequential settings.14–16

Semiautomated programs, in which standardized code is sequentially deployed across a distributed network, are required to rapidly perform each analysis as data accrue in each database. Using few inputs, these modularized programs can perform all steps in a typical study, from cohort identification to confounding adjustment to effect estimation and testing. Such programs ensure that methods are applied uniformly across databases and sequential analyses. These programs also enable simultaneous observation of many exposure-outcome pairs across distributed data networks.

Semiautomated, distributed, sequential analysis will improve public health only if it leads to timelier decision making. Just as the data infrastructure and semiautomated analysis programs must be developed in advance, making the optimal use of information that comes out of an active monitoring system requires upfront planning about how the results will support decision making. One important consideration is in which database(s) a given assessment should be conducted. To address this, Maro and colleagues17 in this issue of EPIDEMIOLOGY propose a four-step planning process for optimal database configuration selection for active monitoring within a distributed data network. This process involves: (1) sample size calculation to determine how many patients must be watched to detect a certain effect with specified statistical power; (2) estimation of utilization functions in each candidate database; (3) aggregation of exposure information across database configurations to estimate calendar times at which statistically significant results or target numbers of exposed patients would be reached; and (4) selection of the optimal database configuration. Perhaps the most useful component of this framework is a figure that enables investigators to easily visualize the trade-offs among all the inputs in a sample size calculation for a sequential analysis.

The configuration that uses all available databases will always minimize time-to-detection. However, Maro and colleagues17 remind us that active monitoring as a public health activity must consider public health costs. In particular, the authors point out that using all available databases may be associated with higher costs in terms of public health dollars if, for example, there is a fixed cost associated with including each database in a particular monitoring activity. On the contrary, not every aspect of routine active surveillance can be automated, and longer surveillance time requires more resources to review, interpret, and double-check results of each sequential analysis. Thus, a key consideration in deciding among various database configurations will be the cost of longer surveillance among a set of data partners versus adding one or more additional databases and conducting surveillance for a shorter duration. Maro and colleagues17 recommend decision analysis with uncertainty to consider the monetary costs of each configuration along with the public health consequences of incurring avoidable adverse events.

Quantifying the costs and consequences of all inputs into a decision analytic model will be a difficult, but necessary, part of making efficient use of information that arises from an active monitoring system. In addition to the costs of the databases and the public health consequences of delayed decision making about an unsafe medical product, the costs to follow-up on false-positive signals are a critical consideration that could affect the selection of parameters to minimize type 1 errors. False positives can have potentially profound implications for patients (who might discontinue a needed medication based on concern that it may be unsafe), manufacturers (who stand to lose financially if patients or healthcare provider avoid using their products), and other stakeholders vested in an active monitoring program because false positives could undermine the credibility of the system.

To enhance the effective specificity of an active monitoring program beyond statistical alphas and betas, stakeholders can use alerting thresholds that also integrate the magnitude of observed increased risk. Such thresholds will help ensure that alerts are not only statistically significant but also reflect findings that are large enough to be actionable on clinical or regulatory grounds. Thresholds might depend on the severity of the outcome being monitored, the availability of alternative treatments, and the relative effectiveness of the product being monitored to those treatments. Additional work is required to develop frameworks for selecting such alerting thresholds. As these thresholds share some similarity with margins used in noninferiority trials, this could be a useful place to start.

Ultimately, for a successful active medical product monitoring system, epidemiology must interdigitate with decision analysis. The framework of Maro and colleagues17 for selecting an optimal database configuration makes this explicit. While the data infrastructure and epidemiologic methods for performing rapid, sequential analyses across databases in distributed networks are easily scalable, innovative approaches to planning and scaling the decision-making processes are needed. With a well-planned decision-making process, we will not only “observe a lot by watching” but also be able to efficiently act on what we observe. As Yogi Berra7 also said, “The future ain’t what it used to be”—true especially for postmarketing drug safety surveillance.


JOSHUA J. GAGNE is an Assistant Professor of Medicine in the Division of Pharmacoepidemiology and Pharmacoeconomics, Department of Medicine, Brigham and Women’s Hospital and Harvard Medical School. His research focuses on methods for generating postmarketing comparative safety and effectiveness evidence for new medical products. He has developed a propensity score-based semiautomated active monitoring program for sequential monitoring in distributed databases.


1. Qureshi ZP, Seoane-Vazquez E, Rodriguez-Monguio R, Stevenson KB, Szeinbach SL. Market withdrawal of new molecular entities approved in the United States from 1980 to 2009. Pharmacoepidemiol Drug Saf. 2011;20:772–777
2. Thompson D, Oster G. Use of terfenadine and contraindicated drugs. JAMA. 1996;275:1339–1341
3. Connolly HM, Crary JL, McGoon MD, et al. Valvular heart disease associated with fenfluramine-phentermine. N Engl J Med. 1997;337:581–588
4. Schneeweiss S, Avorn J. A review of uses of health care utilization databases for epidemiologic research on therapeutics. J Clin Epidemiol. 2005;58:323–337
5. McGettigan P, Henry D. Cardiovascular risk and inhibition of cyclooxygenase: a systematic review of the observational studies of selective and nonselective inhibitors of cyclooxygenase 2. JAMA. 2006;296:1633–1644
6. Topol EJ. Failing the public health—rofecoxib, Merck, and the FDA. N Engl J Med. 2004;351:1707–1709
7. Berra Y, Kaplan DH You can observe a lot by watching: what I've learned about teamwork from the Yankees and life. Hoboken, NJ: John Wiley & Sons; 2008.
8. Brown JS, Kulldorff M, Chan KA, et al. Early detection of adverse drug events within population-based health networks: application of sequential testing methods. Pharmacoepidemiol Drug Saf. 2007;16:1275–1284
9. Toh S, Platt R. Is size the next big thing in epidemiology? Epidemiology. 2013;24:349–351
10. Dormuth CR, Hemmelgarn BR, Paterson JM, et al.Canadian Network for Observational Drug Effect Studies. Use of high potency statins and rates of admission for acute kidney injury: multicenter, retrospective observational analysis of administrative databases. BMJ. 2013;346:f880
11. Vogt TM, Elston-Lafata J, Tolsma D, Greene SM. The role of research in integrated healthcare systems: the HMO Research Network. Am J Manag Care. 2004;10:643–648
12. Platt R, Carnahan RM, Brown JS, et al. The U.S. Food and Drug Administration’s Mini-Sentinel program: status and direction. Pharmacoepidemiol Drug Saf. 2012;21(suppl 1):1–8
13. Yih WK, Kulldorff M, Fireman BH, et al. Active surveillance for adverse events: the experience of the Vaccine Safety Datalink project. Pediatrics. 2011;127(suppl 1):S54–S64
14. Gagne JJ, Glynn RJ, Rassen JA, et al. Active safety monitoring of newly marketed medications in a distributed data network: application of a semi-automated monitoring system. Clin Pharmacol Ther. 2012;92:80–86
15. Gagne JJ, Rassen JA, Walker AM, Glynn RJ, Schneeweiss S. Active safety monitoring of new medical products using electronic healthcare data: selecting alerting rules. Epidemiology. 2012;23:238–246
16. Resnic FS, Gross TP, Marinac-Dabic D, et al. Automated surveillance to detect postprocedure safety signals of approved cardiovascular devices. JAMA. 2010;304:2019–2027
17. Maro JC, Brown JS, Kulldorff M.. Medical product safety surveillance: how many databases to use? Epidemiology. 2013;24:692–699
    © 2013 by Lippincott Williams & Wilkins, Inc