Active Surveillance of Vaccine Safety

A System to Detect Early Signs of Adverse Events

Abstract

Background: 

There currently are no population-based systems in the United States to rapidly detect adverse events after newly introduced vaccines. To evaluate the feasibility of developing such systems, we used 5 years of data from 4 health maintenance organizations within the Centers for Disease Control and Prevention (CDC) Vaccine Safety Datalink.

Methods: 

Within every year, each week's vaccinated children were followed for 4 weeks, and rates of adverse events were compared with rates among children of similar ages before the introduction of the new vaccine. We assessed risks for intussusception after rotavirus vaccination and risks for fever, seizures, and other neurologic adverse events after the change from whole cell diphtheria-tetanus-pertussis (DTPw) to acellular DTP vaccine (DTPa). We used sequential probability ratio testing, adjusted for age, sex, calendar time, season, and HMO, and with a stopping value based on the probability of an adverse event under the null hypothesis and under a preset alternative hypothesis.

Results: 

We detected an increase in intussusception after 2589 vaccine doses of rotavirus vaccine, about the same time initial reports of intussusception were made to the Vaccine Adverse Events Reporting System. Decreases in risk for fever, seizures, and other abnormal neurologic events became detectable within 12 weeks, 42 weeks, and 18 months, respectively, after the change from DTPw to DTPa.

Conclusions: 

We conclude that it is feasible to develop systems for rapid and routine population-based assessments of new vaccine safety.

ArticlePlus

Click on the links below to access all the ArticlePlus for this article.

Please note that ArticlePlus files may launch a viewer application outside of your web browser.

• https://links.lww.com/EDE/A146

Recent events in the United States have underscored the need for surveillance systems that detect adverse events as soon as possible after the introduction of new vaccines (eg, rotavirus vaccine) or the reintroduction of old vaccines to new populations (eg, smallpox vaccine).^1,2 The goal of such a monitoring system would be to speed the recognition of increased rates of adverse events after vaccination to inform public policy.

The discovery of intussusception after rhesus-rotavirus vaccine (RRV) highlighted the need for this type of system. Shortly after the introduction of RRV into clinical practice, reports of intussusception (a rare form of bowel obstruction in children) after vaccination were made to the Vaccine Adverse Events Reporting System (VAERS), operated jointly by the Centers for Disease Control (CDC) and the Food and Drug Administration (FDA).¹ Although the response to these initial reports was rapid, months elapsed before it was possible to estimate the risk associated with the vaccine, underscoring the need to enhance vaccine safety monitoring in the United States. Because VAERS is a passive system that relies on reporting from physicians and other health care providers as well as parents, rates of disease among vaccinated or unvaccinated subjects cannot be calculated. Thus, if only VAERS data are available, it is difficult to make formal causality assessments.^3,4 Additionally, prelicensure trials of the rotavirus vaccine provided only limited information about the vaccine's risk for rare events because of limited study sizes.⁵ Although concern had been raised before licensure about intussusception, no single trial had more than one vaccinated child who developed intussusception, and overall the number of vaccinated children was too small to detect the increased risk subsequently seen. In light of these issues, the FDA held a joint conference with the National Institutes of Health (NIH), the CDC, and the National Vaccine Program Office to focus on developing the capacity to more rapidly detect such adverse events after licensure of vaccines.⁶

In this current study, we used data from the Vaccine Safety Datalink to develop and test a methodology for conducting near real-time surveillance for potential vaccine adverse events. The Vaccine Safety Datalink was started in 1991 to conduct rigorous epidemiologic studies of vaccine safety in the United States and is the largest such collaborative study in the United States.³ The Vaccine Safety Datalink, coordinated by the CDC, includes data from 8 geographically diverse health maintenance organizations (HMOs) and provides information on approximately 3.5% of all children younger than the age of 6 in the United States, or 650,000 children. The Vaccine Safety Datalink has until now operated retrospectively, adding data in yearly increments. We evaluated whether the Vaccine Safety Datalink's historical data on vaccines and health-care encounters could be analyzed as if it had become available on a weekly basis to provide rapid and frequently updated assessments of vaccine safety. We then attempted retrospectively to detect the previously recognized rotavirus-intussusception signal^7,8 and the decreased risk for seizures and other abnormal neurologic events with the changeover from diphtheria-tetanus-whole cell pertussis (DTPw) to diphtheria-tetanus-acellular pertussis (DTPa) vaccine.⁹ These analyses allowed us to evaluate the capability of Vaccine Safety Datalink data to meet 3 challenges facing active surveillance: outcome definition, data management (rapid and routine creation of analytic datasets based on automated data), and statistical analysis of signal detection using methods that account for multiple testing of accumulating data.

METHODS

Overview

All Vaccine Safety Datalink HMOs have automated vaccine tracking systems that track immunizations administered to members. These systems are the initial source of the exposure data for Vaccine Safety Datalink studies. Additionally, these HMOs have the ability to capture medical care utilization by enrolled members in hospital, emergency departments and in the outpatient clinics.

Dataset Creation and Classification of Exposures

We used data the HMOs had submitted to the Vaccine Safety Datalink from 1995 through 2000. These data were created from HMO-specific administrative datasets and include information on member demographics, vaccinations administered, and medical care utilization.³ To simulate an active surveillance adverse event detection system, we segmented the data into weekly cohorts of vaccinated children, yielding 52 separate datasets for each year.

Each weekly cohort contained information about each vaccinated child's age and sex, the vaccine administered that week, and ICD9-CM diagnosis and procedure codes for the 6 weeks preceding and following the week during which immunization occurred. To protect confidentiality, subjects were assigned unique personal identifiers in each weekly cohort. As a result, a child would have a different identifier for each immunization if these occurred in different weeks. Each week's dataset could thus be used to create a count of the vaccinated children for that week, along with the number diagnosed with specific medical conditions within the 30 days after the vaccination under review.

These weekly data were partitioned into a baseline period and a surveillance period. The baseline period was defined as a period before the introduction of the new vaccine; during this time, the probability of a particular adverse event after a comparison vaccine was presumed to be stable. The surveillance period began with the introduction of a new vaccine.

For comparison of DTPw and DTPa, the baseline period was 2 years, from 1 January 1995 through 28 December 1996, when DTPw was in routine use. Surveillance for adverse events after the acellular vaccine began on 9 February 1997, allowing a 6-week transition to routine use of the acellular vaccine.

For rotavirus vaccine, we used a longer baseline period, from 1 January 1995 through 9 January 1999, because the rate of intussusception in children younger than 8 months old after vaccination with routine childhood vaccines was low. The surveillance period for intussusception following the rotavirus vaccine began on 1 January 1999 and extended through 6 November 1999. Rotavirus vaccine was withdrawn during this period, and very few vaccinations were recorded after 17 July 1999.

Classification of Outcomes

Fever, seizures, and intussusception were categorized using standard ICD-9 codes. ICD-9 codes for additional neurologic events were based on VAERS reports. (The codes we used are listed in the appendix, which is available with the electronic version of this article.) We evaluated a 3-day interval after immunization for the occurrence of fever, seizures or other neurologic conditions following DTPw and DTPa vaccine. For intussusception following rotavirus vaccine, we evaluated the rate of events in the 30-day time window after vaccination. For each outcome, a person was censored either on the date of diagnosis or on the end of the follow-up period. These time windows were selected on our understanding of the relationships between these conditions and vaccination.^7,9

Statistical Analysis

Our primary goal was to detect a change in the probabilities of adverse events after the introduction of the new rotavirus vaccine and the change from whole cell pertussis-containing DTPw to acellular pertussis-containing DTPa. Statistical analyses were performed using sequential probability ratio tests (SPRT).^10,11 SPRT has been widely used in industry to monitor process performance and is used for situations where the monitoring is continuous and items or events can be inspected one by one. Recently, the use of SPRT in medical research has received attention, particularly in regards to monitoring surgical failures or in monitoring syndromic data sets for aberrations in disease rates.^10,11 In general, SPRT detects signals earlier than Shewhart p-charts or CUSUM.¹²

To perform a SPRT analysis, specific values must be formulated for the probability of a postvaccination adverse event under the null hypothesis (p₀) and under the alternative hypothesis (p₁). The null hypothesis is that the probability of an adverse event during the surveillance period is the same as the probability of an event during the baseline period. The alternative hypothesis is that there is a difference in the probability of an event that is considered meaningful. The value of p₀ for the null hypothesis was calculated from the counts and events in the baseline period, while the value of p₁ for the alternative hypothesis was based on the effect size we wanted to detect during the surveillance period.

To assess the safety of the switch from DTPw to DTPa, we examined fever, febrile seizures, and other neurologic events among children younger than 24 months old. The rate of these adverse events after DTPw vaccination was used to establish the baseline period, and we then monitored for a 40% decrease in the probability of these events after the changeover. To evaluate intussusception risk after rotavirus vaccination, we monitored for a 10-fold increase in the probability after the introduction of rotavirus vaccination compared with the risk after other routine childhood vaccinations during the baseline period.

Our choice of threshold for rotavirus surveillance was based on the strength of the signal seen in epidemiologic studies.^7,8 The choice of threshold for the DTPa analysis was based on the effect size that we felt to be of public health importance. The ability to detect a decrease of 40% would be analogous to the ability to detect an increase of the same size, or, in this case, a 2.5-fold increase in risk.

The SPRT method involves a likelihood ratio test to evaluate the evidence in favor of both the null and alternative hypotheses. The 2 likelihoods used to form the ratio are functions of the data and either p₀ or p₁. Using each week's data, an X_t is calculated from the previous week's statistic X_t-1 plus the log of the current week's likelihood ratio. The log likelihood ratio is the log of the likelihood using the current week's data and p₁ divided by the likelihood using the current week's data and p₀.

We used the binomial likelihood function because of the grouped data and proportions. Large values of X_t favor the alternative hypothesis; small values of X_t favor the null hypothesis.

The weekly values of the pair (t,X_t) were plotted on the SPRT chart along 2 barriers. If the value of X_t crossed the upper “alternative” barrier, a decision was made to accept the alternative hypothesis, while if it crossed the lower “null” barrier, the decision was made to accept the null hypothesis. If X_t remained between the 2 barriers, no decision was made and data continued to accumulate. The upper and lower barriers are calculated as:

where α* and β* are approximately equal to the Type I and Type II errors respectively. These error levels apply to the entire SPRT process, not to each specific week, hence our analyses accounted for multiple testing. The adjustment for multiple testing is not explicit as in the Bonferroni adjustment, but rather the adjustment is incorporated into the SPRT test in the way that the upper and lower boundaries are calculated. These boundaries preserve the specified alpha and beta until a final decision is reached as to whether the hypothesis should be accepted or rejected.¹³

Although subjective, by necessity our rule depends on a stopping value (in this case, a P value of approximately < 0.05). The P value in this case is not being used to declare a statistically significant relationship between some exposure and disease, but rather is being used in the context of surveillance to trigger an alert.

To control for potential confounders such as HMO, age, calendar time, season, and sex, we used the risk-adjustment methods described in Steiner et al.¹⁴ With this method, each unique combination of risk factors forms a stratum (j), which will have its own baseline probability of an adverse event, p_oj. These probabilities are obtained by conducting a logistic regression analysis on the baseline data. The log likelihood ratio then becomes the product of the likelihoods in the various strata.

RESULTS

Descriptive Data

The analysis of DTPw and DTPa vaccines covered a total of 156 weeks. Of these, 104 weeks of data were used for the baseline period, with 212,634 DTPw vaccinations administered. There were 52 weeks of data used for the surveillance period, with 63,367 DTPa vaccinations. In the baseline period, there were 40 medically attended visits for seizures, 182 for fever, and 23 for other neurologic conditions in the 30 days after vaccination. In the surveillance period, there were 11 medically attended visits for seizures, 46 visits for fever, and 5 visits for other neurologic conditions in the 30 days following vaccination.

The analysis of rotavirus vaccine covered a total of 253 weeks. Of these, 210 weeks of data were used for the baseline period, with 996,733 routine childhood vaccinations administered. Forty-two cases of intussusception were recorded in the 30 days after any vaccination during the baseline period. There were 43 weeks of data used for the surveillance period, with 26,069 rotavirus vaccinations administered; >99% of these vaccinations were administered in the first 27 weeks of the surveillance period. Seven cases of intussusception occurred in the 30 days following rotavirus vaccination.

Change in Adverse Event Probabilities After the Introduction of DTPa Vaccine

Using the risk-adjusted SPRT, we were able to rapidly detect a 40% decrease in the clinical diagnosis of fever in the 3 days after vaccination with the changeover from whole cell pertussis vaccine to the acellular pertussis vaccine (Fig. 1). This signal triggered an alert approximately 12 weeks into the surveillance period.

FIGURE 1.: Decrease in fever after change from DTP to DTPa. (SPRT assessment for 40% decrease, adjusted for season, time, age, and HMO. Risk window 0–3 days after vaccination).

A 40% decrease in the rate of seizures after the changeover from DTPw to DTPa (Fig. 2) triggered an alert later, approximately 42 weeks after the changeover. The less frequent occurrence of febrile seizures compared with fever after vaccination contributed to the 30-week difference in the detection of these 2 signals. A decrease in other neurologic events, such as pallor or hypotonia, within the 3 days after vaccination took much longer to detect, becoming significant 1.5 years after the changeover from DTPw to DTPa (Fig. 3).

FIGURE 2.: Decrease in seizures after change from DTP to DTPa, adjusted for season, time, age, and HMO. Risk window 0–3 days after vaccination.

FIGURE 3.: Decrease in neurologic events after change from DTP to DTPa, adjusted for season, time, age, and HMO. Risk window 0–3 days after vaccination.

Change in Adverse Event Probabilities After the Introduction of Rotavirus Vaccine

A positive signal for a 10-fold increased risk of intussusception triggered an alert within 10 weeks after vaccine introduction (Fig. 4), with the second case of intussusception among vaccine recipients. It remained at this level for the remainder of the observation period following the fourth case of intussusception. Chronologically, the positive signals occurred at about the same time as the first set of intussusception reports to VAERS.¹

FIGURE 4.: Increase in intussusception after introduction of rotavirus vaccine, adjusted for season, time, age, and HMO. Risk window 0–30 days after vaccination among children younger than 8 months old.

DISCUSSION

In this study we demonstrated the value of routinely collected managed-care data for rapidly detecting predefined signals of vaccine adverse events. We were able to find both an expected signal (intussusception following RRV) and signals for situations in which an association has been suggested by prelicensure trials (the decreased risk for fever, seizures, and other neurologic events with DTPa).

A program such as this would add substantially to the current U.S. vaccine safety system by complementing VAERS and other spontaneous report systems. The specific advantages of our system include its relatively complete ascertainment from defined populations (permitting calculation of unbiased event rates) and its adjustment for confounders that might otherwise obscure important signals. Finally, the statistical methods take into account the sequential, multiple nature of testing, and the system's type I and II error rates apply to the entire process. This system can be used as an independent means to corroborate VAERs signals, and vice versa.

There are limitations to this study. We do not have other data to confirm the decreased risks for fever, seizures, and other neurologic conditions after the switch from DTPw to DTPa presented here, although our findings are consistent with smaller prelicensure studies.⁹ Also, our reliance on automated data means that we did not know which visits were for current, as opposed to past problems, or were for the evaluation of symptoms found later not to represent disease. This system relies on routinely collected automated data for signal detection; since automated data have varying degrees of accuracy in diagnostic coding, this approach should not be viewed as the final confirmatory epidemiologic investigation into potential vaccine adverse events. Our approach also is limited to conditions that develop relatively soon after vaccination and would not be suitable for investigation of conditions with a longer induction period. Additionally, our current approach does not account for any nonrandom distribution of onset intervals between vaccination date and the date of adverse event onset. Nonrandom clustering of such events might be another important signal of potential vaccine safety concern.

This study is only the first step towards the creation of a surveillance system for adverse events after vaccination. The move from proof-of-principle to an active system faces a number of challenges. Data must be made routinely available more frequently than in the past, without appreciably increasing costs or compromising patient confidentiality or data quality. The patient confidentiality requirement has likely been met, as the unit of analysis in our study contained only a week's worth of immunization data along with the attendant follow-up time, and we did not link identifiers across immunizations or across weeks. Hence, these datasets did not include any patient-specific identifiers and maintained a robust degree of patient confidentiality. However, generating and analyzing data on a weekly basis in real time—while feasible—will require further work to be done smoothly.

Another challenge is to decide which adverse events to try to monitor and the magnitude of change that is worth detecting. Although prelicensure trials can suggest possible associated adverse events, input from advisory groups such as Advisory Committee on Immunization Practices and from regulatory agencies such as the FDA will also be important. Screening for a wide variety of possible adverse events after vaccination would require careful interpretation of the results because of the inevitable number of false positive signals. Nonetheless, this active surveillance project would be flexible enough to quickly assess new signals (that might arise from VAERS), because of the availability of denominator datasets and statistical programs

Future applications of this approach will focus on surveillance of the routinely administered annual influenza vaccine (for example, to look for serious adverse events such as Guillain-Barré syndrome and other neurologic diseases), and the study of new vaccines such as the live attenuated intranasal influenza flu vaccine (licensed for healthy individuals 5–49 years) and a recently released combination DTPa-inactivated polio- hepatitis b vaccine. Active and prospective surveillance analysis of Vaccine Safety Datalink data provides a valuable, population-based early warning system to complement VAERS in the U.S. immunization safety system.¹⁵

ACKNOWLEDGMENTS

We thank Noel Weiss for his helpful comments.