The National Childhood Vaccine Injury Act of 1986 mandated physician reporting of certain vaccine adverse events to the Secretary of the Department of Health and Human Services. The Act also led to the creation of the Vaccine Adverse Events Reporting System (VAERS), a unified national system for reporting possible adverse reactions after any US-licensed vaccine. 1 VAERS became operational in July 1990 and is operated jointly by the CDC and the Food and Drug Administration (FDA). As part of postlicensure vaccine safety surveillance, VAERS accepts voluntary reports from health care providers, parents, patients or anyone else, as well as a smaller number of mandated reports of specific events. 2–4 Through the end of 2002, the VAERS database contains ≈140 000 reports. In the 11-year period from 1991 to 2001, at least 1.9 billion doses of vaccines were distributed in the US. 5
The primary function of VAERS is to detect early warning signals and generate hypotheses about possible new vaccine adverse events or changes in frequency of known ones. This is necessary once a vaccine is licensed and put into general use. Clinical trials of new vaccines have typically involved a relatively small number of individuals (usually fewer than 10 000) and thus cannot usually detect uncommon adverse events. 6, 7 Several published summaries 8–12 of VAERS reports have provided reassuring information about the safety profile of newly licensed vaccines.
VAERS data have always been available to the public, most recently through a database of case reports with personal identifiers removed that is posted to the VAERS website (www.vaers.org) and regularly updated. Unidentified VAERS reports also continue to be available through the Freedom of Information Act (FOIA). Both FOIA responses and the public use data-set are accompanied by information that explains the strengths and weaknesses of VAERS.
Coincident with increased public interest in vaccine safety and the ready availability of VAERS data, several articles have recently been published based on the publicly available data. The primary aims of this article are to help clinicians understand the strengths and weaknesses of VAERS data as well as to illustrate scientific uses of the database and common pitfalls that may confront the unwary investigator.
DESCRIPTION OF VAERS
VAERS is a passive surveillance, or spontaneous reporting, system. Passive surveillance systems rely on health care professionals (or vaccinees) to voluntarily submit reports of illness after vaccination. Limitations of passive surveillance systems include variability in reporting standards, reporter bias, potentially significant underreporting and inability to assess causality of reports. 13 Physicians, other vaccine providers and patients/parents are encouraged to report adverse events after immunizations, whether or not they believe that the vaccination was the cause. Manufacturers are required to report all adverse events of which they become aware. 14 It is important to understand that submissions to VAERS are not formal case reports, but rather nonstandardized descriptions of symptoms and signs temporally associated with a vaccination or vaccinations. The information in a report is not necessarily complete, nor is it verified in most cases. Reports of special interest, including all deaths and hospitalizations, are followed up by epidemiologists at CDC or FDA. Follow-up often yields important information. For example investigation of reported deaths determined that the cause of death was significantly different from what originally was stated on the VAERS report in 24% of the cases. 4 In one case the vaccinee had in fact not died.
The events described in a report are coded according to Coding symbols for a thesaurus of adverse reaction terms (COSTART), 15 a medical lexicon that includes ≈1300 terms, a relatively small number compared with other medical coding systems. Each report may be assigned several COSTART terms. Coding facilitates searching for reports of particular types of events and may provide a general profile of events associated with a specific vaccine or vaccine lot. However, the coding process is inherently imprecise because it depends on the reporter’s description of the event, which may include symptoms, diagnoses or both. An important use of COSTART is to permit an automated search for related cases. A complementary strategy is to use computer software that searches for specific words in the text of the report. Because of the potential for misclassification, the reports retrieved by a COSTART search must be manually reviewed to determine which reports met the criteria or case definition in question. Pless 16 described in detail how a COSTART term search may retrieve a heterogeneous mixture of reports, some of which bear little resemblance to the search term used. A larger, more precise coding system called MedDRA (Medical dictionary for regulatory activities) may soon replace COSTART in VAERS. 17
Reporters to VAERS are asked to denote whether the adverse event led to hospitalization, prolongation of hospitalization, congenital anomaly, life-threatening illness, death or any other event that could lead to any of these outcomes in the absence of medical intervention. These events are classified as “serious” according to a regulatory definition. 14 Some degree of misclassification is inherent in this serious/nonserious distinction. For example reports of injection site reaction generally are not of great clinical significance but may be classified as serious if they result in a brief hospitalization. Conversely some reports of events that are clearly medically important, such as Bell’s palsy, may not be classified as serious according to the schema described above, because they involve only outpatient medical care.
The primary strengths of VAERS include its national scope and its ability to detect rare events in a cost-effective and timely manner. 3 Because VAERS is a centralized reporting system, isolated rare events that would ordinarily escape attention may become apparent in VAERS. The system also permits lot-specific monitoring of vaccine safety, although lot information was missing in 12 to 20% of reports between 1991 and 1997 18 and is likely to be inaccurate in others. Lot safety monitoring is accomplished primarily via weekly review of serious reports by FDA medical officers. Any apparent clustering of similar events by lot is investigated in detail. Concerns about lot-specific adverse events reported by clinicians or others to either FDA or CDC are also investigated using the VAERS database. Since the inception of VAERS, no lots of vaccine have been recalled because of safety concerns (www.fda.gov/cber/recalls.htm).
Because VAERS, along with all other passive surveillance systems, depends on voluntary submissions, underreporting of events represents an inherent system limitation. 13 The term “reporting efficiency” describes the proportion of occurrences of a specific type of event after administration of a particular vaccine that are actually reported to VAERS. Rosenthal and Chen have estimated that reporting efficiency may be as high as 72% for oral polio vaccine-associated poliomyelitis and as low as 1% for rashes occurring after measles-mumps-rubella vaccine (MMR). Among vaccine-adverse event combinations that have been studied, serious events and events occurring sooner after vaccination are more likely to be reported. 19 A capture-recapture study by Verstraeten et al. 20 found that reporting efficiency of intussusception after rotavirus vaccine approached 50%, although reports in the medical literature as well as in the media undoubtedly stimulated reporting in this instance. This highlights the need for taking into account the effect of publicity on reporting when analyzing VAERS data.
Until recently, standardized case definitions for vaccine adverse events have not been available. This has hindered meaningful interpretation of VAERS reports, because case definitions allow for standardized report review and classification. The first systematic effort to develop a case definition for a vaccine adverse event was a workshop held to review hypotonic hyporesponsive episodes after vaccination. 21 Application of case definitions for encephalopathy, encephalitis and multiple sclerosis to VAERS reports has further shown the value of case definition development. 22 More recently the Brighton Collaboration was initiated to systematically develop globally accepted case definitions for vaccine adverse events. These definitions provide criteria for degrees of diagnostic certainty. Six such definitions have been completed and can be viewed at brightoncollaboration.org/en/index.html. VAERS reports as submitted rarely contain enough information to provide a high degree of diagnostic certainty. Nevertheless the availability and dissemination of the Brighton definitions are anticipated to result not only in higher quality reports but also in more rigorous analysis of VAERS data.
USES AND IMPACT OF VAERS
VAERS has demonstrated its ability to detect rare adverse events that were either not found or incompletely understood during prelicensure clinical trials. The system rapidly detected reports of intussusception cases associated with rotavirus vaccine, 23, 24 leading to the voluntary withdrawal of the vaccine. Enhanced passive surveillance detected rare cases of viscerotropic and neurotropic disease after yellow fever vaccine, allowing them to be more fully characterized. 25 During the recent resumption of smallpox vaccination, cases of myopericarditis and ischemic cardiac events among smallpox vaccine recipients have been reported to VAERS. 26 Although a causal association between smallpox vaccine and cardiac events has not been definitively established, the Advisory Committee on Immunization Practices now recommends that persons with physician-diagnosed cardiac disease and risk factors be excluded from smallpox vaccination.
Detection of increases or decreases in known adverse events is another important goal of vaccine safety surveillance. VAERS has documented the disappearance of vaccine-associated paralytic polio after removal of oral polio vaccine from the recommended immunization schedule, with no cases reported since 1997. 5 An improved safety profile of diphtheria-tetanus toxoids-acellular pertussis vaccines in infants was documented based on review of VAERS data from 1995 to 1998. 10 Conversely an increase in reports to VAERS of Guillain-Barré syndrome after influenza vaccine during the 1993 to 1994 flu season was found in a controlled follow-up study to represent a significant although very small (1 per million persons vaccinated) attributable risk. 27
An additional objective of VAERS is identification of potential risk factors for vaccine adverse events. Advanced age has been identified as a risk factor for illness after yellow fever vaccine. 28 Syncope after vaccination has been spontaneously reported most commonly in adolescents and young adults 29; rarely severe outcomes including head injury have occurred. Partly in response to this finding, the General Recommendations on Immunization and the American Academy of Pediatrics now suggest a 15-min waiting period post-vaccination. 30, 31
PITFALLS OF VAERS
Several articles have been published since 2000 based on publicly available VAERS data. 32–36 These papers frequently fail to acknowledge important limitations of the system. In several cases, despite citing references that clearly outline the weaknesses of passive surveillance systems, some authors apparently have chosen to ignore them. 32–34 When these drawbacks of VAERS are taken into account, many of the conclusions of these studies appear unsupportable.
In some media reports and on some websites on the internet, VAERS reports are misrepresented as verified cases of vaccine-caused deaths and injuries. 37 Statements such as these exemplify the primary misconception about the VAERS surveillance system. Instructions to health care providers and consumers regarding reporting state clearly that VAERS is interested in receiving reports even if causal relation to vaccination is uncertain, consistent with the purpose of VAERS in generating, not testing, hypotheses. An established causal association between a vaccine and an adverse event is not required for reporting; a temporal sequence, meaning that a vaccine was given before the onset of the adverse event and the reporter suspected an association, is enough. Whereas some events reported to VAERS are likely caused by vaccines (for example, injection site reactions and many instances of fever), others may be related to an underlying disease or condition or to drugs being taken concurrently or may occur by chance shortly after a vaccine was administered. 38, 39 Because VAERS does not collect information on the incidence of adverse events in unvaccinated persons, the data almost never can be used to answer the question of whether specific medical outcomes occur more frequently in vaccinated individuals than would be expected in the general population. Many of the reports to VAERS represent purely temporal association. Assessment of causality requires additional investigation.
Except for some obvious situations such as injection site reactions, it is crucial to appreciate that the data in VAERS cannot support a determination of whether a vaccine was more likely than not to have caused an adverse event. Individual reports rarely may suggest a cause and effect relationship if there is a known association based on epidemiologic studies or compelling laboratory evidence presented in the report, such as the finding of polio vaccine virus in vaccine-associated paralytic poliomyelitis. 40 Potential concerns raised by VAERS are often investigated through the Vaccine Safety Datalink, a large-linked database that includes information on more than 6 million people, allowing for timely investigations of hypotheses through selection of unvaccinated controls or other comparison groups. 41
A second misunderstanding relates to case reporting requirements. There appears to be a widespread, but mistaken, perception that all adverse reactions are reported to VAERS. 42 Only selected adverse events, as specified in the Reportable Events Table, 43 are required by law to be reported by vaccine providers. The adverse events listed in the table have been shown to be potentially related to vaccination and therefore may be compensable through the Vaccine Injury Compensation Program (www.hrsa.gov/osp/vicp/) in the absence of an alternate cause.
Therefore because VAERS functions primarily as a voluntary reporting system, reporting occurs for only a proportion of suspected adverse events, and this proportion varies depending on the vaccine and the type of event. 19 For example new vaccines generally have higher reporting rates than older vaccines. This is likely because of provider unfamiliarity with new product side effect profiles. 5, 44 The number of reported adverse events is also highly influenced by publicity. In Canada reports of a mild allergic type reaction after influenza vaccine resulted in a 3- to 5-fold increase in reports of all types of adverse events of influenza vaccine compared with the previous year. 45 Thus reports to VAERS of a particular adverse event may be the result of reporters having heard about it in the media, independent of any possible causal relationship between the vaccine and the event in question.
A third challenge in analyzing VAERS data stems from the coding terms (COSTART) assigned to a particular report. The term may represent the actual diagnosis, or merely an accompanying symptom that may be related or coincident to the underlying diagnosis. Most reports are assigned multiple coding terms, some of which may represent only possible differential diagnoses or even conditions considered to have been ruled out. When vaccines are given in combination, VAERS data may be subject to further misinterpretation. For example reports of localized, injection site skin reactions after oral vaccines are caused in fact by one of the injectable vaccines administered at the same medical visit.
A fourth issue with analysis of VAERS data is the calculation of incidence rates. Because the VAERS database does not receive complete reporting of all adverse events and because many events it contains lack confirmed diagnoses and/or cannot be attributed to vaccines, VAERS cannot be used to calculate the incidence of adverse reactions after vaccination. At best, “reporting rates” (reports/doses of vaccine distributed) could be calculated. Some studies have attempted to calculate incidence rates of adverse events with VAERS data along with the Biologics Surveillance Summaries 5, 46 that provide information on vaccine doses distributed in the United States. The most important limitation of the Biologics Surveillance Summaries is that they do not permit determination of the number of persons who actually received vaccine. Other important unknown variables include whether all reported doses were administered, whether they were given to separate individuals (unlikely because many vaccines require multiple doses to be administered to the same person), and how many individuals received multiple doses of a vaccine during a particular year. Further the Biologics Surveillance data do not contain information about distribution of vaccine by age group or birth cohort. For instance it is not possible to identify how many doses of hepatitis B vaccine were given to adults relative to the number given to infants and children. Biologics Surveillance Summaries also do not provide manufacturer or lot-specific product distribution data. The exact number of individuals who receive a particular lot of a vaccine is likewise unknown. 39 True denominator data of the number of individuals who received specific vaccines or vaccine combinations should become increasingly available as state immunization registries make continued progress. 47
Attempts to use the VAERS data to calculate internal “relative risks” of specific adverse events for a vaccine, using reports for another vaccine as a “control” group, 33 raise a fifth methodologic issue. Relative risks represent a ratio of incidence rates, and incidence rates cannot be calculated from VAERS data as previously discussed. Relative reporting rates might be calculated, but elevated relative reporting rates calculated from VAERS data may be spurious, regardless of the results of statistical significance testing. An excellent example of this problem is the observation of an increased reporting rate in VAERS for one hepatitis B vaccine brand compared with another. 48 This was appropriately considered a hypothesis for further investigation rather than as proof of one brand’s being more reactogenic than the other. On further investigation in a rigorous, controlled epidemiologic study, the VAERS-generated hypothesis of a brand difference in reactogenicity was not confirmed. 48
Comparisons of reported adverse events between vaccines may also be faulty because vaccines given within particular age ranges are sometimes associated with adverse events that follow background risk of illness in that age group. For example the ages at which health care providers are vaccinated against hepatitis B are within the range when multiple sclerosis and related demyelinating diseases are most likely to be diagnosed. Multiple studies have failed to demonstrate an association between multiple sclerosis and hepatitis B vaccination. 40, 49 Similarly infants are most prone to sudden infant death syndrome (SIDS) at precisely the ages at which they receive their primary vaccines (2 to 6 months of age). Careful study has failed to detect any association of SIDS with vaccination. 50, 51 A decrease in SIDS) reports to VAERS occurred in parallel to the decline in SIDS after the “Back to Sleep” campaign. 52
Relative reporting rates from VAERS should not be confused with data-mining methods that attempt to identify adverse events reported more commonly after one vaccine (or group of vaccines) than after others. Three data-mining methods being applied increasingly to medical product safety data are the proportional reporting rate ratio (PRR), 53, 54 empiric Bayesian 55, 56 and neural network approaches. 57, 58 The simplest and most intuitive method is calculation of the PRR, which compares proportions of events for a given vaccine with proportions for another vaccine or for a group of vaccines. An event with a higher proportion for the study vaccine than for other vaccines might be considered a “signal” and require further study. 53, 54 The other methods calculate a similar measure of “disproportionality” of vaccine-adverse event pairs but adjust them for features of the data, such as a small number of reports of a particular vaccine-adverse event pair. 55–58 Just as with relative reporting rate interpretation, PRR and other data-mining statistics can be biased by differences in usage and reporting of adverse events; thus elevated data-mining statistics do not necessarily reflect a causal relationship between a vaccine and an adverse event. PRR and other data-mining statistics should not be interpreted or presented as relative risks of specific vaccine adverse events. Such statistics should be used only as a hypothesis generation tool and are evaluated in the same manner as other hypotheses generated by VAERS.
SUMMARY OF LIMITATIONS OF VAERS
In contrast to investigators in clinical trials, who use standardized data collection methods for adverse event reporting, reporters to VAERS submit information that varies considerably in quality. Information on individual reports or report groupings cannot be assumed to be definitive or complete because there are no minimum reporting standards for information quality. The causal relationship between the signs and symptoms reported after administration of any vaccine is likewise unknown, because a known or probable association is not required for reporting. The true incidence of an adverse event cannot be approximated from VAERS data because of a combination of underreporting, unreliable “diagnoses” and lack of data on numbers of persons vaccinated. Consequently relative risks for specific events also cannot be calculated. These pitfalls are summarized also in Table 1.
EVALUATION OF VAERS REPORTS
VAERS data have been evaluated by classical epidemiologic approaches that combine descriptive epidemiology with medical judgment. 59 Data-mining techniques have been tested as signal detection methods 53–58 and seem especially promising as methods for identifying vaccine safety signals in need of further evaluation.
In observational studies that compare two groups, P values are commonly used as measures of consistency of observed differences with an assumption of no association. Confidence intervals denote the range of values for the population level measure of association that are expected by chance. The numerous acknowledged limitations of VAERS including underreporting, selective reporting, lack of a control group, inadequate denominator data to calculate event rates and diagnostic uncertainty of events result in the presence of strong biases in VAERS data. Although the effect of confounding can often be reduced through statistical adjustment, bias is very difficult to quantify or control in nonexperimental studies, 60 especially in analysis of VAERS data, and is likely to be a far greater source of uncertainty than the effect of chance. The application of statistical significance tests or calculation of confidence intervals for VAERS data should therefore not be undertaken routinely and when used should be approached and reported with extreme caution. 61
Regardless of the method used, interpretation of vaccine-adverse event combinations that are identified as possible signals must take into account biases in reporting, misclassification of reports that occurs with adverse event coding systems, the current state of medical knowledge about the disorders reported to VAERS and other previously discussed limitations of passive surveillance systems. Signals should be subjected to further clinical and descriptive epidemiologic analysis, just as are case reports and case series of spontaneous reports. Confirmation in a controlled study is usually required. Signals identified through VAERS may fail to be validated in rigorous follow-up studies. 48 The Vaccine Safety Datalink, Clinical Immunization Safety Assessment Centers 62 and Phase IV studies 63 are possible routes for hypothesis testing.
In spite of limitations shared with other passive surveillance systems, VAERS data have proved to be very useful in ongoing vaccine safety monitoring. An understanding of the defined objectives and inherent drawbacks of the system is vital to the effective use of VAERS data in vaccine safety investigations. Studies that fail to respect the limits of the system risk arriving at conclusions that cannot be supported or reproduced.
We thank Susan Ellenberg, Ph.D., Larry K. Pickering, M.D., Benjamin Schwartz, M.D. and Susan Chu, Ph.D., for their review of the manuscript.
1. National Childhood Vaccine Injury Act of 1986, at Section 2125 of the Public Health Service Act as codified at 42 U.S.C. Section 300aa–26.
2. Chen RT, Rastogi SC, Mullen JR, et al. The Vaccine Adverse Event Reporting System (VAERS). Vaccine 1994;12:542–50.
3. Singleton JA, Lloyd JL, Mootrey GT, Salive ME, Chen RT. An overview of the Vaccine Adverse Event Reporting System (VAERS) as a surveillance system. Vaccine 1999;17:2908–17.
4. Varricchio F. The Vaccine Adverse Event Reporting System. J Toxicol Clin Toxicol 1998;36:765–8.
5. Zhou W, Pool V, Iskander J, et al. Surveillance for safety after immunization: Vaccine Adverse Event Reporting System (VAERS): United States, 1991–2001. In: CDC surveillance summaries (January 24). MMWR 2003;52(SS-1).
6. Ellenberg SS. Safety considerations for new vaccine development. Pharmacoepidemiol Drug Safety 2001;10:411–15.
7. Fritzell B. Detection of adverse events: what are the current sensitivity limits during clinical development. Vaccine 2001;20:S47–8.
8. Braun MM, Ellenberg SS. Descriptive epidemiology of adverse events after immunization: reports to the Vaccine Adverse Event Reporting System. J Pediatr 1997;131:529–35.
9. Wise RP, Salive ME, Braun MM, et al. Postlicensure safety surveillance for varicella vaccine. JAMA 2000;284:1271–9.
10. Braun MM, Mootrey GT, Salive ME, Chen RT, Ellenberg SS, and the VAERS Working Group. Infant immunization with acellular pertussis vaccine in the US: assessment of the first two years’ data from the Vaccine Adverse Event Reporting System. Pediatrics 2000;106:E51. Available at http://pediatrics.aappublications.org/cgi/reprint/106/4/e51.pdf
11. Niu MT, Davis DM, Ellenberg S. Recombinant hepatitis B vaccination of neonates and infants: emerging safety data from the Vaccine Adverse Event Reporting System. Pediatr Infect Dis J 1996;15:771–6.
12. Niu MT, Salive M, Krueger C, Ellenberg SS. Two-year review of hepatitis A vaccine safety: data from the Vaccine Adverse Event Reporting System (VAERS). Clin Infect Dis 1998;26:1475–6.
13. Teutsch SM, Churchill RE, eds. Principles and practice of public health surveillance. Oxford, UK: Oxford University Press, 2000.
14. U. S. Code of Federal Regulations, 21CFR600.80. Postmarketing reporting of adverse experiences. Available at www.accessdata.fda.gov/scripts/cdrh/cfdocs/cfcfr/CFRSearch
15. Coding symbols for thesaurus of adverse reaction terms (COSTART). 5th ed. Rockville, MD: US Food and Drug Administration, Center for Drug Evaluation and Research, 1995.
16. Pless R. DT, DTP, DTP-HIB vaccines: skin striae response. WHO Signal September 2001:20–2.
17. Medical dictionary for regulatory activities terminology (MedDRA) Version 5.1. International Conference on Harmonization of Technical Requirements for Registration of Pharmaceuticals for Human Use (ICH), September 2002.
18. Vaccine Identification Standards Initiative. Accessed at National Immunization Program Website 06/29/03. Available at www.cdc.gov/nip/visi/images/vaersmissing.jpg
19. Rosenthal S, Chen RT. Reporting sensitivities of two passive surveillance systems for vaccine adverse events. Am J Public Health 1995;85:1706–9.
20. Verstraeten T, Baughman AL, Cadwell B, Zanardi L, Haber P, Chen RT, Vaccine Adverse Event Reporting System Team. Enhancing vaccine safety surveillance: a capture-recapture analysis of intussusception after rotavirus vaccination. Am J Epidemiol 2001;154:1006–12.
21. Braun MM, Terracciano G, Salive ME, et al. Report of a US Public Health Service workshop on hypotonic-hyporesponsive episode (HHE) after pertussis immunization. Pediatrics 1998;102. URL: http://www.pediatrics.org/cgi/content/full/102/5/e52
22. Ball R, Halsey N, Braun MM, et al. Development of case definitions for acute encephalopathy, encephalitis, and multiple sclerosis reports to the Vaccine Adverse Event Reporting System. J Clin Epidemiol 2002;55:821–30.
23. CDC. Intussusception among recipients of rotavirus vaccine: United States, 1998 –1999. MMWR 1999;48:577–81.
24. Zanardi LR, Haber P, Mootrey GT, Niu MT, Wharton ME. Intussusception among recipients of rotavirus vaccine: reports to the Vaccine Adverse Event Reporting System. Pediatrics 2001;107:E97. Available at http://www.pediatrics.org/cgi/content/full/107/6/e97
25. CDC. Adverse events associated with 17D-derived yellow fever vaccination: United States, 2001–2002. MMWR 2002;51:989–93.
26. CDC. Cardiac adverse events following smallpox vaccination: United States, 2003. MMWR 2003;52:248–50.
27. Lasky T, Terracciano GJ, Magder L, et al. The Guillain-Barré syndrome and the 1992–1993 and 1993–1994 influenza vaccines. N Engl J Med 1998;339:1797–802.
28. Martin M, Weld LH, Tsai TF, et al. Advanced age a risk factor for illness temporally associated with yellow fever vaccination. Emerg Infect Dis 2001;7:945–51.
29. Braun MM, Patriarca PA, Ellenberg SS. Syncope after immunization. Arch Pediatr Adolescent Med 1997;151:255–9.
30. CDC. General recommendations on immunization: recommendations of the Advisory Committee on Immunization Practices and the American Academy of Family Physicians. MMWR 2002;51(RR-2):14.
31. American Academy of Pediatrics. Vaccine Administration. In: Pickering LK, ed. Red Book: 2003 Report of the Committee on Infectious Diseases. 26th ed. Elk Grove Village, IL: American Academy of Pediatrics;2003:17.
32. Geier MR, Geier DA. Immunologic reactions and hepatitis B vaccine. Ann Intern Med 2001;134:1185.
33. Geier MR, Geier DA. Hepatitis B vaccination safety. Ann Pharmacother 2002;36:370–4.
34. Geier MR, Geier DA. Thimerosal in childhood vaccines, neurodevelopment disorders, and heart disease in the United States. J Am Physicians Surgeons 2003;8:6–11.
35. Nass M. The anthrax vaccine program: an analysis of the CDC’s recommendations for vaccine use. Am J Public Health 2002;92:715–21.
36. Singhal P. DT, DTP, DTP-HIB: skin striae. WHO Signal, September 2001:19.
37. Vaccine Adverse Event Reporting System (VAERS), “Hot Lots”. National Vaccine Information Center Website. Available on June 25, 2003 at www.nvic.org/Issues/hotlots.htm
38. Chen RT, Hibbs B. Vaccine safety: current and future challenges. Pediatr Ann 1998;27:445–64.
39. Ellenberg SS, Chen RT. The complicated task of monitoring vaccine safety. Public Health Rep 1997;112:10–19.
40. Halsey N. The science of evaluation of adverse events associated with vaccination. Semin Pediatr Infect Dis 2002;13:205–14.
41. Chen RT, Glasser JW, Rhodes PH, et al. The Vaccine Safety Datalink Project: a new tool for improving vaccine safety monitoring in the United States. Pediatrics 1997;99:765–73.
42. Geier DA, Geier MR. A review of hepatitis B vaccination. Expert Opin Drug Safety 2003;2:113–22.
43. CDC, Food and Drug Administration. Vaccine Adverse Reporting System: updated reportable events table. Available at http://www.vaers.org/reportable.htm
44. Wallenstein EJ, Fife D. Temporal patterns of NSAID spontaneous adverse event reports: The Weber effect revisited. Drug Safety 2001;24:233–7.
45. Bureau of Infectious Diseases. Oculo-respiratory syndrome in association with the influenza vaccine: Canada, October–November 2000. Can Commun Dis Rep 2000;26:1–2.
46. CDC. Biologics Surveillance 1991–1995. Report No. 94. Washington, DC: US Department of Health and Human Services.
47. Centers for Disease Control and Prevention. Immunization Registry Progress: United States, 2002. MMWR 2002;51:760–2.
48. Niu MT, Rhodes P, Salive M, et al. Comparative safety data of two recombinant hepatitis B vaccines in children: data from the Vaccine Adverse Event Reporting System (VAERS) and Vaccine Safety Datalink (VSD). J Clin Epidemiol 1998;51:503–10.
49. DeStefano F, Verstraeten T, Jackson L, et al. for the Vaccine Safety Datalink Research Group. Vaccinations and risk of central nervous system demyelinating diseases in adults. Arch Neurol 2003;60:504–9.
50. Fleming PJ, Blair PS, Platt MW, et al. The UK accelerated immunization programme and sudden unexpected death in infancy: case-control study. BMJ 2001;322:822–5.
51. Stratton K, Almario DA, Wizeman TM, McCormick MC, eds. Immunization Safety Review Committee. Immunization safety review: vaccinations and sudden unexpected death in infancy. Washington, DC: Institute of Medicine, 2003.
52. Silvers LE, Ellenberg SS, Wise RP, Varricchio FE, Mootrey GT, Salive ME. The epidemiology of fatalities reported to the Vaccine Adverse Event Reporting System, 1990 –1997. Pharmacoepidemiol. Drug Safety 2001;10:279–85.
53. Evans SJW, Waller PC, Davis S. Use of proportional reporting ratios for signal generation from spontaneous adverse drug reaction reports. Pharmacoepidemiol Drug Safety 2001;10:483–6.
54. Haber P, Iskander J, English-Bullard R, Pool V. Use of proportional reporting rate ratio in monitoring vaccine adverse event reports, US. Pharmacoepidemiol Drug Safety 2002;11:S229.
55. Dumouchel W. Bayesian data mining in large frequency tables, with an application to the FDA spontaneous reporting system. Am Statistician 1999;53:177–90.
56. Niu MT, Erwin DE, Braun MM. Data mining in the US Vaccine Adverse Event Reporting System (VAERS): early detection of intussusception and other events after rotavirus vaccination. Vaccine 2001;19:4627–34.
57. Tiwari J, Hou C, Varricchio F, Lachenbruch P. Comparison of neural networks and logistic regression for classification of vaccine adverse events. Washington, DC: International Biometric Society, 1996.
58. Bate A, Lindquist M, Edwards IR, et al. A Bayesian neural network method for adverse drug reaction signal generation. Eur J Clin Pharmacol 1998;54:315–21.
59. Ball R. Methods for ensuring vaccine safety. Expert Rev Vaccines 2002;1:161–8.
60. Sparrow JM, Thompson JR. Bias: adding to the uncertainty. Br J Ophthalmol 1999;83:637–8.
61. Brennan P, Croft P. Interpreting the results of observational research: chance is not such a fine thing. BMJ 1994;309:727–30.
62. Pless R, Casey C, Chen R. CISA: improving the evaluation, management and understanding of adverse events possibly related to immunizations. Available at www.cdc.gov/nip/vacsafe/cisa/intro-cisa.htm
63. Plotkin SA, Orenstein W. Vaccines. 4th ed. Philadelphia: Saunders, 2003.
EDITED BY LARRY K. PICKERING, M.D.