Immunization information systems (IISs) are confidential, population-based databases that collect and consolidate vaccination histories for individuals residing within a specified geographic area. As of 2012, IISs had been established in all but 1 of the 50 US states and included an estimated 86% of US children younger than 6 years.1 Immunization information systems have been recommended as an effective tool for increasing vaccination coverage rates because of their ability to determine patient vaccination status, forecast recommended vaccinations, facilitate reminder/recall efforts, and assess population vaccination coverage.2,3 The effectiveness of an IIS, however, is dependent on its completeness and accuracy. For client vaccination forecasting, reminder/recall and population assessments to be accurate, residents of the catchment area need to have client records in the IIS, and the number of doses of vaccines received and the dates on which they were received need to be entered into the IIS in an accurate and timely manner. Furthermore, for some vaccine types, such as Haemophilus influenzae type b (Hib) and rotavirus, accurate trade names (TNs) are needed to forecast the correct number of doses needed to complete the vaccine series.4 In addition, in the event of a vaccine recall, accurate documentation in the IIS of vaccine lot numbers (LNs) facilitates the identification of affected patients and providers.
Several national initiatives may improve the completeness and accuracy of IISs.1,3 During 2011, the Centers for Medicare and Medicaid Services began providing health care providers with financial incentives to ensure the “meaningful use” (MU) of electronic health records (EHRs) by connecting EHRs with other health information systems such as IISs.5–7 Data must be exchanged electronically from EHRs to IISs by using standardized health level 7 (HL7) messages,7 which were previously demonstrated to be more timely and include more data elements than non-HL7 data exchange (DE) methods.8
Wisconsin's statewide, population-based IIS, the Wisconsin Immunization Registry (WIR), was established in 2000 by the Wisconsin Divisions of Public Health (WDPH) and Health Care Access and Accountability. The WIR is populated with client demographic information for all Wisconsin births since 1995 by the WDPH Vital Records Office. WIR receives new client demographic information (for Wisconsin residents born elsewhere or before 1995) and vaccination information for all clients through manual data entry into the Web-based WIR user interface (UI) and through electronic DE with EHRs (DEEHR) and billing systems (DEbilling). Data are received from mandatory participation by local health departments and voluntary participation by private health care providers, health maintenance organizations, Medicaid, and the Women, Infants and Children program. As a result of these multiple, overlapping data sources, the WIR may receive client and vaccination information for patients even when their vaccination providers do not submit data to the WIR. Through MU and other initiatives, the number of organizations transmitting information to the WIR has increased9 and the receipt of information via DE, including DE with HL7 messaging, has also increased.8
The objective of this evaluation was to assess the completeness and accuracy of the WIR for children born during 2009 and receiving vaccinations during 2009-2011, before and at the start of MU initiatives. In addition, we investigated provider characteristics associated with the completeness and accuracy of their patients' data in the WIR. We also describe methods employed by WDPH to improve WIR data quality.
We conducted a cross-sectional evaluation of the WIR by selecting a random sample of patient medical records (MRs) from vaccination providers associated with the Vaccines for Children (VFC) Program10 throughout Wisconsin and comparing the vaccination histories documented in the patient MRs with the vaccination histories documented in the WIR. The University of Wisconsin–Madison Health Sciences Institutional Review Board determined that this project met criteria for exemption because it does not constitute human subjects research.
Data Collection From the WIR
Information on all WIR clients with birth dates during 2009 and vaccinations they received during 2009-2011 were extracted from the WIR on January 27, 2012. Client names, birth dates, addresses, vaccines administered, administration dates, TNs, and LNs were extracted. Vaccines administered included diphtheria-tetanus-acellular pertussis (DTaP), hepatitis A, hepatitis B (HepB), Hib, measles, mumps, rubella (MMR), pneumococcal conjugate vaccine (PCV), polio, rotavirus, and varicella. HepB birth doses were defined as administration of HepB vaccine between birth and age 3 days.11 For each vaccination, the method by which WIR received the record (via UI or DE) was extracted.
Data Collection From Medical Records
After data were extracted from the WIR, all Wisconsin clinics within multiclinic organizations (MCOs; defined as ≥3 clinics affiliated with the same organization) associated with the VFC program and all non-MCO clinics receiving VFC compliance visits during August 2012 through December 2013 were invited to participate. Local health departments were not invited to participate. Clinics that did not regularly vaccinate children younger than 4 years and clinics that reported documenting vaccination data only in the WIR (ie, clinics with no separate vaccination records from the WIR) were excluded.
From each participating clinic, a random sample of MRs from patients born during 2009 was selected using a stratified sampling scheme on the basis of the number of clinic patients born during 2009: 2 MRs were randomly selected from clinics with fewer than 10 patients, 4 MRs from clinics with 10 to 50 patients, 8 MRs from clinics with 51 to 500 patients, and 17 MRs from clinics with more than 500 patients. Simulation studies were conducted to determine the optimal allocation of MRs sampled from the 4 strata so the standard error of the estimated accuracy rate was minimized. The total number of MRs being sampled was determined on the basis of an equivalence test with a 5% significance level and equivalence margin of ±1%.
From each MR, patient information (including name, birth date, and address) and vaccination information (including vaccines administered during 2009-2011, administration dates, TNs, LNs, and whether the vaccination was administered by this clinic) were collected.
In addition, each clinic was asked whether they provided data to the WIR and, if so, which method of data entry was used (UI, DEEHR, DEbilling). The clinic-reported method of data entry into WIR (UI or DE) was verified by comparing it with the method by which WIR received vaccination data entered by the clinic during 2009-2011.
For each selected patient, the WIR was searched for a matching client record by first name, last name, birth date, and, when necessary, address. For each patient with a matching client record in the WIR, the total number of doses received of each vaccine was summed using the MR only and the WIR record only. The percentages and 95% confidence intervals (CIs) of patients up-to-date with Advisory Committee on Immunization Practices (ACIP) recommended vaccinations12 by age 24 through 35 months were calculated using the MR only and the WIR record only, and compared with estimates from the National Immunization Survey (NIS) for Wisconsin children of similar age (19-35 months).13
In addition, for each patient, we attempted to match each of the vaccinations documented in the MR to vaccinations documented in the WIR record by vaccine type and administration date (±10 days).14 Then, for each patient, we compared the unmatched MR vaccinations with the unmatched WIR record vaccinations to identify any possible matches with administration dates more than 10 days apart. Possible matches were used to evaluate the ±10-day matching criterion only and were not included in the following analyses.
Percentages of MR vaccinations matched to vaccinations in the WIR record were calculated for each vaccine type. Among matched vaccinations, we compared administration dates to detect differences. Among matched DTaP, Hib, PCV, and rotavirus vaccinations with TNs available in the MR and the WIR record, we compared TNs to detect differences. Trade names were evaluated among DTaP vaccinations to assess the appropriate documentation of DTaP-containing combination vaccines. Trade names were evaluated among Hib, PCV, and rotavirus vaccinations because the TN received would impact the number of doses necessary to complete the vaccine series. Among matched vaccinations with LNs available in the MR and the WIR record, we compared LNs to detect differences. Percentages of matched vaccinations with the same administration date, same TN, and same LN were calculated by vaccine type and by the method that the vaccination data were received by WIR.
Using multivariate binomial regression, we evaluated the association of provider characteristics including clinic type (affiliated or not affiliated with an MCO), clinic size (≤50 or >50 patients born during 2009), and method of WIR data entry (UI, DEEHR, DEbilling, or no WIR data entry), with 2 measures of completeness and 1 measure of accuracy.
a. Proportion of clinics' patients with client records in the WIR.
b. Proportion of patients' vaccinations documented in the MR that were matched to vaccinations in the WIR.
c. Proportion of patients' matched DTaP, Hib, PCV, and rotavirus vaccinations with the same TN.
Models for measures (b) and (c) were constructed in the following manner. To account for the nonindependence among vaccinations received by a patient, vaccinations received by each patient were summarized as a percentage. To account for the nonindependence among patients from the same clinic, models were adjusted for repeated measurements within clinics. To avoid misclassification of completeness and accuracy caused by transcription errors from other providers' MRs, only vaccinations administered by the clinic that provided the MR were included. Parameter coefficients were exponentiated and expressed as odds ratios (ORs) with corresponding 95% CIs. Data analyses were conducted using SAS version 9.3 (Cary, North Carolina).
Of the 25 MCOs identified in Wisconsin, 3 (12%) reported documenting vaccination data only in the WIR and were excluded. Among the remaining 22 MCOs, 9 organizations (41%) with 180 clinics participated. Of the 101 non-MCO clinics receiving VFC compliance visits during the specified period, 5 (5%) did not regularly vaccinate children younger than 4 years and 7 (7%) reported documenting vaccination data only in the WIR; these clinics were excluded. Of the remaining 89 non-MCO clinics, 71 (80%) participated.
In total, 251 clinics located throughout Wisconsin (∼30% of Wisconsin VFC-associated clinics) participated and provided data on 1863 patients. Among participating clinics, 166 (92%) MCO and 38 (54%) non-MCO clinics reported using an EHR as their vaccination MR (Table 1). Most clinics (n = 242; 96%) reported providing data to the WIR. All 9 (4%) clinics that reported not providing data to WIR were non-MCO clinics. Among clinics that reported providing data to the WIR, methods of data entry included DEEHR (n = 166; 68%), DEbilling (n = 16; 7%), and UI (n = 60; 25%). DEEHR use was more commonly reported among MCO clinics than among non-MCO clinics (80% vs 34%; P < .001).
Among the 1863 selected patients, 1833 (98%) were matched to WIR client records. For each vaccine type, the percentage of patients up-to-date with the ACIP-recommended number of doses was greater using the WIR record compared with the MR, and the percentage of patients up-to-date using the WIR record was similar to NIS estimates for Wisconsin children of similar age (Table 2).13 The percentage of patients up-to-date with the 4:3:1:3:3:1:4 vaccination series was 49.3% (95% CI: 47.0%-51.6%) using the MR only, 76.5% (74.6%-78.4%) using the WIR record only, and 75.2% (68.7%-81.7%) as estimated by the NIS (Table 2).
Among the 30 899 vaccinations documented in the selected patients' MRs, 97% were matched to vaccinations in the WIR, with little variation by vaccine type (Table 3). Among the 853 MR vaccinations not matched to WIR vaccinations, only 22 (<0.1% of all MR vaccinations) were identified as possible matches to vaccinations in the WIR, indicating the 10-day matching criterion identified most matches.
Among the 30 046 matched vaccinations, 99% had the same administration date in the WIR as in the MR (Table 3). HepB birth doses had the lowest percentage (91%) of vaccinations with the same date. When HepB birth doses were excluded, 99% of matched HepB vaccinations had the same administration date.
Among matched DTaP, Hib, PCV, and rotavirus vaccinations, 12 070 (68%) had TNs documented in the MR and the WIR. Of these, 96% had the same TN (Table 3). The percentage of matched vaccinations with the same TN was greater among DTaP and rotavirus than among Hib and PCV vaccinations.
Among matched vaccinations, 10 843 (36%) had LNs documented in the MR and the WIR. Of these, 95% had the same LN (Table 3).
Among matched vaccinations, the accuracies of administration dates, TNs, and LNs were highest among data received into WIR via DE with an HL7 message and lowest among data received into WIR via the UI (see the Table, Supplemental Digital Content 1, available at: http://links.lww.com/JPHMP/A120).
Provider Characteristics Associated With Completeness and Accuracy
The percentage of clinics with all of their patients having client records in the WIR was lower among clinics that entered data into WIR via the UI (83%) compared with clinics that used DEEHR (93%) or DEbilling (94%) (Table 4). After adjusting for clinic type and size, clinics that entered data via the UI were significantly less likely than clinics that used DEEHR to have client records in the WIR for their patients (odds ratio [OR]: 0.3; 95% CI: 0.1-0.9).
Among 1340 patients with vaccinations documented in the MR that were administered by the clinic, the percentage of patients with all of their MR vaccinations matched to vaccinations in the WIR was lowest (27%) among patients from clinics that reported not providing data to the WIR and highest (95%) among patients from clinics that used DEEHR (Table 4). In adjusted analyses, patients from clinics that did not provide data to the WIR were significantly less likely to have their vaccinations documented in the WIR than patients from clinics that used DEEHR (OR: 0.1; 95% CI: 0.0-0.6). Compared with patients from MCO clinics, patients from non-MCO clinics were less likely to have their vaccinations documented in the WIR.
Among 1173 patients with matched DTaP, Hib, PCV, or rotavirus vaccinations recorded in the MR as administered by the clinic, the percentage of patients with all of their matched vaccinations having the same TN was higher among patients from clinics that used DEEHR (84%) and DEbilling (86%) than among patients from clinics that used the UI (68%) or did not provide data to the WIR (56%) (Table 4). In adjusted analyses, compared with patients from clinics that used DEEHR, patients from clinics that used the UI (OR: 0.3; 95% CI: 0.1-0.5) and patients from clinics that did not provide data to the WIR (OR: 0.1; 95% CI: 0.0-0.5) were less likely to have the same TN in the WIR as in the MR. Results were similar for each vaccine (data available upon request). Compared with patients from MCO clinics with more than 50 patients, patients from MCO clinics with 50 or fewer patients were less likely to have the same TN (OR: 0.5; 95% CI: 0.3-0.8) (Table 4).
The WIR contained client records for almost all (98%) of the patients in this sample, reflecting a strength of the WIR in that it creates a new client record for each new birth in Wisconsin based on vital records data. In addition, the WIR contained record of 97% of the vaccinations documented in the sampled patients' MRs. Using the ACIP-recommended vaccination series as a benchmark, our results indicate that the WIR generally contained a more complete vaccination history than the MR. This finding demonstrates that the WIR is serving its purpose of consolidating vaccination histories for clients with multiple vaccination providers. In addition, the percentage of children in this sample up-to-date with the ACIP-recommended series using data from the WIR was similar to estimates from the NIS for Wisconsin children of similar age,13 suggesting that the WIR provides a vaccination coverage assessment for Wisconsin children that is similar to the current national survey assessment.
Notably, in the WIR, 14% of the matched vaccinations were missing a TN and 49% were missing an LN. Completeness of these data elements in the IIS is important for forecasting necessary doses and responding to a vaccine recall. To encourage transmission of TN and LN data, beginning in January 2015, the WIR will generate “report cards” for each provider, indicating when they have submitted vaccination data without TNs and LNs.
Among matched vaccinations, differences in administration dates were rare but were more common among HepB birth doses. This was expected because patient records were sampled only from clinics and not from birthing hospitals, where most HepB birth doses would have been administered. To facilitate the systematic and accurate recording of HepB birth dose information in the WIR, on January 1, 2011, the WDPH Vital Records Office began collecting and transmitting to WIR HepB birth dose information for every birth in Wisconsin.
Discrepancies in TNs were relatively rare among records of DTaP and rotavirus vaccinations, but more common among records of Hib and PCV vaccinations. During 2010, a new PCV vaccine, Prevnar13 (Wyeth Pharmaceuticals, Pearl River, New York), was introduced while Wyeth's other PCV vaccine, Prevnar, was still being used.15 These vaccines had similar TNs that could have caused confusion during documentation in the MR, and some EHRs and DE methods were not immediately updated to denote the new product. Accordingly, many Prevnar13 doses were incorrectly documented in the WIR as Prevnar. This type of discrepancy would have resulted in the WIR incorrectly recommending an additional dose of Prevnar13, potentially resulting in an unnecessary vaccination. To correct these data entry errors in the WIR, a list of Prevnar13 LNs obtained from the manufacturer was used to assign the appropriate TN in the WIR. This experience highlights the importance of quickly modifying IIS, EHRs, and DE methods with new vaccine products and educating providers regarding proper documentation of new products in the IIS. In addition, conducting regular data clean-up activities, using LNs or other available information, can be useful for ensuring that TNs are accurately documented in the IIS.
Discrepancies in LNs were somewhat common (5% of matched vaccinations with LNs available had a discrepancy) and not unexpected given LN complexity and length. Proper documentation of LNs in the MR and the WIR is necessary for efficient tracing of vaccine lots during recalls. Scanning of 2-dimensional barcodes that encode the LN, a new feature for some vaccine labels and a new functionality in the WIR, is expected to facilitate accurate documentation of LNs in IISs.16
Provider Characteristics Associated With Completeness and Accuracy
All of the patients from clinics that did not provide data to the WIR had client records and some had vaccinations documented in the WIR. However, these patients' vaccinations were less likely to be documented in the WIR and more likely to have a TN discrepancy. These findings demonstrate the WIR's ability to collect information on patients through DE with vital records, managed care, and other organizations even when patients attend a clinic that does not provide data to the WIR; however, WIR data for these patients were often of poorer quality.
Among participating clinics that provided data to the WIR, use of DEEHR, compared with data entry using the UI, was associated with a higher proportion of patients having WIR client records and greater accuracy of TNs. This difference is likely because DEEHR is an electronic, systematic data entry procedure less prone to omissions and errors than manual data entry using the UI. During a previous evaluation of the WIR, receipt of vaccination data using DEEHR with HL7 messages was noted to be more timely and contain more data elements, such as TN, than DE with non-HL7 messages.8 Although only a small proportion (2%) of the vaccinations we analyzed were received using HL7 messages, the accuracy of vaccination data received by HL7 messages was very high. Our findings underscore the importance of initiatives, such as MU, designed to increase the number of providers sharing data with an IIS and require that data are shared using DEEHR with HL7 messages. Although our evaluation was not designed to measure the effect of MU on WIR data quality, our results provide support for the expectation that data quality in the WIR and other IISs will improve as MU progresses.1,3
Our results also suggest that, independent of the method used to share data with the WIR, clinics not affiliated with MCOs and smaller clinics affiliated with MCOs would benefit from initiatives to improve WIR data quality.
Comparisons to Evaluations of Other IISs
To our knowledge, this is the first report of an evaluation of a state IIS comparing the IIS to records from many providers throughout the state, regardless of health plan or provider group. Comparisons to evaluations of other IISs are complicated by differences in methods, study time intervals, study populations, and size and rate of migration in and out of the IIS catchment area. Results of previously published studies suggest that IISs with smaller geographic catchment areas may have lower percentages of patients with client records in the IIS (49%, Houston-Harris County; 88%, Boston; 92%, Philadelphia).17–19 However, these assessments were conducted more than 4 years ago and may no longer be representative of these IISs. Nonetheless, these differences underscore the importance of communication among IISs from bordering states and cities to track vaccination histories of individuals who cross catchment area borders to receive care or move frequently within bordering catchment areas. For example, the WIR has a reciprocal DE agreement with Minnesota's IIS.
Reports of evaluations of other IISs note frequently missing18,20 and discrepant18 LN and TN information. As we observed, the Washington state evaluation noted that data regarding formulations of Hib vaccines were more likely to be discrepant than data regarding formulations of other vaccines.20 This could have resulted from the Hib shortage during 2008-2009,21,22 and subsequent changes in products used, if EHRs and DE methods were not immediately updated to reflect changes in product use.
DEEHR was previously associated with greater IIS data completeness and accuracy,18,19 whereas DEbilling was previously associated with fewer patient vaccinations documented in the IIS.19,23 While we noted that DEbilling was associated with fewer patient vaccinations documented in the WIR, in adjusted analyses DEbilling was not significantly different from DEEHR regarding completeness or accuracy.
Because of the rapidly changing methods used to enter data into the WIR, the current findings may not be generalizable to WIR data for Wisconsin residents who received vaccinations during a different time period. We gathered information only from VFC-associated clinics, which may be more familiar with the WIR and have higher-quality WIR data than non–VFC-associated clinics. Thus, our estimates of completeness and accuracy may overestimate the completeness and accuracy of all patient records in the WIR for the 2009 birth cohort. Because participation was lower among MCOs, it is possible the observed greater number of doses recorded in the WIR among MCO clinics compared with non-MCO clinics was a result of MCOs selecting to participate on the basis of their perceived WIR data quality. Nonetheless, our findings are plausible, despite this possible selection bias, because MCOs may be more likely than non-MCOs to have systematic data entry procedures or have provider-based quality improvement initiatives. Because only a small proportion of matched vaccinations had LNs documented in the MR and the WIR, we were not able to assess provider characteristics associated with completeness and accuracy of LNs. We gathered MRs for each selected patient from one provider only; therefore, we were not able to verify the accuracy of vaccination data documented in the WIR that was not documented in the participating provider's MR. Finally, we were not able to assess completeness and accuracy among clinics that only documented vaccinations in the WIR, because they did not have an MR for comparison. Methods for measuring and ensuring the quality of data from these clinics are needed.
The WIR contained relatively complete and accurate client and vaccination information for this sample of Wisconsin residents who were born during 2009 and received vaccinations prior to and at the start of MU. Functionality of the WIR continues to evolve, employing diverse methods to improve data quality. Our results provide support for the expectation that MU and other initiatives that result in more providers sharing data with an IIS, and more sharing of data using DEEHR with HL7 messages, will improve IIS data quality. To ensure that data quality is improved and maintained, IIS and provider staff should monitor data quality and institute improvement initiatives. With complete and accurate data, IISs will be increasingly important to reduce missed opportunities to vaccinate, decrease administration of unnecessary doses of vaccine, and improve vaccination rates.