Share this article on:

Completeness and Accuracy of the Wisconsin Immunization Registry: An Evaluation Coinciding With the Beginning of Meaningful Use

Koepke, Ruth MPH; Petit, Ashley B. MPH; Ayele, Roman A. MPH; Eickhoff, Jens C. PhD; Schauer, Stephanie L. PhD; Verdon, Matthew J. BS; Hopfensperger, Daniel J. BS; Conway, James H. MD; Davis, Jeffrey P. MD

Journal of Public Health Management & Practice: May/June 2015 - Volume 21 - Issue 3 - p 273–281
doi: 10.1097/PHH.0000000000000216
Original Articles

Context: Vaccination coverage rates can be improved through the application of complete and accurate immunization information systems (IISs).

Objective: Evaluate the completeness and accuracy of Wisconsin's IIS, the Wisconsin Immunization Registry (WIR).

Design: Cross-sectional evaluation, comparing vaccination medical records (MRs) from provider clinics with WIR records.

Participants: Medical records of patients born during 2009 were randomly selected from 251 Wisconsin clinics associated with the Vaccines for Children Program.

Main Outcome Measures: Completeness: percentage of patients with client records in the WIR, percentage of patients up-to-date (%UTD) with the 4:3:1:3:3:1:4 vaccination series, and percentage of patients' MR vaccinations matched by administration date (±10 days) and type to vaccinations documented in the WIR. Accuracy: percentages of matched vaccinations with the same administration date, same trade name (TN), and same lot number.

Results: Of the 1863 selected patient MRs, 98% (n = 1833) had WIR client records and 97% of their 30 899 vaccinations were documented in the WIR. The %UTD was 49.3% using the MR only, 76.5% using the WIR only, and 75.2% as estimated by the National Immunization Survey. Among matched vaccinations, 99% had the same administration date, 96% had the same TN, and 95% had the same lot number. Compared with patients from clinics that entered data into the WIR using data exchange from electronic health records, patients from clinics that entered data using the Web-based user interface were less likely to have client records in the WIR (odds ratio: 0.3; 95% confidence interval: 0.1-0.9) and less likely to have accurate TNs (odds ratio: 0.3; 95% confidence interval: 0.1-0.5).

Conclusions: The WIR was complete and accurate among this sample of children born during 2009 and provided a vaccination coverage assessment similar to the National Immunization Survey. Our results provide support for the expectation that meaningful use and other initiatives that increase data exchange from electronic health records to IISs will improve IIS data quality.

The objective of this article was to evaluate the completeness and accuracy of Wisconsin's IIS, the Wisconsin Immunization Registry (WIR).

Bureau of Communicable Diseases, Wisconsin Division of Public Health, Madison (Mss Koepke, Petit, and Ayele, Dr Schauer, Messrs Verdon and Hopfensperger, and Dr Davis); Department of Pediatrics, School of Medicine and Public Health, University of Wisconsin–Madison (Mss Koepke and Ayele and Dr Conway); and Department of Biostatistics and Medical Informatics, School of Medicine and Public Health, University of Wisconsin–Madison (Dr Eickhoff).

Correspondence: Ruth Koepke, MPH, Wisconsin Division of Public Health, 1 West Wilson Street, Madison, WI 53702 (Ruth.Koepke@wisconsin.gov).

Roman A. Ayele is currently affiliated with the University of Colorado-Denver School of Public Health and the University of Colorado–Denver School of Medicine, Department of Pediatrics, Denver, Colorado.

Ruth Koepke, Roman A. Ayele, Jens C. Eickhoff, and James H. Conway received funding support from Sanofi-Pasteur, Swiftwater, Pennsylvania (grant IND 8502, Study M5A16). Stephanie L. Schauer and Ashley B. Petit received funding from the Centers for Disease Control and Prevention, National Immunization Program (grant IDs H23/CCH522563 and 1H23IP000788-01).

We thank all of the vaccination providers in Wisconsin who participated in this project, Wisconsin Immunization Program Vaccines for Children site monitors for performing data collection, Wisconsin Immunization Registry staff members Matthew Jacobs and Jason Suchon for providing consultation on Wisconsin Immunization Registry data and Meaningful Use, University of Wisconsin–Department of Pediatrics staff members Stacey Moyer and Cherie Schommer for providing valuable administrative support, and Wyeth Pharmaceuticals for providing vaccine lot numbers to facilitate data cleanup activities.

Supplemental digital content is available for this article. Direct URL citation appears in the printed text and is provided in the HTML and PDF versions of this article on the journal's Web site (http://www.JPHMP.com).

The authors declare no conflicts of interest.

This is an open-access article distributed under the terms of the Creative Commons Attribution-NonCommercial-NoDerivatives 3.0 License, where it is permissible to download and share the work provided it is properly cited. The work cannot be changed in any way or used commercially.

Immunization information systems (IISs) are confidential, population-based databases that collect and consolidate vaccination histories for individuals residing within a specified geographic area. As of 2012, IISs had been established in all but 1 of the 50 US states and included an estimated 86% of US children younger than 6 years.1 Immunization information systems have been recommended as an effective tool for increasing vaccination coverage rates because of their ability to determine patient vaccination status, forecast recommended vaccinations, facilitate reminder/recall efforts, and assess population vaccination coverage.2,3 The effectiveness of an IIS, however, is dependent on its completeness and accuracy. For client vaccination forecasting, reminder/recall and population assessments to be accurate, residents of the catchment area need to have client records in the IIS, and the number of doses of vaccines received and the dates on which they were received need to be entered into the IIS in an accurate and timely manner. Furthermore, for some vaccine types, such as Haemophilus influenzae type b (Hib) and rotavirus, accurate trade names (TNs) are needed to forecast the correct number of doses needed to complete the vaccine series.4 In addition, in the event of a vaccine recall, accurate documentation in the IIS of vaccine lot numbers (LNs) facilitates the identification of affected patients and providers.

Several national initiatives may improve the completeness and accuracy of IISs.1,3 During 2011, the Centers for Medicare and Medicaid Services began providing health care providers with financial incentives to ensure the “meaningful use” (MU) of electronic health records (EHRs) by connecting EHRs with other health information systems such as IISs.5–7 Data must be exchanged electronically from EHRs to IISs by using standardized health level 7 (HL7) messages,7 which were previously demonstrated to be more timely and include more data elements than non-HL7 data exchange (DE) methods.8

Wisconsin's statewide, population-based IIS, the Wisconsin Immunization Registry (WIR), was established in 2000 by the Wisconsin Divisions of Public Health (WDPH) and Health Care Access and Accountability. The WIR is populated with client demographic information for all Wisconsin births since 1995 by the WDPH Vital Records Office. WIR receives new client demographic information (for Wisconsin residents born elsewhere or before 1995) and vaccination information for all clients through manual data entry into the Web-based WIR user interface (UI) and through electronic DE with EHRs (DEEHR) and billing systems (DEbilling). Data are received from mandatory participation by local health departments and voluntary participation by private health care providers, health maintenance organizations, Medicaid, and the Women, Infants and Children program. As a result of these multiple, overlapping data sources, the WIR may receive client and vaccination information for patients even when their vaccination providers do not submit data to the WIR. Through MU and other initiatives, the number of organizations transmitting information to the WIR has increased9 and the receipt of information via DE, including DE with HL7 messaging, has also increased.8

The objective of this evaluation was to assess the completeness and accuracy of the WIR for children born during 2009 and receiving vaccinations during 2009-2011, before and at the start of MU initiatives. In addition, we investigated provider characteristics associated with the completeness and accuracy of their patients' data in the WIR. We also describe methods employed by WDPH to improve WIR data quality.

Back to Top | Article Outline

Methods

Study Design

We conducted a cross-sectional evaluation of the WIR by selecting a random sample of patient medical records (MRs) from vaccination providers associated with the Vaccines for Children (VFC) Program10 throughout Wisconsin and comparing the vaccination histories documented in the patient MRs with the vaccination histories documented in the WIR. The University of Wisconsin–Madison Health Sciences Institutional Review Board determined that this project met criteria for exemption because it does not constitute human subjects research.

Back to Top | Article Outline

Data Collection From the WIR

Information on all WIR clients with birth dates during 2009 and vaccinations they received during 2009-2011 were extracted from the WIR on January 27, 2012. Client names, birth dates, addresses, vaccines administered, administration dates, TNs, and LNs were extracted. Vaccines administered included diphtheria-tetanus-acellular pertussis (DTaP), hepatitis A, hepatitis B (HepB), Hib, measles, mumps, rubella (MMR), pneumococcal conjugate vaccine (PCV), polio, rotavirus, and varicella. HepB birth doses were defined as administration of HepB vaccine between birth and age 3 days.11 For each vaccination, the method by which WIR received the record (via UI or DE) was extracted.

Back to Top | Article Outline

Data Collection From Medical Records

After data were extracted from the WIR, all Wisconsin clinics within multiclinic organizations (MCOs; defined as ≥3 clinics affiliated with the same organization) associated with the VFC program and all non-MCO clinics receiving VFC compliance visits during August 2012 through December 2013 were invited to participate. Local health departments were not invited to participate. Clinics that did not regularly vaccinate children younger than 4 years and clinics that reported documenting vaccination data only in the WIR (ie, clinics with no separate vaccination records from the WIR) were excluded.

From each participating clinic, a random sample of MRs from patients born during 2009 was selected using a stratified sampling scheme on the basis of the number of clinic patients born during 2009: 2 MRs were randomly selected from clinics with fewer than 10 patients, 4 MRs from clinics with 10 to 50 patients, 8 MRs from clinics with 51 to 500 patients, and 17 MRs from clinics with more than 500 patients. Simulation studies were conducted to determine the optimal allocation of MRs sampled from the 4 strata so the standard error of the estimated accuracy rate was minimized. The total number of MRs being sampled was determined on the basis of an equivalence test with a 5% significance level and equivalence margin of ±1%.

From each MR, patient information (including name, birth date, and address) and vaccination information (including vaccines administered during 2009-2011, administration dates, TNs, LNs, and whether the vaccination was administered by this clinic) were collected.

In addition, each clinic was asked whether they provided data to the WIR and, if so, which method of data entry was used (UI, DEEHR, DEbilling). The clinic-reported method of data entry into WIR (UI or DE) was verified by comparing it with the method by which WIR received vaccination data entered by the clinic during 2009-2011.

Back to Top | Article Outline

Data Analysis

For each selected patient, the WIR was searched for a matching client record by first name, last name, birth date, and, when necessary, address. For each patient with a matching client record in the WIR, the total number of doses received of each vaccine was summed using the MR only and the WIR record only. The percentages and 95% confidence intervals (CIs) of patients up-to-date with Advisory Committee on Immunization Practices (ACIP) recommended vaccinations12 by age 24 through 35 months were calculated using the MR only and the WIR record only, and compared with estimates from the National Immunization Survey (NIS) for Wisconsin children of similar age (19-35 months).13

In addition, for each patient, we attempted to match each of the vaccinations documented in the MR to vaccinations documented in the WIR record by vaccine type and administration date (±10 days).14 Then, for each patient, we compared the unmatched MR vaccinations with the unmatched WIR record vaccinations to identify any possible matches with administration dates more than 10 days apart. Possible matches were used to evaluate the ±10-day matching criterion only and were not included in the following analyses.

Percentages of MR vaccinations matched to vaccinations in the WIR record were calculated for each vaccine type. Among matched vaccinations, we compared administration dates to detect differences. Among matched DTaP, Hib, PCV, and rotavirus vaccinations with TNs available in the MR and the WIR record, we compared TNs to detect differences. Trade names were evaluated among DTaP vaccinations to assess the appropriate documentation of DTaP-containing combination vaccines. Trade names were evaluated among Hib, PCV, and rotavirus vaccinations because the TN received would impact the number of doses necessary to complete the vaccine series. Among matched vaccinations with LNs available in the MR and the WIR record, we compared LNs to detect differences. Percentages of matched vaccinations with the same administration date, same TN, and same LN were calculated by vaccine type and by the method that the vaccination data were received by WIR.

Using multivariate binomial regression, we evaluated the association of provider characteristics including clinic type (affiliated or not affiliated with an MCO), clinic size (≤50 or >50 patients born during 2009), and method of WIR data entry (UI, DEEHR, DEbilling, or no WIR data entry), with 2 measures of completeness and 1 measure of accuracy.

Completeness:

a. Proportion of clinics' patients with client records in the WIR.

b. Proportion of patients' vaccinations documented in the MR that were matched to vaccinations in the WIR.

Accuracy:

c. Proportion of patients' matched DTaP, Hib, PCV, and rotavirus vaccinations with the same TN.

Models for measures (b) and (c) were constructed in the following manner. To account for the nonindependence among vaccinations received by a patient, vaccinations received by each patient were summarized as a percentage. To account for the nonindependence among patients from the same clinic, models were adjusted for repeated measurements within clinics. To avoid misclassification of completeness and accuracy caused by transcription errors from other providers' MRs, only vaccinations administered by the clinic that provided the MR were included. Parameter coefficients were exponentiated and expressed as odds ratios (ORs) with corresponding 95% CIs. Data analyses were conducted using SAS version 9.3 (Cary, North Carolina).

Back to Top | Article Outline

Results

Of the 25 MCOs identified in Wisconsin, 3 (12%) reported documenting vaccination data only in the WIR and were excluded. Among the remaining 22 MCOs, 9 organizations (41%) with 180 clinics participated. Of the 101 non-MCO clinics receiving VFC compliance visits during the specified period, 5 (5%) did not regularly vaccinate children younger than 4 years and 7 (7%) reported documenting vaccination data only in the WIR; these clinics were excluded. Of the remaining 89 non-MCO clinics, 71 (80%) participated.

In total, 251 clinics located throughout Wisconsin (∼30% of Wisconsin VFC-associated clinics) participated and provided data on 1863 patients. Among participating clinics, 166 (92%) MCO and 38 (54%) non-MCO clinics reported using an EHR as their vaccination MR (Table 1). Most clinics (n = 242; 96%) reported providing data to the WIR. All 9 (4%) clinics that reported not providing data to WIR were non-MCO clinics. Among clinics that reported providing data to the WIR, methods of data entry included DEEHR (n = 166; 68%), DEbilling (n = 16; 7%), and UI (n = 60; 25%). DEEHR use was more commonly reported among MCO clinics than among non-MCO clinics (80% vs 34%; P < .001).

Back to Top | Article Outline

Completeness

Among the 1863 selected patients, 1833 (98%) were matched to WIR client records. For each vaccine type, the percentage of patients up-to-date with the ACIP-recommended number of doses was greater using the WIR record compared with the MR, and the percentage of patients up-to-date using the WIR record was similar to NIS estimates for Wisconsin children of similar age (Table 2).13 The percentage of patients up-to-date with the 4:3:1:3:3:1:4 vaccination series was 49.3% (95% CI: 47.0%-51.6%) using the MR only, 76.5% (74.6%-78.4%) using the WIR record only, and 75.2% (68.7%-81.7%) as estimated by the NIS (Table 2).

Among the 30 899 vaccinations documented in the selected patients' MRs, 97% were matched to vaccinations in the WIR, with little variation by vaccine type (Table 3). Among the 853 MR vaccinations not matched to WIR vaccinations, only 22 (<0.1% of all MR vaccinations) were identified as possible matches to vaccinations in the WIR, indicating the 10-day matching criterion identified most matches.

Back to Top | Article Outline

Accuracy

Among the 30 046 matched vaccinations, 99% had the same administration date in the WIR as in the MR (Table 3). HepB birth doses had the lowest percentage (91%) of vaccinations with the same date. When HepB birth doses were excluded, 99% of matched HepB vaccinations had the same administration date.

Among matched DTaP, Hib, PCV, and rotavirus vaccinations, 12 070 (68%) had TNs documented in the MR and the WIR. Of these, 96% had the same TN (Table 3). The percentage of matched vaccinations with the same TN was greater among DTaP and rotavirus than among Hib and PCV vaccinations.

Among matched vaccinations, 10 843 (36%) had LNs documented in the MR and the WIR. Of these, 95% had the same LN (Table 3).

Among matched vaccinations, the accuracies of administration dates, TNs, and LNs were highest among data received into WIR via DE with an HL7 message and lowest among data received into WIR via the UI (see the Table, Supplemental Digital Content 1, available at: http://links.lww.com/JPHMP/A120).

Back to Top | Article Outline

Provider Characteristics Associated With Completeness and Accuracy

The percentage of clinics with all of their patients having client records in the WIR was lower among clinics that entered data into WIR via the UI (83%) compared with clinics that used DEEHR (93%) or DEbilling (94%) (Table 4). After adjusting for clinic type and size, clinics that entered data via the UI were significantly less likely than clinics that used DEEHR to have client records in the WIR for their patients (odds ratio [OR]: 0.3; 95% CI: 0.1-0.9).

Among 1340 patients with vaccinations documented in the MR that were administered by the clinic, the percentage of patients with all of their MR vaccinations matched to vaccinations in the WIR was lowest (27%) among patients from clinics that reported not providing data to the WIR and highest (95%) among patients from clinics that used DEEHR (Table 4). In adjusted analyses, patients from clinics that did not provide data to the WIR were significantly less likely to have their vaccinations documented in the WIR than patients from clinics that used DEEHR (OR: 0.1; 95% CI: 0.0-0.6). Compared with patients from MCO clinics, patients from non-MCO clinics were less likely to have their vaccinations documented in the WIR.

Among 1173 patients with matched DTaP, Hib, PCV, or rotavirus vaccinations recorded in the MR as administered by the clinic, the percentage of patients with all of their matched vaccinations having the same TN was higher among patients from clinics that used DEEHR (84%) and DEbilling (86%) than among patients from clinics that used the UI (68%) or did not provide data to the WIR (56%) (Table 4). In adjusted analyses, compared with patients from clinics that used DEEHR, patients from clinics that used the UI (OR: 0.3; 95% CI: 0.1-0.5) and patients from clinics that did not provide data to the WIR (OR: 0.1; 95% CI: 0.0-0.5) were less likely to have the same TN in the WIR as in the MR. Results were similar for each vaccine (data available upon request). Compared with patients from MCO clinics with more than 50 patients, patients from MCO clinics with 50 or fewer patients were less likely to have the same TN (OR: 0.5; 95% CI: 0.3-0.8) (Table 4).

Back to Top | Article Outline

Discussion

Completeness

The WIR contained client records for almost all (98%) of the patients in this sample, reflecting a strength of the WIR in that it creates a new client record for each new birth in Wisconsin based on vital records data. In addition, the WIR contained record of 97% of the vaccinations documented in the sampled patients' MRs. Using the ACIP-recommended vaccination series as a benchmark, our results indicate that the WIR generally contained a more complete vaccination history than the MR. This finding demonstrates that the WIR is serving its purpose of consolidating vaccination histories for clients with multiple vaccination providers. In addition, the percentage of children in this sample up-to-date with the ACIP-recommended series using data from the WIR was similar to estimates from the NIS for Wisconsin children of similar age,13 suggesting that the WIR provides a vaccination coverage assessment for Wisconsin children that is similar to the current national survey assessment.

Notably, in the WIR, 14% of the matched vaccinations were missing a TN and 49% were missing an LN. Completeness of these data elements in the IIS is important for forecasting necessary doses and responding to a vaccine recall. To encourage transmission of TN and LN data, beginning in January 2015, the WIR will generate “report cards” for each provider, indicating when they have submitted vaccination data without TNs and LNs.

Back to Top | Article Outline

Accuracy

Among matched vaccinations, differences in administration dates were rare but were more common among HepB birth doses. This was expected because patient records were sampled only from clinics and not from birthing hospitals, where most HepB birth doses would have been administered. To facilitate the systematic and accurate recording of HepB birth dose information in the WIR, on January 1, 2011, the WDPH Vital Records Office began collecting and transmitting to WIR HepB birth dose information for every birth in Wisconsin.

Discrepancies in TNs were relatively rare among records of DTaP and rotavirus vaccinations, but more common among records of Hib and PCV vaccinations. During 2010, a new PCV vaccine, Prevnar13 (Wyeth Pharmaceuticals, Pearl River, New York), was introduced while Wyeth's other PCV vaccine, Prevnar, was still being used.15 These vaccines had similar TNs that could have caused confusion during documentation in the MR, and some EHRs and DE methods were not immediately updated to denote the new product. Accordingly, many Prevnar13 doses were incorrectly documented in the WIR as Prevnar. This type of discrepancy would have resulted in the WIR incorrectly recommending an additional dose of Prevnar13, potentially resulting in an unnecessary vaccination. To correct these data entry errors in the WIR, a list of Prevnar13 LNs obtained from the manufacturer was used to assign the appropriate TN in the WIR. This experience highlights the importance of quickly modifying IIS, EHRs, and DE methods with new vaccine products and educating providers regarding proper documentation of new products in the IIS. In addition, conducting regular data clean-up activities, using LNs or other available information, can be useful for ensuring that TNs are accurately documented in the IIS.

Discrepancies in LNs were somewhat common (5% of matched vaccinations with LNs available had a discrepancy) and not unexpected given LN complexity and length. Proper documentation of LNs in the MR and the WIR is necessary for efficient tracing of vaccine lots during recalls. Scanning of 2-dimensional barcodes that encode the LN, a new feature for some vaccine labels and a new functionality in the WIR, is expected to facilitate accurate documentation of LNs in IISs.16

Back to Top | Article Outline

Provider Characteristics Associated With Completeness and Accuracy

All of the patients from clinics that did not provide data to the WIR had client records and some had vaccinations documented in the WIR. However, these patients' vaccinations were less likely to be documented in the WIR and more likely to have a TN discrepancy. These findings demonstrate the WIR's ability to collect information on patients through DE with vital records, managed care, and other organizations even when patients attend a clinic that does not provide data to the WIR; however, WIR data for these patients were often of poorer quality.

Among participating clinics that provided data to the WIR, use of DEEHR, compared with data entry using the UI, was associated with a higher proportion of patients having WIR client records and greater accuracy of TNs. This difference is likely because DEEHR is an electronic, systematic data entry procedure less prone to omissions and errors than manual data entry using the UI. During a previous evaluation of the WIR, receipt of vaccination data using DEEHR with HL7 messages was noted to be more timely and contain more data elements, such as TN, than DE with non-HL7 messages.8 Although only a small proportion (2%) of the vaccinations we analyzed were received using HL7 messages, the accuracy of vaccination data received by HL7 messages was very high. Our findings underscore the importance of initiatives, such as MU, designed to increase the number of providers sharing data with an IIS and require that data are shared using DEEHR with HL7 messages. Although our evaluation was not designed to measure the effect of MU on WIR data quality, our results provide support for the expectation that data quality in the WIR and other IISs will improve as MU progresses.1,3

Our results also suggest that, independent of the method used to share data with the WIR, clinics not affiliated with MCOs and smaller clinics affiliated with MCOs would benefit from initiatives to improve WIR data quality.

Back to Top | Article Outline

Comparisons to Evaluations of Other IISs

To our knowledge, this is the first report of an evaluation of a state IIS comparing the IIS to records from many providers throughout the state, regardless of health plan or provider group. Comparisons to evaluations of other IISs are complicated by differences in methods, study time intervals, study populations, and size and rate of migration in and out of the IIS catchment area. Results of previously published studies suggest that IISs with smaller geographic catchment areas may have lower percentages of patients with client records in the IIS (49%, Houston-Harris County; 88%, Boston; 92%, Philadelphia).17–19 However, these assessments were conducted more than 4 years ago and may no longer be representative of these IISs. Nonetheless, these differences underscore the importance of communication among IISs from bordering states and cities to track vaccination histories of individuals who cross catchment area borders to receive care or move frequently within bordering catchment areas. For example, the WIR has a reciprocal DE agreement with Minnesota's IIS.

Reports of evaluations of other IISs note frequently missing18,20 and discrepant18 LN and TN information. As we observed, the Washington state evaluation noted that data regarding formulations of Hib vaccines were more likely to be discrepant than data regarding formulations of other vaccines.20 This could have resulted from the Hib shortage during 2008-2009,21,22 and subsequent changes in products used, if EHRs and DE methods were not immediately updated to reflect changes in product use.

DEEHR was previously associated with greater IIS data completeness and accuracy,18,19 whereas DEbilling was previously associated with fewer patient vaccinations documented in the IIS.19,23 While we noted that DEbilling was associated with fewer patient vaccinations documented in the WIR, in adjusted analyses DEbilling was not significantly different from DEEHR regarding completeness or accuracy.

Back to Top | Article Outline

Limitations

Because of the rapidly changing methods used to enter data into the WIR, the current findings may not be generalizable to WIR data for Wisconsin residents who received vaccinations during a different time period. We gathered information only from VFC-associated clinics, which may be more familiar with the WIR and have higher-quality WIR data than non–VFC-associated clinics. Thus, our estimates of completeness and accuracy may overestimate the completeness and accuracy of all patient records in the WIR for the 2009 birth cohort. Because participation was lower among MCOs, it is possible the observed greater number of doses recorded in the WIR among MCO clinics compared with non-MCO clinics was a result of MCOs selecting to participate on the basis of their perceived WIR data quality. Nonetheless, our findings are plausible, despite this possible selection bias, because MCOs may be more likely than non-MCOs to have systematic data entry procedures or have provider-based quality improvement initiatives. Because only a small proportion of matched vaccinations had LNs documented in the MR and the WIR, we were not able to assess provider characteristics associated with completeness and accuracy of LNs. We gathered MRs for each selected patient from one provider only; therefore, we were not able to verify the accuracy of vaccination data documented in the WIR that was not documented in the participating provider's MR. Finally, we were not able to assess completeness and accuracy among clinics that only documented vaccinations in the WIR, because they did not have an MR for comparison. Methods for measuring and ensuring the quality of data from these clinics are needed.

Back to Top | Article Outline

Conclusions

The WIR contained relatively complete and accurate client and vaccination information for this sample of Wisconsin residents who were born during 2009 and received vaccinations prior to and at the start of MU. Functionality of the WIR continues to evolve, employing diverse methods to improve data quality. Our results provide support for the expectation that MU and other initiatives that result in more providers sharing data with an IIS, and more sharing of data using DEEHR with HL7 messages, will improve IIS data quality. To ensure that data quality is improved and maintained, IIS and provider staff should monitor data quality and institute improvement initiatives. With complete and accurate data, IISs will be increasingly important to reduce missed opportunities to vaccinate, decrease administration of unnecessary doses of vaccine, and improve vaccination rates.

Back to Top | Article Outline

REFERENCES

1. Centers for Disease Control and Prevention. Progress in immunization information systems—United States, 2012. MMWR Morb Mortal Wkly Rep. 2013;62(49):1005–1008.
2. The Guide to Community Preventive Services. Increasing appropriate vaccination: immunization information systems. http://www.thecommunityguide.org/vaccines/RRimminfosystems.html. Published 2014. Accessed June 26, 2014.
3. Groom H, Hopkins DP, Pabst LJ, et al. Immunization information systems to increase vaccination rates: a community guide systematic review [published online ahead of print June 6, 2014]. J Public Health Manag Pract. doi: 10.1097/PHH.0000000000000069.
4. Akinsanya-Beysolow I. Advisory Committee on Immunization Practices recommended immunization schedules for persons aged 0 through 18 years—United States, 2014. MMWR Morb Mortal Wkly Rep. 2014;63(5):108–109.
5. Centers for Medicare and Medicaid Services. The Official Web Site for the Medicare and Medicaid Electronic Health Records (EHR) Incentive programs. https://www.cms.gov/Regulations-and-Guidance/Legislation/EHRIncentivePrograms/index.html?redirect=/EHRIncentivePrograms/. Published 2014. Accessed June 26, 2014.
6. Centers for Disease Control and Prevention. Meaningful use. http://www.cdc.gov/ehrmeaningfuluse/introduction.html. Published 2012. Accessed June 26, 2014.
7. Centers for Disease Control and Prevention. Meaningful use and immunization information systems. http://www.cdc.gov/vaccines/programs/iis/meaningful-use/index.html. Published 2012. Accessed June 26, 2014.
8. Schauer SL, Maerz TR, Verdon MJ, Hopfensperger DJ, Davis JP. The Wisconsin Immunization Registry experience: comparing real-time and batched file submissions from health care providers. WMJ. 2014;113(3):102–106.
9. Koepke R, Eickhoff JC, Ayele RA, et al. Estimating the effectiveness of tetanus-diphtheria-acellular pertussis vaccine (Tdap) for preventing pertussis: evidence of rapidly waning immunity and difference in effectiveness by Tdap brand. J Infect Dis. 2014;210(6):942–953.
10. Centers for Disease Control and Prevention. Vaccines for Children Program (VFC). http://www.cdc.gov/vaccines/programs/vfc/about/index.html. Published 2014. Accessed June 18, 2014.
11. Centers for Disease Control and Prevention. National, state, and local area vaccination coverage among children aged 19-35 months—United States, 2012. MMWR Morb Mortal Wkly Rep. 2013;62(36):733–740.
12. Centers for Disease Control and Prevention. Recommended immunization schedules for persons aged 0 through 18 years—United States, 2012. MMWR Morb Mortal Wkly Rep. 2012;61(5):1–4.
13. Centers for Disease Control and Prevention. National Immunization Survey (NIS) table data for 2012, by state. http://www.cdc.gov/vaccines/imz-managers/coverage/nis/child/data/tables-2012.html. Published 2014. Accessed June 18, 2014.
14. AIRA Modeling of Immunization Registry Operations Workgroup. Vaccination Level Deduplication in Immunization Information Systems. Atlanta, GA: American Immunization Registry Association; December 2006.
15. Centers for Disease Control and Prevention. Licensure of a 13-valent pneumococcal conjugate vaccine (PCV13) and recommendations for use among children—Advisory Committee on Immunization Practices (ACIP), 2010. MMWR Morb Mortal Wkly Rep. 2010;59(9):258–261.
16. Centers for Disease Control and Prevention. Progress in immunization information systems—United States, 2011. MMWR Morb Mortal Wkly Rep. 2013;62(3):48–51.
17. Sahni LC, Boom JA, Patel MM, et al. Use of an immunization information system to assess the effectiveness of pentavalent rotavirus vaccine in US children. Vaccine. 2010;28(38):6314–6317.
18. Mahon BE, Shea KM, Dougherty NN, Loughlin AM. Implications for registry-based vaccine effectiveness studies from an evaluation of an immunization registry: a cross-sectional study. BMC Public Health. 2008;8:160.
19. Kolasa MS, Chilkatowsky AP, Clarke KR, Lutz JP. How complete are immunization registries? The Philadelphia story. Ambul Pediatr. 2006;6(1):21–24.
20. Jackson ML, Henrikson NB, Grossman DC. Evaluating Washington State's immunization information system as a research tool. Acad Pediatr. 2014;14(1):71–76.
21. Centers for Disease Control and Prevention. Updated recommendations for use of Haemophilus influenzae type b (Hib) vaccine: reinstatement of the booster dose at ages 12-15 months. MMWR Morb Mortal Wkly Rep. 2009;58(24):673–674.
22. Centers for Disease Control and Prevention. Licensure of a Haemophilus influenzae type b (Hib) vaccine (Hiberix) and updated recommendations for use of Hib vaccine. MMWR Morb Mortal Wkly Rep. 2009;58(36):1008–1009.
23. Kolasa MS, Cherry JE, Chilkatowsky AP, Reyes DP, Lutz JP. Practice-based electronic billing systems and their impact on immunization registries. J Public Health Manag Pract. 2005;11(6):493–499.

data quality; electronic health records; immunization information systems; vaccination

Supplemental Digital Content

Back to Top | Article Outline
Copyright © 2015 Wolters Kluwer Health, Inc. All rights reserved.