Regulatory pressure has increased the need to document the quality and value of health care.1 The advent of electronic health records and their increasing penetration into medical practice has provided a means to inexpensively collect vast amounts of information. These data may be used for research, for business analysis, or to improve the quality and delivery of health care. Databases are important and widely used tools in both modern medical practice and clinical research.2 Registries such as the American College of Surgeons National Surgical Quality Improvement Program have been used to assess the organizational aspects of health care and new treatment principles in relation to outcomes and health economics.3
Recognizing the need to assist members with emerging regulatory requirements and the enormous potential of digital case information to understand and improve the quality of clinical care, the American Society of Anesthesiologists (ASA) chartered the Anesthesia Quality Institute (AQI) in 2008. The AQI is a separately incorporated 501(c)(3) public charity funded over its first 2 years with $1,750,000 from the ASA. The AQI continues to derive the bulk of its funding from the ASA (currently two-thirds of an annual operating budget of $2,500,000) to promote specific services of value to ASA members. The AQI has a 7-member Board of Directors (appointed by the ASA Board) that is responsible for governance and strategy. The ASA Chief Executive Officer is the ex-officio President of the AQI. The President and the AQI Board are responsible for hiring and annually evaluating an Executive Director, who manages day-to-day operations. Case-level data collected by the AQI are not accessible to ASA officers or staff.
The mission of the AQI is to promote quality improvement in anesthesiology. The AQI was designated as a patient safety organization by the Agency for Healthcare Research and Quality in 2011. The central activity of the AQI is creation and continued development of the National Anesthesia Clinical Outcomes Registry (NACOR), as a census registry of anesthesia practice in the United States. Reports from NACOR are primarily intended to support the continuous quality improvement and performance benchmarking at the individual, facility, practice, and national level. The purpose of this paper is to provide a technical description of how NACOR data are collected and curated, including the participant recruitment process, mechanisms for transmission of data, validation and security issues, and uses as a research tool.
Participation in NACOR is the voluntary choice of an anesthesia practice, on behalf of its members. This reflects the ownership by the practice of the administrative data that forms the bedrock of NACOR. Practice governance models vary widely in the United States and include solo and small group private practices tied to single facilities, larger groups providing contracted services at multiple facilities, very large private conglomerates serving multiple facilities in multiple states, university systems that employ physicians in a multispecialty group, health maintenance organizations such as Kaiser, and salaried public employment in the Department of Defense or Veterans Administration.4 The AQI relies on the practice leadership to determine how participation will be authorized.
After self-nomination through the AQI Web site, the participating anesthesia practices enter into a business associate agreement (BAA) with the AQI. Similar to the contracts between practices and their billing service vendors, the BAA provides legal protection for both parties and defines permissible and impermissible uses of data. The template for the AQI BAA is compliant with the requirements of the Health Insurance Portability and Accountability Act (HIPAA) and similar to those used by other national-level registries. The BAA template may be modified by the participant in accordance with the local governance or state law but must be agreed to and signed by both parties before any further activity occurs.
The cost of participation in NACOR is $1000 per independent provider in the group per year. This cost is discounted to $0 for ASA members. The intent of the discount is to promote ASA membership. Fellows, residents, anesthesiologist assistants, and nurse anesthetists working with physician anesthesiologists in the care team model are included in NACOR participation at no charge. Many participating groups have elected to enforce 100% ASA membership among their physician anesthesiologists and thus pay no additional fee to the AQI.
Participating practices designate an AQI Champion to be the single point of contact for ongoing communications. In most cases, this is the physician anesthesiologist responsible for quality improvement activities in the practice; in some cases, it is a volunteer with interest or experience in information technology. The AQI Champion receives a practice-specific login to the AQI server and instructions to complete 3 surveys on the practice portal home page. The provider survey requests information on each anesthesia team member (faculty, residents, nurse anesthetists, and anesthesiologist assistants), including board and subspecialty certification, years in practice, and part-time/full-time status. Inclusion of national provider identifier numbers for practitioners speeds this process. The facility survey requests information on the locations where the anesthesia team practices, including hospitals, surgery centers, offices, and clinics. The technology survey requests information on the health care software used by the practice, including billing software, quality capture tools, and clinical documentation systems. Completion of these surveys can be directly online or by transmitting aggregate data in spreadsheet form.
From initiation of participation to transmission of data typically takes a practice 9 months. Some practices have completed this process within a week; others have yet to send a test file years after execution of the BAA. For the 283 practices that have successfully contributed data to NACOR as of April 21, 2015, the median time from completion of the BAA to data loading is 202 days, with an interquartile range of 101 to 382 days (minimum 3 days, maximum 1370 days). The AQI provides a regular status update to all contracted participants that includes prompting for the next needed step, but there remain groups that have completed participation agreements but never progressed to providing clinical data.
Under an arrangement between the ASA and the AQI, the AQI owns the database server that houses the NACOR database, while it leases the Web, secure file transfer protocol (sFTP), reporting, and 2 testing servers from ASA. ASA provides physical protection to the servers in a secure server room at ASA headquarters, and AQI data are included in both local and daily off-site back-up. Access to the AQI database is limited to designated AQI analysts. Data are transmitted to NACOR as .xml or .csv files through encrypted secure socket-layer e-mail attachments or by direct sFTP upload to the server. The database server of AQI, which houses the NACOR database, is protected by 2 firewalls within the ASA network infrastructure. Although there have been random attempts at unauthorized access to the sFTP servers, there have been no security breaches involving NACOR data. There have also been no reported security breaches of NACOR data at the practice level or in AQI transactions with authorized researchers.
The AQI accepts any digitized data owned by the practice, beginning with universally available billing information. The AQI works with software providers (also called preferred vendors) that have developed data extracts for uploads to NACOR from their billing software, quality capture programs, or electronic clinical documentation systems.a Data extracts have been created for >36 vendors to date. AQI engineers have developed a standard Extensible Markup Language schema and have experience in working with vendors to create data extracts. The Extensible Markup Language schema is based on current standard definitions from the Centers for Medicare and Medicaid Services, Health Level Seven International, Standard Nomenclature for Medicine, and International Organization for Terminology in Anesthesiology. Format specifications are publicly available on the AQI Web site.b
Each practice begins with a test file typically including 1 month of data. Engineers at the AQI confirm the file format and correct mapping of each data element submitted, often in an iterative process involving the practice, the vendor, and the AQI. When successful transmission and mapping have been confirmed, the data are loaded to NACOR and a monthly reporting schedule established. The AQI will also request backwards reporting of available data going as far back as January 1, 2010, provided as either a series of monthly reports or as a yearly or even multiyear batch.
AQI requests reporting on every patient cared for; there is no sampling of cases. The minimum data reported by an AQI participant are the elements common to an anesthesia billing record.c From this starting point, the AQI will next seek case-specific outcome information. In some cases, this requires the practice to create these data, that is, to begin recording outcomes for each case. In other cases, the practice already has a system in place and efforts focus on digitization, mapping, and transmission to NACOR. The final stage of data transmission for participants at a high level of information technology maturity is the extraction of electronic clinical records. At present, NACOR collects administrative (billing) data from 100% of participating practices, routine quality outcome data from 32.5% of participants, and routine anesthesia information management system (AIMS) data from 4.6%. Accessing clinical data is the greatest technical challenge and requires the greatest commitment of time and effort. Extracts have been developed for 6 AIMS vendors, with others in process, but may still require substantial local customization before they can be shared from 1 participant to another using the same software. The format for AIMS data is deliberately kept in synchrony with the format created by the Multicenter Perioperative Outcomes Group, a national effort to pool AIMS data for research purposes.d
AUTHENTICATION AND VALIDATION OF DATA
Quality assurance in NACOR is an ongoing process that incorporates multiple layers of electronic and manual inspection (Fig. 1). The first challenge is technical: assuring that electronic data are completely and correctly transmitted. Test files are closely reviewed by AQI analysts as they are mapped into NACOR and again as subsequent data sets are received. Systematic data errors are referred to the practice AQI Champion for review, and files are not loaded until the practice and AQI are satisfied that data are being correctly transmitted. Transmission errors can arise even in practices that have been reporting to NACOR for some time as the result of upgrades or modifications in software; automated routines during data loading and extraction flag systematically missing or corrupt data for follow-up by the AQI analyst and practice champion.
As an additional check on data quality, data are returned to practices through an online reporting system accessible from their practice dashboard. The front page summarizes what the practice has submitted, what data are owed, and what the next steps in participation would be. Tabs provide access to dozens of prepopulated reports that display trends over time, allow segmentation by facility or provider, and provide relevant national and peer-group benchmarks. In the event of odd or unusual results in these reports, practices are encouraged to inform the AQI of the abnormality. For example, 1 practice submitted data showing that the rate of intraoperative cardiac arrests for 1 provider was 100 times higher than for colleagues. This individual-level outlier was noted by the practice AQI Champion and reported back to the AQI. The AQI analyst worked with the practice champion to identify the cause of the problem: the provider was checking the box to the right of the complication (actually the box for postoperative nausea) rather than the (correct) box to the left of the complication on a 2-column paper data form. The practice both re-educated the provider and changed its paper form. Similar AQI-practice-AQI feedback loops have identified inadvertent double submission of the same month’s data, erroneous calculations of patient age (in software that defaulted to January 1, 1900 for unknown birthdate), and failure to remove provider codes for physicians who had left the practice.
Most ongoing quality improvement of NACOR data occurs at the level of discussion between analysts and practice champions each time new data are loaded. Some improvement occurs later, for example, when a focused research query identifies specific practices as far-outliers. The data-sharing agreement between AQI and participating practices includes a provision for retrospective on-site auditing of submitted case records as well as periodic on-site reviews.
Direct identifiers of patients, providers, facilities, or practices are not maintained in NACOR but are coded by the participant. The AQI adheres to provisions of the HIPAA Privacy Rule for the federal protections of personal health information for research and health care operations. Based on personal health information parameters, NACOR accepts a limited data set defined by HIPAA as containing fields such as year of birth, date of service, and zip code but not including fields such as the patient’s name or medical record number.e Practices (and their software vendors) are encouraged to report data within these defined limits; where protected information is inadvertently transmitted to NACOR, it is censored before publication or distribution in any format. Public reporting from NACOR is with deidentified aggregates only, with additional censoring of data in small bins to prevent reidentification, for example, of very large or very isolated hospitals. Patient age ≥90 years is truncated to >89, and zip codes covering <20,000 people are truncated to just the first 3 digits.
Ongoing data validation has identified dozens of entry and transmission errors; each of these has led to a corresponding automated validation routine applied on a continual basis to all existing and incoming data from all sources (e.g., routine screening to prevent double-loading of a single month’s data) (Table 1). Automated validation routines not only censor obviously invalid data (e.g., negative patient age; replaced with a −1 code for unknown) but also identify patterns of repeated errors that indicate systematic issues in capture or transmission. These are shared with vendors and practices and often lead to improvement in the quality of data at the local level, as in the examples above.
ACCESSING THE PARTICIPANT USER FILE
Investigators in NACOR participating practices can request access to the participant user file (PUF). The PUF includes deidentified case-level data from January 1, 2010 to the present from every practice in NACOR and is updated once each quarter. To access the PUF, researchers must describe the purpose of their research, must agree not to attempt any reidentification of the data, and must pledge to acknowledge NACOR as the source in any presentation or publication. The AQI Data Use Committee has established criteria for release of the PUF to investigators, including standards for projects that can be immediately approved, projects that can be approved by the executive director, and projects that require review by the committee. Since 2010, several studies based on NACOR data have been published.5–8 A current list of research output can be found online.f
Submission of data to NACOR is for the primary purpose of supporting local practice quality improvement, through both direct feedback of data to the practice and through aggregation into external benchmarks that can be used to suggest areas of high or low performance that would benefit from focused review. IRB approval of NACOR participation has not been required because participation has been determined to be a quality improvement activity, not human subjects research. Researchers wishing to repurpose NACOR data for scientific research must apply to their local IRB for permission to conduct a 1-time abstraction of deidentified information from the PUF to a research dataset, with a waiver of the requirement for individual informed consent. The AQI itself follows a similar mechanism for research arising internally, through a national IRB (Chesapeakeg). This overall practice, and the discrimination between quality improvement and scientific research as the primary purpose for data collection, is common among specialty-based clinical data registries.
Although NACOR itself contains all data submitted by the practice, the PUF is filtered to include only those data fields with high submission rates across practices. The PUF also includes additional, derived fields that may be of use to researchers. For example, some billing software reports all patients who are ASA physical status (PS) I or II as I, presumably because payers have not historically required greater granularity. The absence of ASA PS II cases was identified by an AQI analyst during initial loading of data from the first such practice encountered and an automated routine deployed to identify this failing in any submitted dataset. In the PUF ASA 1 cases from these groups are listed as unknown in the ASA PS column but as 12 in the Derived ASA PS column. This offers the researcher the choice of censoring data from those sites if examining a question where the distinction between I and II would matter or including those sites and reporting all PS using a I or II category.
Another example of derived data in the PUF is the designation of a primary surgical procedure from among the Current Procedural Terminology (CPT®) codes submitted. This depends on a hierarchical algorithm beginning with the procedure with the greatest base unit value in the associated anesthesia CPT code. Orthotopic liver transplant would thus be derived as the primary procedure, over placement of arterial catheter, intraoperative cholangiogram, or transesophageal echocardiography. Whereas it is straightforward to crosswalk any surgical CPT code to an appropriate anesthesia code, the converse is not true. When only anesthesia CPT codes are available, no primary surgical CPT code is assigned. This is true of about 15% of cases in NACOR, either because the practice does not record surgical CPT codes or because the procedure involves an anesthetic activity but no associated surgery, for example, intubation of a patient in the medical intensive care unit.
Missing data are a substantial concern in NACOR and a consequence of the cost-efficient but hands-off nature of the model. Data may be missing systematically, for example, when the billing software accessed does not record surgical CPT codes. Data may also be missing at random, as when a single case from the practice is lacking a patient age. Systematic issues are identified and pursued by AQI analysts to the point of identifying the failure in the primary software, communicating it to the practice and the vendor and recommending correction in future upgrades. Limited human capacity in both the AQI and practice generally preclude pursuit and re-entry of data missing at random. Researchers working with the PUF are encouraged to present a flow chart showing the selection of the final analytic data from the original file, including the steps at which cases with missing data are identified and excluded.
USES AND LIMITATIONS OF NACOR DATA
In contrast to other clinical data registries, the AQI did not begin by prespecifying the desired data elements, but rather by collection of what was readily available in digital form.9 The limitation is that data may be more heterogeneous in definition and method of collection than in other registries or purpose-built research datasets. The absence of a requirement for dedicated on-site data collection reduces the barrier to participation, but necessitates greater care when analyzing the data. Elements with subjective definitions or multiple means of collection and reporting, such as the occurrence of postoperative nausea and vomiting, are likely less reliable. The acceptable boundaries of reliability will be different for different purposes and should be assessed carefully by the investigator when selecting a research dataset from the PUF.
Because of the focus on administrative data as the desired minimal dataset, NACOR has the advantage of including some information for every procedure for which a patient might be anesthetized, and every sort of patient who might present for procedural care. This has already revealed some truths about American anesthesia practice, including the 30% of all cases now occurring outside of the traditional operating room (OR)7 and the 34.1% of all surgical anesthesia occurring in patients aged 65 years and older.6 Researchers seeking comparisons to OR practice, however, must carefully filter the PUF to exclude labor epidural analgesia, pain clinic procedures, off-floor line placements, and intubations and other such activities. Results will otherwise be skewed, especially when considering clinical outcomes such as difficult airway management or intraprocedure cardiac arrest that are more commonly seen in the OR.
Focusing on an anesthetic episode as the basic unit of capture is intuitive for anesthesiologists and makes transfer of administrative data straightforward but does not enable linkage of multiple procedures occurring over time in the same patient. Longitudinal tracking of patients to address longer-term outcomes, such as metastatic spread of cancer, anatomic progression of a difficult airway, or cognitive dysfunction after surgery is not currently possible in NACOR.
Researchers and reviewers must be aware that historical data in the PUF expand with each new release; when publishing, they must indicate the specific version of the PUF used. It is common for a researcher to request the PUF, develop and test study methodology, and then run the final analytics on a newer version, often with a substantial increase in numbers.
NACOR is based on periodic transfers of case-specific data directly from 1 electronic system to another. Traditional medical specialty registries use manual abstraction of information from patient medical records into the registry. This decreases the risk of missing data and encourages application of uniform data definitions but increases expense and decreases throughput.10 Registry participation may therefore require case sampling rather than a census of all eligible cases, creating the potential for selection bias. Bias may also occur at the level of participant recruitment because smaller facilities cannot afford the substantial cost of this mechanism.
Individual hospitals and practices may have reasons for participating in NACOR that could lead to a nonrandom sample of national anesthesia practice. For example, Veterans Affairs hospitals have been prohibited from transmitting case data outside of the system and so are barred from participation.
Research in NACOR must therefore begin with the assessment of the availability of necessary information and consideration of the possibility of bias. AIMS data are more likely to be available in larger than in smaller hospitals, for example. Whether and how this bias affects the proposed analysis will depend on the project in question.
NACOR is a unique resource for anesthesia practices seeking to improve their quality of care. NACOR may have a secondary use in support of clinical research. Large quantities of data are available, representing a substantial fraction of national anesthesia practice, but researchers must understand the limitations of NACOR data.
Name: Adrian Liau, PhD.
Contribution: This author helped to design the study, analyze the data, and prepare the manuscript.
Attestation: Adrian Liau approved the final manuscript. He attests to the integrity of the original data and the analysis reported in this manuscript and is the archival author.
Conflicts of Interest: Adrian Liau is an employee of the Anesthesia Quality Institute, the nonprofit corporation that operates the National Anesthesia Clinical Outcomes Registry.
Name: Jeana E. Havidich, MD, MS, FAAP.
Contribution: This author helped to design the study, analyze the data, and prepare the manuscript.
Attestation: Jeana E. Havidich approved the final manuscript. She attests to the integrity of the original data and the analysis reported in this manuscript.
Conflicts of Interest: The author has no conflicts of interest to declare.
Name: Tracy Onega, PhD, MS.
Contribution: This author helped to design the study and prepare the manuscript.
Attestation: Tracy Onega approved the final manuscript.
Conflicts of Interest: The author has no conflicts of interest to declare.
Name: Richard P. Dutton, MD, MBA
Contribution: This author helped to design the study and prepare the manuscript.
Attestation: Richard P. Dutton approved the final manuscript.
Conflicts of Interest: Richard P. Dutton is the Chief Quality Officer of the American Society of Anesthesiologists and Executive Director of the Anesthesia Quality Institute, the nonprofit corporation that operates the National Anesthesia Clinical Outcomes Registry.
This manuscript was handled by: Franklin Dexter, MD, PhD.
The authors thank Hubert Kordylewski, PhD (AQI Systems Architect, Anesthesia Quality Institute, Schaumburg, IL), Benjamin Westlake, BS (AQI Applications Development Manager, Anesthesia Quality Institute, Schaumburg, IL), and Antonio Sepulveda, BS (AQI Application Developer, Anesthesia Quality Institute, Schaumburg, IL), for clarifications to portions of the manuscript.