Anesthesia information management system (AIMS) data are entered as routine clinical documentation or acquired automatically from patient monitors and anesthesia machines.1,2 Secondary use of such data involves activities such as quality assurance, governmental reporting, clinical decision support, and research.3,4 Studies using AIMS data have provided valuable, pragmatic insights that could not have been realized practically within the constraints of traditional randomized clinical trials.5,6 In addition, retrospective data from large numbers of patients can easily be collected and analyzed.7 Such processes avoid the corresponding expense and logistical difficulties of enrolling patients in a prospective, observational study.8 Because AIMSs are ubiquitous in academic anesthesia departments,9 nearly all departments can be performing perioperative informatics research. However, to our knowledge, few departments worldwide published >5 informatics papers involving AIMS data from 2006 through 2017 (eg, 4 departments in an annotated bibliography of such studies).10
In this special article, we reviewed the computer code used to extract the data and text for all 47 studies published between January 2006 and August 2017 using AIMS data from Thomas Jefferson University Hospital (TJUH).11–57 Our approach was deliberately not a systematic or narrative review but rather a quantitative, self-reflective study of our experiences at a single institution that was highly productive in the area of AIMS informatics. Data from this institution were used in the largest number (26/90; P = .0007 versus the next largest group, 9/90) of papers describing the use of AIMS published in this time frame (ie, 26/90 in the bibliography; P = .0007).10 The objective of the current article was to identify factors that made TJUH successful in publishing anesthesia informatics studies so that others might learn from our experiences. Given the ubiquity of AIMS,9,58 we think that the information provided in the article should be of benefit to anesthesia researchers relying on AIMS data and to departments wishing to assess their own internal performance using their AIMS. Metrics on the quality of the data contained in the TJUH AIMS data warehouse are presented to provide benchmarks that departments can use when establishing internal processes and budgets for a research-level database. Importantly, we explain how access to data sources external to the AIMS was critical to success at TJUH and why this finding matters, even in the era of enterprise-wide electronic health record systems.
We studied data from TJUH for 2 reasons. First, to perform this study, we needed to examine the original structured query language (SQL) used to pull the studies’ data. This examination was done to know which data sources external to the AIMS were relied on for each study. For example, data missing in the AIMS data warehouse (ie, not recorded by the anesthesia providers) sometimes were populated with data recorded in the operating room (OR) management system. Such details were typically absent from the Methods sections of the published papers. Because the first author (R.H.E.) was the AIMS database administrator and had written all the data extract queries for the studies, we were assured that every study that used the AIMS data was included. Second, data from TJUH were studied because the AIMS on which the studies were based had a finite existence: October 18, 2005 through March 17, 2017. The TJUH AIMS has been replaced with Epic Anesthesia (Epic Systems, Verona, WI). Consequently, the percentages reported in the current article are accurate and unchanging. Had use of the system been ongoing, we likely could have improved the deficiencies noted in the areas of poor data quality, thereby altering the results. All of the studies cited from TJUH were approved by the Thomas Jefferson University Institutional Review Board.11–57 The institutional review board provided an exemption of informed consent for studies where protected health information was used to enable linking to records from other databases.
We followed Weiskopf and Weng’s59 process of assessing data quality among 5 dimensions: (1) completeness (ie, the extent to which data are missing); (2) correctness (ie, the extent to which data are correct); (3) concordance (ie, internal consistency or agreement with an external data source); (4) plausibility (ie, does a data element make sense); and (5) currency (ie, was the data element recorded close to the time it was measured or occurred).
Table 1 summarizes the implications of each of the sections that follow.
1. EXTERNAL DATA SOURCES
1.1 AIMS Interfacing to External Data Sources
The AIMS used at TJUH was Innovian (Dräger, Telford, PA). The backend database was SQL Server (Microsoft, Redmond, WA). The system was interfaced with the hospital admission, discharge, and transfer (ADT) system (Centricity, GE Healthcare, Wauwatosa, WI), which transmitted patient demographic information, medical record and hospital account numbers, allergies, and insurance information. Interfaces to the laboratory information management system transmitted a selected list of laboratory tests and blood type information to the AIMS. These labs included both preoperative and postoperative tests evaluated under the patient’s account number for that hospitalization. The accessibility of such labs was necessary for the studies related to blood transfusion.25,53 The preoperative assessment, including problems identified by their International Classification of Diseases, Ninth Revision, Clinical Modification, was captured in a hybrid of the AIMS database and an external database built for fields not represented in the AIMS.23,32,50 We linked data from the anesthesia billing system (Centricity) to the AIMS for studies in which anesthesia Current Procedural Technology was used.18,25,37 This was accomplished by adding the AIMS unique case identifier as an invoice field.
There was an interface with the OR management system (ORSOS, McKesson, San Francisco, CA). When cases were scheduled using the OR management system, a corresponding anesthesia case was automatically created; modifications to the case also were transmitted to the AIMS. The unique scheduled case identifier was transmitted from the OR management system to the primary case table in the TJUH AIMS, facilitating the linkage of cases in the OR management system to cases in the AIMS. The ADT system sent the same patient demographic information to the OR management system and to the AIMS, including the patient’s name, birth date, medical record number, and account number, alternatively allowing linkage using this information. If a case was moved from one location to another location but not updated in the OR management system,12 then the case would not appear in the AIMS workstations’ display of the list of cases for the current day. The provider would need to remove the room filter in the AIMS to display all the patients for the day. For 2.2% (9470/433,411) of cases, providers did not do this lookup. Instead, they created a new case and entered the patient’s medical record number. This pulled in the correct patient information previously transmitted to the AIMS by the ADT interface, but there was no linkage with the scheduled case that was already in the AIMS. When obtaining data for studies, for all but 0.2% (994/433,411) of records, the original case from the OR management system could be located by matching the medical record number and the date of surgery. This OR management system data, including the scheduled start time, case duration, scheduled procedure, and/or primary surgeon, were used in many TJUH AIMS studies.13,25,29,33–35,38,39,41,45,46,53
With the exception of the magnetic resonance imaging suite, cases in all locations were documented using the AIMS. This included 2 delivery rooms and the gastrointestinal endoscopy suite (10 ORs), the cardiovascular interventional radiology and cardiac catheterization suite, the electrophysiology rooms, the transesophageal echocardiography room, and the bays in the postanesthesia care unit utilized for patients undergoing electroconvulsive therapy for depression.
The implication is that linking the AIMS to the OR management system, lab system, and hospital ADT system is recommended, as the other systems provide substantial data that would otherwise require manual entry by an anesthesia provider (Table 1).
1.2 Detailed Review of Each Research Study’s Computer Code
The first author (R.H.E.) had database owner privileges on the production and development AIMS servers, had read-access privileges to the OR management system database, and received automated transfers of transactions from the pharmacy. He effectively functioned as a member of the information technology department for both coordinating and operationalizing data transfers between the OR management system and the AIMS, as well as software and hardware updates on servers and workstations. He is board-certified in clinical informatics and certified by the Health Information Management Systems Society. The chair provided 3 nonclinical days a week for development and maintenance activities related to the AIMS.
To evaluate the importance of the interfaces quantitatively, we systematically reviewed the SQL queries from the 47 peer-reviewed research studies obtained using the TJUH AIMS. Most of these studies could not have been completed without external database sources (36/47, 76.6%, P = .0003 compared with 50%, 95% CI, 62.5%–86.8% using the method of Blyth-Still-Casella [StatXact-11, Cytel, Inc, Cambridge, MA]). Among the external sources, the OR management system was used significantly more frequently (26/36, 72%) than pharmacy, anesthesia billing, electronic medical record (EMR), hospital materials management, or registry data (Figure). Several studies could not have been performed without the pharmacy information.11,20,47
The implication is that accessibility to data maintained in systems external to the AIMS is critical to the ability to publish a large number of studies, especially the OR management system (Table 1).
2. DATA QUALITY
2.1 AIMS Configuration Settings Related to Data Quality
In the TJUH AIMS, several configuration settings were related to data quality. All changes to any anesthesia provider’s documentation of a case were recorded in an audit table. Thus, it was possible to determine the history of all changes to documentation, including the time that each change was made, the prior value, the provider who made the change, and the workstation from which the change was made. These audit tables were used in studies of case cancellations.29,34,35,39,41
The interval at which vital sign data were recorded to the database was set at every 1 minute; this was a nominal interval, and the actual intervals between successive measurements varied slightly. Innovian had the option of recording data as often as every 15 seconds but at the cost of greater storage requirements and times to read the vital signs table. We reviewed the impact of the decision at TJUH to use 1-minute intervals. When the original TJUH AIMS was discontinued in 2017, the vital signs table contained records for 443,411 cases in which vital sign trending had been initiated. The 2,042,741,912 rows consumed 207 GB. This represented ≈0.47 MB per case. During the lifetime of the AIMS, latency issues related to accessing this table were not an issue, in part, due to the partitioning of the table and efficient indexing. The 1-minute recording interval proved sufficient for all of the studies at TJUH that used vital sign data.12,13,19,21,22,24,26,42,52 However, we could not perform studies looking at transient events. For example, we did not have clinical records of brief episodes of supraventricular tachycardia or vagally mediated sinus arrest. The recorded heart rate in the AIMS would have been the median over the 1-minute interval. For example, we could not study retrospectively what would have been the predictive value of pulse pressure variation analysis algorithms.60,61 In retrospect, we wished that we had been recording data every 15 seconds, not every 1 minute. In some systems (eg, Epic), the maximum data resolution is every 1 minute.
All timestamps in the AIMS database used the date-time format, with times recorded to the nearest 3 milliseconds. These timestamps refer to the times of documentation including the pressing of buttons. Some other systems (eg, Epic) have timestamps only accurate to the nearest minute. The studies of latency of responses could not have been performed accurately without timestamps accurate to ≥1 second (see Section 3).
The implications are that high-resolution timestamps and access to the audit tables should be provided if one wants to be able to evaluate the sequences of events. A frequency of vital sign data recording at a minimum of at least every 1 minute is recommended, with higher resolution (eg, every 15 seconds) necessary to study transient events (Table 1). Given the decreasing cost of data storage and improvements in computer processing power, capture of waveforms may become more feasible; this would allow study of heart rate variability and a detailed assessment of pulse pressure or systolic pressure variability.7
2.2 Data Collected Automatically
The TJUH AIMS software captured all values transmitted continually from patient monitors (eg, pulse oximeter saturation) and the anesthesia machine over the most recent 1-minute interval but only stored in the database the median of up to the most recent 15 values recorded during the previous 1 minute.26 Values sent intermittently (eg, noninvasive blood pressure) were recorded without averaging. In comparison, some AIMS (eg, Epic) store only the most recently transmitted data during the recorded interval, with a timestamp rounded to the nearest minute. This introduces systematic bias in the recording process. If we had known only the last value during each interval, we could not have performed the study on resolved hypoxemia related to the potential utility of near real-time alerts.26 We also could not have performed the studies on gaps in the recording of intraoperative blood pressures.19,21
The TJUH AIMS was configured to prevent data collected automatically from being edited by the anesthesia providers. If the vital signs had been editable, correctness, concordance, and plausibility of the data would have been affected, analogous to the “smoothing” problems in manual records.62,63 The anesthesia providers could identify intervals with artifacts with a comment in the AIMS (eg, the surgeon was leaning on the blood pressure cuff, methylene blue was injected), but the values were not editable. Thus, artifacts from devices were recorded into the primary AIMS database and data warehouse. Data cleaning was performed during the analytical stage for each study published by our group involving vital signs.19,21,22,24,26,27,30,40,42,44,51
The implication is that understanding how data are being processed before storing in the database is necessary to assess bias (Table 1). Cleaning of data artifacts can occur successfully during data analysis.
2.3 Required Data Fields
A hard stop is a required data field that must be completed before the record can be printed or otherwise finalized. For example, in the Innovian implementation at the University of Miami, 22 fields needed to be completed before the anesthesia provider could print the anesthesia record in the postanesthesia care unit. Several of these hard stops are attestation notes to be entered and signed electronically by the anesthesiologist. A consequence of this use of hard stops is that if the anesthesiologist forgets to enter one of these events before the anesthesia provider attempts to finalize the case in the postanesthesia care unit or the intensive care unit, there can be delayed care for other patients. To avoid such scenarios, at TJUH, no events were programmed as hard stops in the AIMS.
Soft stops are events that generate a warning message when printing or otherwise finalizing a record but can be ignored. At TJUH, soft stops were configured for American Society of Anesthesiologists physical status (ASA PS), diagnosis, procedure, and the patient’s first and last names, height, and weight.
A notification system was built outside the AIMS in VB.Net (Microsoft) that monitored the AIMS for the absence of an anesthesia provider or surgeon in the staff list, surgery begin or end time, estimated blood loss (EBL), or intraoperative outcome. The notification system sent near real-time messages about such missing values to the anesthesia workstation, the anesthesia provider, or the anesthesiologist at specified offsets from milestones during the case or the next day.42 Because of software limitations in the TJUH AIMS, such elements could not be configured as either hard or soft stops.
The implication is that successful completion of substantial research11–57 shows that high-quality data can be achieved by means other than creating hard stops (Table 1).
3. DATA CURRENCY
Data currency relates to how quickly data in the AIMS are available for use. There is always a degree of latency from the time when events are recorded in the AIMS and when those data elements are available in the database for use (ie, currency).14,28,64 At TJUH, the AIMS software was built to be installed on each individual workstation, with local, temporary storage of data. This structure allowed for documentation of clinical care even when the workstation was not able to communicate with the network (eg, if there was a router problem). There was transmission from the local database back to the central database on the server every 30 seconds when available.14 This allowed near real-time checking of missing data, interfacing with communication systems to provide messages to anesthesia providers and anesthesiologists, and centralized display of information at the anesthesia control desk in the OR (see Section 2.3).13,19,42,47
The projects involving near real-time decision support could not have been executed without a high degree of data currency. At TJUH, such studies involved decision support related to Bayesian updating of time remaining in cases,13 automated reconciliation of controlled substances,47 and alerting providers about gaps in recording blood pressure.19
The implication is that studies involving near real-time decision support of anesthesia providers in ORs require access to AIMS data within a brief interval after their entry (Table 1).
4. ASSESSMENT OF DATA QUALITY IN THE AIMS DATA WAREHOUSE
For previous studies of data quality in the TJUH AIMS, data in the primary AIMS database were extracted into a data warehouse and enhanced with data from the OR management system database (see Section 1.1). These steps involved extensive coding. Before 2015, the steps were performed individually for each study. In 2015, the extraction, transformation, and loading process were automated to run daily. The change improved the efficiency of the data extraction and analysis process, as described by Hofer et al65 (Supplemental Digital Content, http://links.lww.com/AA/B369). In the remainder of Section 5, when we discuss data quality, we are talking about the data warehouse, not the production AIMS database. For this article, we quantified the quality of the data in the AIMS data warehouse by systematically evaluating documentation errors, missing elements, and measurement artifacts.66
4.1 Assessment of Data Quality
In Table 2, we provide the frequency of missing data for fields used in many AIMS studies. These percentages can serve as a reference for other anesthesia AIMS researchers looking at the completeness of their own data. Because the original TJUH AIMS (Innovian) has been retired, there are no more cases being entered into the database (ie, the denominator is fixed). All of the corrections and additions to the data warehouse have been made, so the numerator is also fixed. Thus, the data quality metrics reported in Table 2 are not subject to future change.
The 2 data fields at TJUH with the poorest quality were the closing time and the EBL (Table 2). Closing time was not a milestone that the TJUH providers typically documented as part of their workflow, being absent for 89.6% of cases. The consequence was that we could not perform studies using TJUH data looking at prolonged intervals from the start of closing to the end of surgery16,17,36,37 or examine the operational benefits of improving the prediction of the end time of cases using this event.67
Similar to what has been reported from other institutions,68 EBL was frequently not recorded by TJUH providers (Table 2).25,68 This absence of documentation occurred for procedures where blood loss was usually minimal and substantive blood loss was common.25,53 Missing EBL created issues for the analyses of the appropriateness of the blood transfusions.38,69 Although we published several studies in which EBL was used,25,53 this required extensive statistical analyses for the missing values. Because EBL could not be made a required field in the TJUH AIMS, we added absent EBL to the list of missing documentation on March 21, 2012. A popup message to the AIMS workstation was sent 10 minutes after entry of surgery end if the EBL had not been recorded. Although this increased the incidence of EBL documentation, the incidence of missing values remained insufficient to avoid the use of statistical inference even for internal hospital use: 58.0% (134,153/231,157) missing before the trigger was added reduced to 46.9% (99,645/212,254) thereafter.
Other fields frequently missing were height and weight (9.9% and 7.9%, respectively), which was problematic for studies that included drug doses or tidal volume.40,48
Filling in missing data from the OR management system resulted in improved data quality for milestones. For example, the surgery begin time was missing in 1.56% of 443,411 cases in the primary AIMS database but only in 0.68% of cases in the data warehouse because the latter included data filled in from the OR management system. Not all records could be corrected because some AIMS cases were never scheduled in the OR management system (usually non-OR cases), or there was no postcase data entry into the OR management system (again, usually for non-OR cases; see Section 1.1). The surgery begin milestone was important in the study of breaks provided to anesthesia providers, in which we found waiting 13 minutes after the surgery begin time was the most suitable interval to allow completion of induction documentation.28
The correctness of manually entered quantitative data was an occasional problem, with the entry of the right number but incorrect units. Particularly common were red blood cell transfusions entered using values of milliliter but listed as red blood cell units in 127/57,665 (0.22%) of transfusions. For example, 500 mL of autologous whole blood was sometimes entered as 500 units. These errors were recognized, and the values converted to units when the SQL queries were written (by R.H.E.) to extract and clean the data before analysis for the paper looking at the type and screen decision.25
The TJUH AIMS would not accept negative values in the drug or fluid fields, an intrinsic behavior of the AIMS. Default and totaling units were selected during system configuration. The most common doses used were added as quick entry buttons.49 However, users occasionally changed the units to incorrect selections. For example, in 25/6110 (0.41%) infusion rate entries for dexmedetomidine, units of μg/min or μg/kg/min were selected instead of the intended units of μg/kg/h.
The ASA PS was configured as a required field, with a prompt generated (ie, soft stop) if it was missing when trying to print (close) the case. The decision to use a soft stop was made because the field was used for billing.70 ASA PS was missing for 1.15% of the cases.
There were only 2 weekdays during the 11-year lifetime of the AIMS where the system was not usable due to network or data center issues. These issues were identified by determining the sequential count of cases done by day and looking for unexpected gaps between days with cases. Continuous periods of no cases in the data warehouse typically represented a failure of the extraction process. Such periods were identified as part of the data quality analysis for each study, the reason determined, the extraction code modified, as necessary, and the extract repeated. An ongoing, scheduled process of determining when the extract process failed would have been a better approach. The data warehouse did not include a field indicating days with an unusually low caseload. Rather, cases were counted on each day when analyses were performed to identify holidays and other slow-down days (eg, the day after Thanksgiving).71 Excluding days that were not typical workdays was important for studies involving analysis of normal weekday activity.12,25,29,36,39,41
When the AIMS was first installed at TJUH, there was a 4% incidence with cases being documented as having been done in the wrong OR.12 Such instances were detected by looking for cases with overlaps in the same OR based on the recorded enter. We fixed this problem by mapping the workstation transmitting pulse oximetry values to the room where we knew the workstation was physically attached to the anesthesia machine. This resulted in room location errors in the AIMS subsequently being 0.4%.12 Without having made that revision to processes, studies of turnover times50 and concurrency of cases would have been infeasible.24
The implication of this section is that the quality of information in the AIMS should be assessed and not assumed, as high-quality data are necessary for publication (Table 1).
4.2 Out-of-Sequence Events
An issue related to milestones is correctness and plausibility of sequence of events. For example, the OR enter time should always be before the OR leave time. Table 3 lists the frequency with which various pairs of events were out of sequence (ie, lacked internal validity). The only 2 sequences with frequencies >1% were (1) induction time after intubation, and (2) surgery end time after extubation (1.85% and 1.74%, respectively). These are high workload periods with documentation typically entered after the fact, possibly explaining the sequencing issue.14
The implication of this section is that the sequence of pairs of events with an expected order should be assessed as part of the internal validation of the AIMS data (Table 1).
4.3 Storage of Vital Sign and Anesthesia Machine Data
Another issue was that some vital sign and anesthesia machine variables of interest were stored in different fields in the AIMS database, depending on the label attached to the variable on the patient monitor. For example, for the study of end-of-case temperatures,52 data were in 8 possible fields, depending on if the anesthesia provider changed the label on the monitor from the default setting or if values were manually entered from a noninterfaced monitor. Among all 23,549,679 temperatures recorded in the AIMS, 86.8% were in the field “temp 1” (the default), 5% were in “blood temp,” 3.7% were in “temp skin 1,” 1.9% were in “temp esophagus 1,” 1.8% were entered as a manual temperature, and 0.7% were from other locations. In addition, the way the AIMS processed data sent from patient monitors and the anesthesia machine resulted in some values being stored multiple times in the AIMS database with different units (eg, partial pressures in units of %, mm Hg, and kPa). Given that the TJUH AIMS recorded hundreds of variables transmitted from the anesthesia machine and patient monitors, many of which were not displayed on the devices or AIMS screens, extensive evaluation was required.
A separate issue related to temperature recording was that, although all patient monitors were supposed to have been configured with the temperature in °C, the units were set in °F in 7984 of 23,549,679 (0.034%) temperatures in the database. Conversion of such values was addressed during the stage of analysis.52
The implication is that a detailed examination of the AIMS database is required to determine what data elements are being recorded, their units, and potential alternative locations that may be used, depending on monitor and device settings (Table 1).
4.4 Drug and Fluid Documentation Issues
If drugs or fluids were given that had not been configured in the drug selection list, or if the provider did not locate the drug, an “other drug” or “other fluid” entry was selected with the drug name entered as a comment. For example, in the database, Keppra (levetiracetam) was entered as an “other drug” 81 times before the drug was added to the drug list in January 2014. Since then, it was entered 1318 times as Keppra and only 9 times as an “other drug” with a comment including the word “Keppra.” Such occurrences were identified in the study of the most common doses of intraoperative drugs.49 Having to parse manually entered text to determine the drugs and fluids that were administered, similar to what was done for the study of drug allergies,43 would add considerable complexity to studies.
For the study of scheduled drug discrepancies,47 we had to adjust for occasional situations where the intravenous version of midazolam was selected and the route was changed to oral, rather than the provider selecting the oral version of midazolam. Otherwise, the incidence of discrepancies would have been inflated because we only reconciled the intravenous formulation.
The implication is that an AIMS database needs to be assessed periodically for new drugs and fluids that are not included as discrete items in the selection lists, with subsequent addition of the missing items to the lists (Table 1).
The unique and important feature of this special article is that the database studied is closed, with no new cases being added. Thus, the reported percentages and counts from the TJUH AIMS (Tables 2 and 3) will not change in the future.
From this review of the TJUH AIMS data warehouse and the manuscripts published, 3 major themes emerged (Table 1). First, the investigators had relatively unconstrained access to a variety of external data sources, most notably the OR management system. This contributed to the data quality in the AIMS data warehouse and allowed for the completion of many studies that otherwise would not have been possible. Second, near real-time monitoring of missing data and contemporaneous feedback were necessary to obtain AIMS data that were generally high quality, but still with some extreme exceptions (eg, EBL). Third, from our examination of the SQL code and comments in the stored procedures for the TJUH studies, there were unexpected issues in nearly every study discovered during the data extraction process; these required more complex coding than originally written to retrieve the appropriate data for analysis. Without having direct access to explore and validate the data in the AIMS database, many of these data quality issues likely would not have been recognized. We do not think that the analysts working for the TJUH information systems department would have been able to identify such problems because they lacked sufficient domain knowledge of anesthesia workflow and documentation issues.
For anesthesia departments with stand-alone AIMSs seeking to increase their academic productivity using perioperative data sources, the review of the experiences at TJUH (Section 1.2) shows that interfacing with the OR management system is a critical step. More than half of the studies published from TJUH required such access. Near real-time (1 minute) latency checks were used for these data to be accurate (eg, preventing errors in the room where the case was being performed). Finally, access to the AIMS and OR management system databases appears to be necessary to assess the quality of the data. Close coordination and cooperation with the managers and stakeholders of the OR management system are important to ensure high-quality data.
Concerns related to external data quality will exist even when an AIMS is a component of an enterprise-wide electronic health system (eg, Epic, Cerner). There are separate modules for each functional area in these systems (eg, anesthesia, OR scheduling, obstetrics, billing), each of which was developed independently over relatively long periods of time. Data quality in these modules will depend on the uses of those data by other departments.
The ability to access data in near real time may be impossible due to imposed restrictions on such access from the information systems department or from limitations resulting from the architecture of the enterprise system. Architectural issues are relevant for hospitals where most data are not exposed for querying until the next day (eg, typical for Epic), as well as those hosting their applications on remote servers provided by the vendor (eg, typical for Cerner).
Our study has several limitations. First, although we hypothesize that the managerial and operational roles of the first author (R.H.E.) and his informatics credentials were crucial in the scientific production arising from the TJUH AIMS, we could not assess the generalizability or validity of this hypothesis from the TJUH data alone. However, the anesthesia department at Vanderbilt University Medical Center (VUMC), which had the second highest academic output using their AIMS data during the same period,10 had a similar but more extensive structure than that described at TJUH. AT VUMC, 3 anesthesiologists also board-certified in Clinical Informatics (Jonathan Wanderer, Jesse Ehrenfeld, and Brian Rothman) oversaw the development and research efforts related to their locally created and maintained AIMS.72 Extensive support was provided by the department, including dedicated time for the anesthesiologists responsible for the AIMS, a database administrator, analysts, and system developers (Warren Sandberg, Chair, Vanderbilt University Medical Center Department of Anesthesiology, personal communication, 2012). The first author (R.H.E.) did a 6-month sabbatical at Vanderbilt between 2012 and 2013 and was given full access to their AIMS data warehouse and support by an analyst from their OR management system; 11 manuscripts resulted from this collaboration.9,33,34,66,73–79 Examination of the SQL code and comments in the stored procedures written for those studies revealed similar features related to data elements (ie, unexpected situations requiring complex coding) as found in the stored procedures at TJUH. The productive outcome of this sabbatical highlights the implications in Table 1. Lower volume academic departments would not be relevant to the determination of factors responsible for high academic productivity.
A second limitation is that our study interval preceded the replacement at TJUH in April 2017 of its “best-of-breed” AIMS (Innovian) to Epic. VUMC replaced its locally developed AIMS in November 2017. The ability of these 2 institutions to maintain a high level of academic productivity using AIMS data from Epic is unknown. However, it likely will be degraded, at least over the short term, given the current limitations of the Epic system as compared with features identified to be important in our current article (eg, timestamp resolution and lack of availability of data in near real-time access, technical barriers to near real-time database access). At the first author’s current institution (University of Miami), the AIMS (Innovian) was replaced with Epic at the end of October 2017, and at the affiliated Jackson Memorial Hospital, the legacy AIMS (Picis, N. Harris Computer Corporation, Wakefield, MA) was replaced with SurgiNet Anesthesia (Cerner, Kansas City, MO) in July 2016. At both TJUH and the University of Miami, limited database access to the enterprise systems has resulted in a hiatus to research using their AIMS data.
A final limitation is that the specific situations and details present at TJUH are not likely to apply to any individual hospital; however, we believe that they would be represented among all hospitals either directly or analogously. For example, the GE Centricity ADT system at TJUH would be present at some but not most hospitals, but another vendor’s ADT system likely would be in use. A similar situation would exist for the interface with ORSOS and a different OR scheduling system.
In conclusion, our analysis of the AIMS publications from an anesthesia department indicates that the primary factors critical to data quality and successful publication were interfacing with the OR management system and securing access to external database sources. The presence of anesthesiologists in the department who are either board-certified in clinical informatics or have equivalent training may also be an important factor.
Name: Richard H. Epstein, MD.
Contribution: This author helped design the study, conduct the study, analyze the data, and write the article.
Name: Franklin Dexter, MD, PhD, FASA.
Contribution: This author helped design the study, conduct the study, analyze the data, and write the article.
This manuscript was handled by: Maxime Cannesson, MD, PhD.
1. Ehrenfeld JM, Rehman MAAnesthesia information management systems: a review of functionality and installation considerations. J Clin Monit Comput. 2011;25:71–79.
2. Glance LG, Wanderer JP, Dick AW, Dutton RPBuilding bridges across clinical registries. Anesth Analg. 2017;125:689–691.
3. Safran C, Bloomrosen M, Hammond WE, et al.Toward a national framework for the secondary use of health data: an American Medical Informatics Association White Paper. J Am Med Inform Assoc. 2007;14:1–9.
4. Hersh WRAdding value to the electronic health record through secondary use of data for quality assurance, research, and surveillance. Am J Manag Care. 2007;13:277–278.
5. Patsopoulos NAA pragmatic view on pragmatic trials. Dialogues Clin Neurosci. 2011;13:217–224.
6. Tunis SR, Stryer DB, Clancy CMPractical clinical trials: increasing the value of clinical research for decision making in clinical and health policy. JAMA. 2003;290:1624–1632.
7. Levin MA, Wanderer JP, Ehrenfeld JMData, big data, and metadata in anesthesiology. Anesth Analg. 2015;121:1661–1667.
8. Battelle Technology Partnership Practice. Biopharmaceutical industry-sponsored clinical trials: impact on state economies. Available at: http://phrma-docs.phrma.org/sites/default/files/pdf/biopharmaceutical-industry-sponsored-clinical-trials-impact-on-state-economies.pdf
. Accessed May 1, 2017.
9. Stol IS, Ehrenfeld JM, Epstein RHTechnology diffusion of anesthesia information management systems into academic anesthesia departments in the United States. Anesth Analg. 2014;118:644–650.
10. Dexter FAnestheisa Information Management systems (AIMS). Available at: http://www.franklindexter.net/bibliography_AIMS.htm
. Accessed June 26, 2017.
11. Epstein RH, Gratch DM, Grunwald ZDevelopment of a scheduled drug diversion surveillance system based on an analysis of atypical drug transactions. Anesth Analg. 2007;105:1053–1060.
12. Epstein RH, Dexter F, Piotrowski EAutomated correction of room location errors in anesthesia information management systems. Anesth Analg. 2008;107:965–971.
13. Dexter F, Epstein RH, Lee JD, Ledolter JAutomatic updating of times remaining in surgical cases using bayesian analysis of historical case duration data and “instant messaging” updates from anesthesia providers. Anesth Analg. 2009;108:929–940.
14. Epstein RH, Dexter F, Ehrenfeld JM, Sandberg WSImplications of event entry latency on anesthesia information management decision support systems. Anesth Analg. 2009;108:941–947.
15. Dexter F, Epstein RH, Elgart RL, Ledolter JForecasting and perception of average and latest hours worked by on-call anesthesiologists. Anesth Analg. 2009;109:1246–1252.
16. Dexter F, Bayman EO, Epstein RHStatistical modeling of average and variability of time to extubation for meta-analysis comparing desflurane to sevoflurane. Anesth Analg. 2010;110:570–580.
17. Agoliati A, Dexter F, Lok J, et al.Meta-analysis of average and variability of time to extubation comparing isoflurane with desflurane or isoflurane with sevoflurane. Anesth Analg. 2010;110:1433–1439.
18. Simpao A, Heitz JW, McNulty SE, Chekemian B, Brenn BR, Epstein RHThe design and implementation of an automated system for logging clinical experiences using an anesthesia information management system. Anesth Analg. 2011;112:422–429.
19. Ehrenfeld JM, Epstein RH, Bader S, Kheterpal S, Sandberg WSAutomatic notifications mediated by anesthesia information management systems reduce the frequency of prolonged gaps in blood pressure documentation. Anesth Analg. 2011;113:356–363.
20. Epstein RH, Gratch DM, McNulty S, Grunwald ZValidation of a system to detect scheduled drug diversion by anesthesia care providers. Anesth Analg. 2011;113:160–164.
21. Epstein RH, Dexter FMean arterial pressures bracketing prolonged monitoring interruptions have negligible systematic differences from matched controls without such gaps. Anesth Analg. 2011;113:267–271.
22. Dexter F, Maguire D, Epstein RHObservational study of anaesthetists’ fresh gas flow rates during anaesthesia with desflurane, isoflurane and sevoflurane. Anaesth Intensive Care. 2011;39:460–464.
23. Dexter F, Witkowski TA, Epstein RHForecasting preanesthesia clinic appointment duration from the electronic medical record medication list. Anesth Analg. 2012;114:670–673.
24. Epstein RH, Dexter FInfluence of supervision ratios by anesthesiologists on first-case starts and critical portions of anesthetics. Anesthesiology. 2012;116:683–691.
25. Dexter F, Ledolter J, Davis E, Witkowski TA, Herman JH, Epstein RHSystematic criteria for type and screen based on procedure’s probability of erythrocyte transfusion. Anesthesiology. 2012;116:768–778.
26. Epstein RH, Dexter FImplications of resolved hypoxemia on the utility of desaturation alerts sent from an anesthesia decision support system to supervising anesthesiologists. Anesth Analg. 2012;115:929–933.
27. Mraovic B, Schwenk ES, Epstein RHIntraoperative accuracy of a point-of-care glucose meter compared with simultaneous central laboratory measurements. J Diabetes Sci Technol. 2012;6:541–546.
28. Epstein RH, Dexter FMediated interruptions of anaesthesia providers using predictions of workload from anaesthesia information management system data. Anaesth Intensive Care. 2012;40:803–812.
29. Dexter F, Shi P, Epstein RHDescriptive study of case scheduling and cancellations within 1 week of the day of surgery. Anesth Analg. 2012;115:1188–1195.
30. Schwenk ES, Mraovic B, Maxwell RP, Kim GS, Ehrenfeld JM, Epstein RHRoot causes of intraoperative hypoglycemia: a case series. J Clin Anesth. 2012;24:625–630.
31. Dexter F, Epstein RH, Wachtel RE, Rosenberg HEstimate of the relative risk of succinylcholine for triggering malignant hyperthermia. Anesth Analg. 2013;116:118–122.
32. Dexter F, Ahn HS, Epstein RHChoosing which practitioner sees the next patient in the preanesthesia evaluation clinic based on the relative speeds of the practitioner. Anesth Analg. 2013;116:919–923.
33. Dexter F, Ledolter J, Tiwari V, Epstein RHValue of a scheduled duration quantified in terms of equivalent numbers of historical cases. Anesth Analg. 2013;117:205–210.
34. Ehrenfeld JM, Dexter F, Rothman BS, Johnson AM, Epstein RHCase cancellation rates measured by surgical service differ whether based on the number of cases or the number of minutes cancelled. Anesth Analg. 2013;117:711–716.
35. Epstein RH, Dexter FRescheduling of previously cancelled surgical cases does not increase variability in operating room workload when cases are scheduled based on maximizing efficiency of use of operating room time. Anesth Analg. 2013;117:995–1002.
36. Epstein RH, Dexter F, Brull SJCohort study of cases with prolonged tracheal extubation times to examine the relationship with duration of workday. Can J Anaesth. 2013;60:1070–1076.
37. Dexter F, Epstein RHIncreased mean time from end of surgery to operating room exit in a historical cohort of cases with prolonged time to extubation. Anesth Analg. 2013;117:1453–1459.
38. Dexter F, Epstein RHApplying systematic criteria for type and screen based on procedure’s probability of erythrocyte transfusion. Anesthesiology. 2014;120:241.
39. Dexter F, Maxbauer T, Stout C, Archbold L, Epstein RHRelative influence on total cancelled operating room time from patients who are inpatients or outpatients preoperatively. Anesth Analg. 2014;118:1072–1080.
40. Wanderer JP, Ehrenfeld JM, Epstein RH, et al.Temporal trends and current practice patterns for intraoperative ventilation at US academic medical centers: a retrospective study. BMC Anesthesiol. 2015;15:40.
41. Epstein RH, Dexter FManagement implications for the perioperative surgical home related to inpatient case cancellations and add-on case scheduling on the day of surgery. Anesth Analg. 2015;121:206–218.
42. Epstein RH, Dexter F, Patel NInfluencing anesthesia provider behavior using anesthesia information management system data for near real-time alerts and post hoc reports. Anesth Analg. 2015;121:678–692.
43. Epstein RH, Jacques PS, Wanderer JP, Bombulie MR, Agarwalla NProphylactic antibiotic management of surgical patients noted as “allergic” to penicillin at two academic hospitals. A A Case Rep. 2016;6:263–267.
44. Epstein RH, Dexter F, Maguire DP, Agarwalla NK, Gratch DMEconomic and environmental considerations during low fresh gas flow volatile agent administration after change to a nonreactive carbon dioxide absorbent. Anesth Analg. 2016;122:996–1006.
45. Shi P, Dexter F, Epstein RHComparing policies for case scheduling within 1 day of surgery by Markov Chain Models. Anesth Analg. 2016;122:526–538.
46. Dexter F, Wachtel RE, Epstein RHDecreasing the hours that anesthesiologists and nurse anesthetists work late by making decisions to reduce the hours of over-utilized operating room time. Anesth Analg. 2016;122:831–842.
47. Epstein RH, Dexter F, Gratch DM, Perino M, Magrann JControlled substance reconciliation accuracy improvement using near real-time drug transaction capture from automated dispensing cabinets. Anesth Analg. 2016;122:1841–1855.
48. Schwenk ES, Goldberg SF, Patel RD, et al.Adverse drug effects and preoperative medication factors related to perioperative low-dose ketamine infusions. Reg Anesth Pain Med. 2016;41:482–487.
49. Rodriquez LI, Smaka TJ, Mahla M, Epstein RHDefault drug doses in anesthesia information management systems. Anesth Analg. 2017;125:255–260.
50. Epstein RH, Dexter F, Schwenk ES, Witkowski TABypass of an anesthesiologist-directed preoperative evaluation clinic results in greater first-case tardiness and turnover times. J Clin Anesth. 2017;41:112–119.
51. Goldhammer JE, Herman CR, Merguson MW, Torjman MC, Epstein RH, Sun J-ZPreoperative aspirin does not Increase transfusion or reoperation in isolated valve surgery. J Cardiothorac Vasc Anesth. 2017;31:1618–1623.
52. Epstein RH, Dexter F, Hofer IS, et al.Perioperative temperature measurement considerations relevant to reporting requirements for national quality programs using data from anesthesia information management systems. Anesth Analg. 2018;126:478–486.
53. Dexter F, Epstein RH, Ledolter J, et al.Validation of a new method to automatically select cases with intraoperative red blood cell transfusion for audit. Anesth Analg. 2018;126:1654–1661.
54. Epstein RH, Dexter F, Gratch DM, Lubarsky DAIntraoperative handoffs among anesthesia providers increase the incidence of documentation errors for controlled drugs. Jt Comm J Qual Patient Saf. 2017;43:396–402.
55. Dexter F, Epstein RHFor assessment of changes in intraoperative red blood cell transfusion practices over time, the pooled incidence of transfusion correlates highly with total units transfused. J Clin Anesth. 2017;39:53–56.
56. Epstein RH, Dexter F, Schwenk ESHypotension during induction of anaesthesia is neither a reliable nor a useful quality measure for comparison of anaesthetists’ performance. Br J Anaesth. 2017;119:106–114.
57. Wanderer JP, Gratch DM, Jacques PS, Rodriquez LI, Epstein RHTrends in the prevalence of intraoperative adverse events at 2 academic hospitals after implementation of a mandatory reporting system. Anesth Analg. 2017;126:134–140.
58. Trentman TL, Mueller JT, Ruskin KJ, Noble BN, Doyle CAAdoption of anesthesia information management systems by US anesthesiologists. J Clin Monit Comput. 2011;25:129–135.
59. Weiskopf NG, Weng CMethods and dimensions of electronic health record data quality assessment: enabling reuse for clinical research. J Am Med Inform Assoc. 2013;20:144–151.
60. Desebbe O, Joosten A, Suehiro K, et al.A novel mobile phone application for pulse pressure variation monitoring based on feature extraction technology: a method comparison study in a simulated environment. Anesth Analg. 2016;123:105–113.
61. Maguire S, Rinehart J, Vakharia S, Cannesson MTechnical communication: respiratory variation in pulse pressure and plethysmographic waveforms: intraoperative applicability in a North American academic center. Anesth Analg. 2011;112:94–96.
62. Wax DB, Beilin Y, Hossain S, Lin HM, Reich DLManual editing of automatically recorded data in an anesthesia information management system. Anesthesiology. 2008;109:811–815.
63. Reich DL, Wood RK Jr, Mattar R, et al.Arterial blood pressure and heart rate discrepancies between handwritten and computerized anesthesia records. Anesth Analg. 2000;91:612–616.
64. Nair BG, Peterson GN, Newman SF, Wu WY, Kolios-Morris V, Schwid HAImproving documentation of a beta-blocker quality measure through an anesthesia information management system and real-time notification of documentation errors. Jt Comm J Qual Patient Saf. 2012;38:283–288.
65. Hofer IS, Gabel E, Pfeffer M, Mahbouba M, Mahajan AA systematic approach to creation of a perioperative data warehouse. Anesth Analg. 2016;122:1880–1884.
66. Weiner MG, Embi PJToward reuse of clinical data for research and quality improvement: the end of the beginning? Ann Intern Med. 2009;151:359–360.
67. Tiwari V, Dexter F, Rothman BS, Ehrenfeld JM, Epstein RHExplanation for the near-constant mean time remaining in surgical cases exceeding their estimated duration, necessary for appropriate display on electronic white boards. Anesth Analg. 2013;117:487–493.
68. Frank SM, Rothschild JA, Masear CG, et al.Optimizing preoperative blood ordering with data acquired from an anesthesia information management system. Anesthesiology. 2013;118:1286–1297.
69. Wanderer JP, Anderson-Dam J, Levine W, Bittner EADevelopment and validation of an intraoperative predictive model for unplanned postoperative intensive care. Anesthesiology. 2013;119:516–524.
70. Marian AA, Bayman EO, Gillett A, Hadder B, Todd MMThe influence of the type and design of the anesthesia record on ASA physical status scores in surgical patients: paper records vs electronic anesthesia records. BMC Med Inform Decis Mak. 2016;16:29.
71. Starnes JR, Wanderer JP, Ehrenfeld JMMetadata from data: identifying holidays from anesthesia data. J Med Syst. 2015;39:44.
72. Rothman B, Sandberg WS, St Jacques PUsing information technology to improve quality in the OR. Anesthesiol Clin. 2011;29:29–55.
73. Ehrenfeld JM, Dexter F, Rothman BS, et al.Lack of utility of a decision support system to mitigate delays in admission from the operating room to the postanesthesia care unit. Anesth Analg. 2013;117:1444–1452.
74. Wanderer JP, Ehrenfeld JM, Sandberg WS, Epstein RHThe changing scope of difficult airway management. Can J Anaesth. 2013;60:1022–1024.
75. Rothman BS, Dexter F, Epstein RHCommunication latencies of Apple push notification messages relevant for delivery of time-critical information to anesthesia providers. Anesth Analg. 2013;117:398–404.
76. Epstein RH, St Jacques P, Stockin M, Rothman B, Ehrenfeld JM, Denny JCAutomated identification of drug and food allergies entered using non-standard terminology. J Am Med Inform Assoc. 2013;20:962–968.
77. Epstein RH, Dexter F, Rothman BCommunication latencies of wireless devices suitable for time-critical messaging to anesthesia providers. Anesth Analg. 2013;116:911–918.
78. Epstein RH, Dexter F, Lopez MG, Ehrenfeld JMAnesthesiologist staffing considerations consequent to the temporal distribution of hypoxemic episodes in the postanesthesia care unit. Anesth Analg. 2014;119:1322–1333.
79. Wanderer JP, Shi Y, Schildcrout JS, Ehrenfeld JM, Epstein RHSupervising anesthesiologists cannot be effectively compared according to their patients’ postanesthesia care unit admission pain scores. Anesth Analg. 2015;120:923–932.