Secondary Logo

Journal Logo

Technology, Computing, and Simulation: Review Article

Influencing Anesthesia Provider Behavior Using Anesthesia Information Management System Data for Near Real-Time Alerts and Post Hoc Reports

Epstein, Richard H. MD*; Dexter, Franklin MD, PhD; Patel, Neil MD

Author Information
doi: 10.1213/ANE.0000000000000677
  • Free


Anesthesia information management systems (AIMS) have progressed during the past 25 years1 from their initial role as simple electronic record keepers to tools used to provide feedback to individual anesthesia providers to modify their behavior.a These decision support tools can be characterized as related to: (1) enhancing reimbursement or reducing costs2–6; (2) increasing adherence to clinical, compliance, or regulatory protocols7–15; (3) improving documentation16–19; and/or (4) monitoring physiologic data in near real time and providing intraoperative recommendations at the point of care.20–25 We characterize these near real-time alert and post hoc report systems as included in the broad category of clinical decision support (CDS) defined by Greenes26 as “the use of information and communication technologies to bring relevant knowledge to bear on the health care and well-being of a patient.” Greenes26 summarized the characteristics of CDS as follows:

  • CDS is provided to a user (who may be a provider or a patient);
  • CDS selects relevant knowledge or creates that knowledge from data;
  • CDS involves a computerized process of inference (e.g., application of an algorithm, rule, or method of association);
  • CDS produces an action, usually in the form of a recommendation (i.e., this is not a closed-loop system).

The focus of our review is computer-based CDS for AIMS. We therefore exclude non-computerized anesthesia decision support tools, such as checklists, printed protocols, and other educational material. We consider all peer-reviewed, PubMed-indexed articles describing CDS for AIMS (Table 1)30–33 but not the broader field of CDS, as this has been reviewed adequately elsewhere,26,27 and much of the published work revolves around areas outside the scope of anesthesia practice (e.g., reminders to perform vaccinations or to test for occult stool blood). Although general CDS may have increasing importance in the future if the concept of the anesthesiologist-led perioperative surgical home is widely embraced by the specialty,28,29 we are concerned primarily with CDS relevant to modifying the behavior of anesthesia providers. CDS researchers outside of anesthesiology have largely ignored the contributions by the specialty to CDS (Table 1), with only passing references noted by Greenes.26,27

Table 1
Table 1:
Peer-Reviewed, Published Articles Related to Decision Support Using Data from AIMS

In this article, we discuss issues related to the development and implementation of AIMS-based systems designed to modify anesthesia provider performance. Our goal is to inform developers and users of CDS for AIMS about the multitude of concerns they should consider during development and implementation to increase effectiveness and mitigate potentially disruptive aspects of this technology. We provide 2 case studies from our own CDS system: one related to the reduction of fresh gas flow (FGF) during the administration of volatile anesthetics and another related to monitoring for interruptions in the continual recording of blood pressure (BP) during anesthesia. In our companion article, we consider the application of e-mail for the different process of operating room (OR) and anesthesia group managerial decision making.b


The 2 case studies that we describe are examples of CDS provided external to our AIMS (Innovian™; Dräger, Telford, PA). They required customized computer code for rule implementation, communication, and tracking of effectiveness. (The only native CDS capability in our AIMS is the ability to generate periodic workstation popups, which users must enable manually for each distinct reminder for each case.) Consequently, our examples and the presentation are relevant to any system in which near real-time access to the required electronic health record data is available because the data that feed the logic modules would be the same. Sending messages via workstation popups, alphanumeric pagers, cell phones, or e-mail are well-developed technologies that are not system dependent because the functions necessary to accomplish these tasks are included in modern relational databases. Our approach to build CDS outside of electronic health record systems is increasingly commonplace34 because of the large effort required to develop and maintain CDS rules26 and general lack of portability among proprietary systems.26

There is a trend in some countries (especially in the United States) for electronic health records to migrate from best-of-breed or homegrown software to enterprise-wide systems. However, the inherent CDS capabilities in these systems for AIMS are currently quite limited. We were unable to find a single research study that described use of the CDS modules within these systems for AIMS data.c Even outside of anesthesiology, there have been very few CDS studies published from enterprise-wide systems involving electronic health records.d In contrast, there have been many studies for which the developers of a local system had the ability to manipulate systematically the presentation of the CDS and determine its effectiveness (see the textbook by Greenes26 for a comprehensive bibliography), including applications specific to AIMS (Table 1).

When decision support systems are implemented in conjunction with an AIMS, factors that need to be considered include (1) the breadth of data sources (e.g., AIMS, pharmacy, laboratory), (2) modalities of message delivery (e.g., popup on the AIMS workstation, alphanumeric pager, e-mail),35–40 (3) latencies between event occurrence and message receipt,41,42 (4) timing of message delivery with respect to ongoing patient care (e.g., during, shortly after, or multiple days after the anesthetic has concluded),43 (5) frequency with which messages are retransmitted (e.g., once, a few times, or repeatedly until the desired behavioral change has occurred), (6) tracking of improvements (and unintended consequences),44 and (7) how the entire feedback system will be maintained through the lifecycles of the underlying AIMS and the communication systems.26 We consider these in subsequent sections for each of our 2 case studies.

Case Study 1: FGF Reduction

Our department desired to lower the cost of administering volatile anesthetics and to reduce the environmental impact of venting excessive amounts of such agents to the atmosphere.45,46 For example, our average sevoflurane FGF during the surgical portion of cases had been >2 L/min for the previous 3 years, with progressively increasing flow rates (Fig. 1).

Figure 1
Figure 1:
Total fresh gas flow (FGF) during sevoflurane anesthesia before initiation of feedback. The weighted average FGF was calculated for the interval between the beginning and end of surgery using four-week bins between March 7, 2010, and March 31, 2013. Periodic undulations in the FGF were noted, with peaks indicated by the vertical red lines, but with no apparent relation to external events. Increasing FGF is evident (P < 0.0001 by both Pearson linear and Spearman rank correlation tests).

Variation in mean FGF among anesthesia providers tends to be small.47 We previously found that substantive reductions in FGF were small reductions in FGF for the many cases with <3 L/min rather than large reductions in FGF for the few cases with FGF >3 L/min.47 Consequently, it was appropriate to provide feedback to all of the anesthesia providers, not just a few outlier individuals, because targeting outliers would not have materially reduced the overall mean FGF.

We initiated this project as a quality improvement effort within the department and subsequently received approval from our IRB to publish the data without requirement for written consent. Individualized, automated feedback to all anesthesia providers was provided approximately monthly via e-mail based on the most recent 10 cases they had performed. The data used were from our AIMS database, recorded at 1-minute intervals. For each volatile drug, the mean FGF was calculated for each case during the interval between the beginning and end of surgery. Using the duration of that interval, we calculated the simple average. Values outside this interval were excluded because greater FGF flows are indicated to facilitate the drug uptake or washout. Cases were excluded from calculations for any of the following reasons: (1) >1 volatile drug was used at a concentration ≥0.25 minimal alveolar concentration, (2) the drug concentration was <0.25 minimal alveolar concentration for the entire case, (3) the surgical duration was <15 minutes, (4) >1 resident or Certified Registered Nurse Anesthetist was involved in the case (for provider reports), (5) >1 supervising anesthesiologist was involved (for attending reports), or (6) the surgery start or end time was missing.

Figure 2
Figure 2:
Reduction in fresh gas flow (FGF) after implementation of individualized e-mail feedback. In April 2013, a baseline FGF report was sent via e-mail to all providers informing them of their individual FGF for each volatile agent during the most recent 10 cases performed with that agent. The e-mail outlined the department’s desire to reduce the average FGF for sevoflurane to 2 L/min and for desflurane and isoflurane to 1 L/min during the surgical portion of cases (i.e., surgery begin to surgery end). After initiation of the project, individualized reports were sent to each provider at approximately monthly intervals. No other attempts to reduce FGF were made during this time, and individual performance was not made public. The weighted FGF (open blue circles) was determined for each 4-week interval, and the data were fit (dark blue line) by a LOESS curve (span = 0.75). The blue shaded area around the fit represents the 99% uncertainty interval as estimated from the standard errors of the weighted polynomial least square fits around each of the data points. There was a significant reduction in FGF between the preintervention and the postintervention period (P < 0.0001 by Student’s t test with unequal variances and by Wilcoxon-Mann-Whitney U test).

The effectiveness of the FGF feedback system was assessed by calculating the weighted mean FGF calculated during each sequential 4-week interval (Fig. 2). Computational details are described in the section “Tracking of Improvement,” to follow. For purposes of this case study, only data for sevoflurane are provided because >90% of our volatile anesthetics are conducted with this drug.

Case Study 2: BP Gap Reduction

Measuring a BP at least every 5 minutes except “under extenuating circumstances” is an American Society of Anesthesiologists (ASA) basic monitoring practice standard.e Nevertheless, 10-minute gaps in such measurements occur commonly.20 One study demonstrated that providing near real-time feedback to providers reduces the occurrence of such gaps in the electronic anesthesia record,20 whereas a later study (using different alert criteria and excluding cases where BP was measured invasively) demonstrated a reduced incidence of >15-minute gaps but not 7–15 minute gaps.24 These gaps represent skill-based errors rather than knowledge-based mistakes,48f because all anesthesia providers should be aware of the ASA monitoring standards.e

Figure 3
Figure 3:
Long-term follow-up of a blood pressure (BP) gap alert system. The percentage of cases in each four-week bin with at least 1 occurrence of a gap in the measurement of BP >10 minutes is displayed as a p-control chart. A decision support system was implemented on July 1, 2008, that sent a popup to the anesthesia information management system workstation in the operating room where cases were ongoing when no BP had been recorded in the database for 10 minutes. The detection threshold was subsequently reduced to 6 minutes because of the ineffectiveness of the longer interval in reducing the frequency of cases with 10 minutes gaps in BP. Subsequently, performance was measured using a p-control chart, with the long-term mean defect rate (solid red line) estimated as the batch mean of the n = 14 four-week bins between July 1, 2009, and June 30, 2010. Control lines were established at
Table 1
. Data were fit to a LOESS curve (dark blue line, span = 0.2), with the shaded blue area representing 99% uncertainty in the fit as estimated from the standard errors of the weighted polynomial least square fits around each of the data points. Also plotted is the LOESS fit (dark green line, span = 0.2) to the number of alerts sent during each four-week bin, normalized to a weekly rate. During the follow-up interval of nearly 4 years, the process has been under control. Undulations in the frequency of the number of alerts sent each week correspond to the month of each year when clinical anesthesia year 1 residents begin taking care of patients in the operating room without continuous in-room supervision by a more experienced anesthesia provider.

We have been providing BP gap recommendations as described for Hospital B in the article by Ehrenfeld et al.20 In brief, at 1-minute intervals, a query is run on our AIMS server that detects all active cases (i.e., those currently transmitting pulse oximeter readings).42,49,50 All cases are identified in which an arterial BP reading (measured invasively or noninvasively) has not been recorded in the database within the previous 6 minutes.20 For such cases, a popup message is sent to the workstation where the affected case is being recorded, alerting the provider of the discontinuation of periodic BP determination. Effectiveness of the system has been assessed using a p-control chart methodology (Fig. 3).


Systems to alert clinicians wirelessly via text messages for abnormal physiologic, laboratory, and medication data have been available for nearly 20 years.51–53 Selection of the most appropriate modality through which to send AIMS-based messages depends on how the correct recipient will be determined and on the communication ranges of the devices. For messages sent after cases have ended (e.g., the next day), it is usually easy to determine the provider(s) involved in the care of the patient.

Who should get the message varies. For example, when documentation completed at the end of the case is missing (e.g., the presence or absence of any intraoperative complications), then the last provider on the case would get the message.17 Conversely, if the missing documentation relates to the start of the case (e.g., formulation of an anesthetic plan by the anesthesiologist), then the anesthesiologist present at induction would get the message. If a message needs to be sent while the patient is still undergoing anesthesia care, the provider logged in to the case would get the message, or a popup message could be sent to the workstation where care is being documented. Doing so avoids the limitation that it can be challenging to determine the anesthesia provider in the OR at any given moment, whether due to delays in recording such presence in the AIMS or movements of cases among rooms that are not documented in a timely manner in the OR information system or not updated in the AIMS.49 For this reason, in our AIMS decision support system, when we want to send a message to the in-room provider related to a specific case, we send a popup to the workstation that is transmitting pulse oximetry data for that case (because this signal is nearly always present for ongoing cases and is updated every minute in the database). This action avoids communication delays that might result if the correct provider were misidentified.

Several methods have been used to send messages based on AIMS data to providers (Table 1), including popup messages on the AIMS workstation (presented either from within or through software external to the AIMS software),20 text messages sent to alphanumeric pagers,18,21 e-mail notices,17 and combinations of the above.3,4 Reports of systems that used cell phone text messages as the primary message pathway were not found in our literature review (Table 1), likely a reflection of the lack of reliability of such methods (Fig. 4)39 and their potential to expose protected health information. However, some providers may forward system-generated text pages to their cell phones in preference to receiving them on their hospital-issued text pagers (Brian Rothman, Vanderbilt University, personal communication, 2013).

Figure 4
Figure 4:
Example of marked latency for cell phone text messages sent during a winter storm advisory on Tuesday, January 11, 2011, in Philadelphia. In this example, prolonged delays were noted for test Short Message Service messages being sent from Thomas Jefferson University Hospital in Philadelphia via a third party vendor (MessageMedia, San Francisco, CA) to a cell phone in Philadelphia between 3:15 PM and 8:15 PM as a winter storm moved into the area. One message (red circle) was never received. During the affected interval, cell phone voice communications were also severely disrupted. These data highlight the impact of high network traffic on Short Message Service message latency. Reprinted with permission from reference 39.

When e-mail is used for message delivery that contains protected health information, use of a secure (i.e., encrypted) delivery process is necessary if public Internet routing is involved. This may require recipients to log on to a remote server to retrieve their messages, as opposed to simply receiving them in the inbox of their e-mail client.

Case Study 1: FGF Reduction

Previous work from the late 1990s by Lubarsky et al.54 and Body et al.55 established the effectiveness of e-mail communications to providers with presentation of their individual gas flows compared with that of their colleagues in reducing FGF.

Nair et al.5 achieved FGF reductions by using their Smart Anesthesia Messenger™ (University of Washington, Seattle, WA), a proprietary decision support software system configured to provide agent-specific recommendations about FGF as popup messages on the AIMS workstation where care is being documented.

Although our AIMS decision support system has the same functionality as is present in Smart Anesthesia Messenger, and the implementation of near real-time FGF feedback would have been straightforward (i.e., <1 day of work), we elected first to try the e-mail approach. Under either scenario, we would have needed to write code to monitor compliance with the FGF objectives and to deliver feedback to our providers. The plan was that if we were not successful in meeting our goals with just the monitoring and notification process, we would then add near real-time feedback. Reasons for this strategy are explained in the sections to follow.

Our hospital mail server automatically detects the presence of potential protected health information in message content and routes such mail via ZixMail® (ZixCorp, Burlington, MA), an encrypted e-mail system. Recipients must login to a Web site to retrieve such messages. However, because no protected health information was included in the individualized FGF reports, messages were delivered via conventional e-mail. A sample e-mail is presented in the Appendix.

Case Study 2: BP Gap Detection

We had previously demonstrated that AIMS workstation popups reduced the incidence of cases with BP gaps >10 minutes. Control chart monitoring of performance demonstrated continued acceptable performance (see the section “Tracking of Improvement,” below). Therefore, we continued to provide such feedback solely by using our AIMS decision support system (Fig. 3).


Latency between the time that an event occurs and when the recipient receives the message related to that event is a factor that should be accounted for when assessing the performance of a “real-time” communication pathway. For data entered manually into the AIMS (e.g., tracheal extubation), there can be delays of many minutes from the time when the event occurs and when it is documented in the AIMS.37,43 In addition, there is a mean delay equal to half the scheduling interval from the time an event is recorded in the database until it is detected.41 Depending on the architecture of the AIMS (e.g., local storage and periodic synchronization with the central database versus a direct connection, next-day availability in a data warehouse), there can be substantive delays from the time of documentation at a workstation to when the event is available in the database.

There also can be considerable delays from message transmission to receipt by the anesthesia provider. The latter has been described with cell phone text messaging during periods of network congestion (Fig. 4)39 but also can occur unpredictably when using both 1-way and 2-way alphanumeric paging devices that rely on the Internet as part of the transmission pathway.38,39 Use of text-paging devices running entirely within the hospital’s network infrastructure or use of the Apple Push Network greatly mitigates the potential for such substantial delays.38,39 (Google Cloud Messaging for Android might also be a reliable message delivery mechanism but has not been rigorously studied.) For most applications, the terminology “near real time” is preferred to “real time” and expected latencies with upper percentiles of performance provided.38,41

For example, we evaluated the maximum potential utility of providing alphanumeric pages to anesthesiologists when patients whose care they were supervising experienced an episode of oxygen saturation <90% for ≥2 minutes.56 Only 8% (99% confidence interval, 6%–9%) of such episodes would not already have been resolved by the in-room provider within 3 minutes of the onset of the episode. Wickens and Dixon57 showed that for a decision support alert system to be effective, the minimum threshold for utility is 70%. Consequently, we elected not to implement this hypoxemia alert system.

Our research group also evaluated the potential utility of providing alerts to nurse managers to mitigate delays in patients leaving the OR when the postanesthesia care unit (PACU) was close to full census.58 At most 23% (95% upper confidence limit) of such alerts sent when the census was 1 less than maximum would have allowed the PACU nurse manager to avoid a subsequent PACU delay (with assumptions favoring utility of the system). Again, this was far lower than the 70% threshold required for utility.57 Consequently, we also elected not to implement this PACU census management system.

From these examples, we recommend strongly to organizations that they not implement alert systems unless proven useful by researchers or after performing their own rigorous scientific evaluation. Lack of utility should be assumed until testing shows otherwise, even if a benefit seems apparent.

Hospital-issued alphanumeric pagers have variable ranges that need to be considered when messages are sent. For example, if the device only works within a limited distance (e.g., 10 miles) from the hospital, alphanumeric pages sent during non-working hours may not be received depending on where the provider lives. A message sent but not received is equivalent to infinite latency. Issues related to communication “dead zones” within the hospital also need to be addressed (e.g., lack of reception in a subterranean area). Sometimes, repeaters need to be installed or settings of broadcast transmitters or antennae adjusted. For e-mail communications, distance is not a concern, but there can be delays in receipt of the message, depending on whether the provider needs to login to get his or her messages or if they are “pushed” to the provider’s smartphones. Messages sent internally within the hospital’s network infrastructure generally do not need special considerations related to protected health information in the message content, because the messages will exist only inside the hospital’s firewall. However, for messages sent outside the hospital Intranet (e.g., the routing for many alphanumeric paging vendors),39 issues related to the clear text transmission of protected health information over the Internet need to be considered.39

Case Study 1: FGF Reduction

Because our FGF feedback mechanism involved monthly reporting, latency from when FGF values were measured by our anesthesia machines and when written to the database was not a concern. However, for daily reporting at facilities for which AIMS data are not available for creating reports until the next day, the time of day when the data are extracted would need to be considered. Even for near real-time reporting, latency needs to be considered. For example, in the previously described Smart Anesthesia Messenger system, the maximum data latency was 5 minutes.9 Setting the frequency of feedback lower than the maximum latency of data availability will result occasionally in missing interim changes and transmission of an incorrect alert.

Case Study 2: BP Gap Detection

In our AIMS, the upper 95th percentile for the time that vital sign data are recorded at the AIMS workstation until available in the database was determined previously to be 57 seconds (median = 30 seconds).41 The popup message software that runs on our AIMS workstations polls the database for messages every 60 seconds, resulting in an average delay in the time when a message is generated until it appears of 30 seconds. The query that looks for active cases (i.e., where oxygen saturation values are being transmitted to the database) runs every 1 minute. Because our default repeat cycle for noninvasive BP measurement is 3 minutes, 2 consecutive measurements typically need to fail before an alert will trigger. However, because of the latencies involved, occasionally there will be an alert sent that collides with the acquisition of a BP after resumption of monitoring. Thus, the tone of the popup is that of a friendly suggestion to check the monitor, not an accusation of substandard care.


Fundamentally, each message delivered constitutes an interruption that, while providing useful feedback to providers, has the potential to affect task performance adversely if the message occurs during the course of clinical care.36,59–61 Research in human−computer interactions has shown that both the complexity and timing of the underlying tasks62,63 influence the effect of interruptions.63 Interruptions have been characterized as immediate (i.e., occurring without considering the individual’s current activity and yet requiring immediate attention), scheduled (i.e., occurring predictably at a fixed time, potentially allowing planning around the time of the anticipated interruption), mediated (i.e., occurring via software that selects the least intrusive time for the interruption based on knowledge of the individuals’ activities), or negotiated (i.e., allowing the individual to defer the interruption into the future).

Distractions such as conversation, music, and equipment issues have been shown experimentally to cause decrements in task performance by surgical interns performing laparoscopic cholecystectomy64 and medical students performing simulated endouroscopy procedures.65 Distractions caused by immediate interruptions (e.g., incoming pages and phone calls) decreased (actual) completion rates of patient safety–related tasks by surgeons performing elective urologic procedures, with correlation to the frequency and severity of the distraction.66 Gillespie et al.67 demonstrated prospectively an inverse correlation between miscommunications among members of surgical teams and the number of interruptions. Numerous studies have addressed the real and potential adverse implications of paging interruptions.61,68,69

We evaluated the optimal timing of interruptions of anesthesia providers in ORs based on workload inferred from AIMS documentation.41 In that study, the interval most suitable for mediated interruptions of the anesthesia provider was found to be 13 minutes after surgical incision until the end of surgery.41 Results were minimally sensitive to considerations such as the type of anesthesia, type of procedure, positioning, and the scheduled case duration.41 The interval from entering the OR through the start of surgery was not suitable for mediated interruptions, consistent with previous observations that induction is a period of high workload.70–73

Feedback systems often use several methods for communication.3,4 For example, in our process that checks to see whether documentation of intraoperative complication(s) is missing, a text message is sent 30 minutes after the patient leaves the OR to the pager of the anesthesia provider who was documented in the AIMS as present at the end of the case. The next morning at 6 AM, an e-mail message is sent to this provider if the documentation is still incomplete. The quality assurance director of the department also gets an e-mail each morning summarizing complications from the previous day. This information is used as a potential trigger for further investigation and as the source of cases for our monthly morbidity and mortality conferences. We maintain our list of alphanumeric pagers and the hospital e-mail addresses in the staff table of our AIMS database.

Considerations related to the timing of message delivery have been absent from most published studies of intraoperative feedback. For some types of messages (e.g., an alert that the BP cuff is not cycling),20 delaying the message would be inappropriate. However, messages about other types of issues (e.g., documentation or billing) can be deferred without adverse clinical effect. Developers of notification systems in which messages are to be delivered while clinical care is being provided should consider the potential unintended consequences of interrupting providers. In our own implementations of intraoperative decision support, we wait until 15 minutes after incision to provide messages related to missing documentation to accommodate the high workload immediately following induction of anesthesia.41

Case Study 1: FGF Reduction

In our FGF feedback system, to reduce the risk of causing an interruption during clinical care for providers who are alerted by their smartphones when e-mail arrives, we chose to send the messages on Mondays at 5:00 AM.

Case Study 2: BP Gap Detection

Because the first BP usually is not measured until several minutes after patients enter the OR, we set a delay from the time of OR entry until the earliest time that a BP gap alert would be sent. We initially set this at 10.2 minutes, which was the 95% upper confidence limit for the time from OR entry until the first BP was taken, but this resulted in complaints from providers about false alerts at the start of some cases. The initial delay was subsequently adjusted to 15 minutes. Once the first BP in the OR is measured, no attempts are made to mediate the interruption from a BP gap alert because subsequent gaps were typically indicative that the BP monitor is set to its manual rather than automatic mode.


Another consideration when designing a CDS system is how many times and at what intervals the messages should be rebroadcast. Some reminders are configured to send a single message.21 Other systems are implemented to continue to page providers periodically during cases and then in an escalating manner for several days or until the missing documentation is completed.3,4 The more aggressive notification protocols in anesthesia have been used for missing documentation necessary for billing purposes.3,4 Alert systems generally should be configured to allow the provider to indicate for apparent noncompliance, so that unnecessary messages can be avoided.

Case Study 1: FGF Reduction

At the start of the quality improvement project, 2 e-mails were sent to all clinical staff explaining the objectives of the program and providing each provider with his/her mean FGF for the most recent 10 cases performed with each volatile agent. Subsequently, personalized e-mails were sent monthly. Other than a brief announcement at Grand Rounds when the project started, no other special educational efforts were made, and no personal contacts were made to providers with high FGF to influence their behavior.

Case Study 2: BP Gap Detection

Figure 5
Figure 5:
Embedded popup from an anesthesia information management system. A popup generated within MV-OR (iMDSoft, Needham, MA) when a blood pressure was not detected within the previous 10 minutes is shown. The provider has the ability to defer or acknowledge the message and optionally provide an explanatory comment. In this implementation, a text message was also sent to the provider’s alphanumeric pager if receipt of the message was not acknowledged. Reprinted with permission from Figure 3B in Ehrenfeld et al.20

A stored procedure in the database that looks for gaps between successive BP readings exceeding the specified threshold for cases currently in the OR runs once a minute. A popup message (similar to Fig. 5) is then sent to each AIMS workstation where this is detected. The provider is asked to click an acknowledge button, but the message can be dismissed without action. The AIMS can continue to be used even with the popup on the screen (i.e., response to the message can be deferred). If no acknowledgement is received and a BP reading is not recorded subsequently, a second (and final) message is resent 5 minutes later. After resumption of BP recording after a gap, the system is reset so that a new BP gap will trigger an alert.


Many studies on improvements attributable to implementation of AIMS-based alert systems have relied on before and after analyses (Table 1). Typically, a baseline period is analyzed for compliance with the metric being studied. Then, after implementation of the new system, compliance is assessed again and compared with the baseline. Such study designs are problematic because of the absence of randomization and unknown interceding events that might have improved compliance even without implementation of reminder messages. For example, department newsletter announcements, e-mails to providers, Grand Rounds presentation(s), publication(s) in anesthesia journals, or public posting of provider performance occurring shortly before or after implementation of a message reminder system may be responsible for improvements seen, not the messaging system. Ideally, if one wishes to measure the impact of a new messaging system, the system is implemented with no (or at most minimal) warning and care is taken not to provide additional educational material in parallel with the new system. This was, for example, how we approached popup messages inquiring of the time remaining in late running surgical cases.42

Another approach is to cycle the messaging system on, then off, then on again, and to determine whether there was improvement, followed by decrement in performance, followed by improvement. This was the approach followed to evaluate the utility of a near real-time alert system designed to reduce FGF during delivery of volatile anesthetics.5 When such study designs are followed, to avoid carryover effects, it is important that sufficient time be allowed to elapse between interventions (“washout period”) so that the outcome being measured returns close to the original baseline.74 Some investigators have randomized the delivery of messages to providers, showing a small, but statistically significant (P = 0.032) reduction in mean delivered tidal volume for providers receiving an alert.21

A second problem that often occurs when trying to assess the impact of a decision support system is that there may be clustering of events as a function of time. Simply computing means and SDs before and after the intervention is suspect because assumptions of the statistical independence of events are usually violated. This typically occurs because both cases and staff are scheduled, and these schedules vary among days of the week and dates of the year (e.g., holidays).75–77 Activity in 1 OR does affect care in another OR because, although the anesthesia provider is not shared, supervising anesthesiologists, anesthesia technicians, housekeepers, and PACU nurses are shared.78–80 Under such circumstances, the effect of this autocorrelation can be eliminated by combining results in suitable intervals (e.g., 4-week intervals) and using the method of batch means.40,81–85

A third issue is that during the period before and/or after the intervention is made, there may be changes over time, and these sometimes need to be modeled in the analysis (Fig. 2). Graphical methods are used ubiquitously because they do not assume specific functional forms (e.g., linearity). To obtain P values and confidence intervals, techniques such as segmented regression, also known as “piecewise regression,” can be used.86,87 The 2 references provided for this method are review articles with the (straightforward) equations and computer code examples.86,87 However, a limitation is that often changes in time are not described well by linear segments.86,87 Consequently, the batched data are often analyzed before and after using Student’s t test with unequal variances (i.e., Satterthwaite’s correction), a method robust to non-normal distributions.44,83

Case Study 1: FGF Reduction

Results from our FGF intervention are shown in Figure 2. The average weighted FGF was computed for all cases in each four-week bin measured from the onset of the project on April 1, 2013. The weighted FGF is equal to the sum for each case of the product of the average FGF and the duration of the measured values, divided by the total duration among all cases. To understand why it is necessary to compute the weighted FGF rather than the typically reported average FGF when calculating benefit, consider the following scenario. One case has a FGF of 3 L/min that is administered for 20 minutes. A second case has a FGF of 1 L/min administered for 100 minutes. From the cost and environmental perspective, the average FGF is not 2 L/min (= (3 + 1) / 2) but rather 1.33 L/min (= (3 × 20 + 1 × 100) / (20 + 100)).

Using Student’s t test with unequal variances, as well as the Wilcoxon-Mann-Whitney U test, there was a significant reduction in FGF between the pre-intervention and the post-intervention period (both P < 0.0001). When the long-term trend in FGF over the 3 years before the start of the FGF reduction project was examined (Fig. 1), increasing FGF over time was evident (P < 0.0001 by both Pearson linear and Spearman rank correlation tests), albeit with multiple undulations (periodicities). To what extent the progressive increase in FGF over time (Fig. 1) would have continued without the intervention is not knowable from the data.

Case Study 2: BP Gap Detection

Ongoing monitoring following our intervention in July 2008 to reduce the incidence of gaps in the recording of BP20 is shown in Figure 3 as a p-chart, a statistical quality control graphical method for monitoring the fraction of defects in a sample when the defect rates are low.g Data from n = 14 four-week bins (July 1, 2009, to June 30, 2010) were used to estimate the long-term mean defect rate

(batch mean of the proportion of cases with ≥1 BP gap >10 minutes per total number of cases). Control lines were calculated as

. Also plotted is the LOESS (locally weighted scatterplot smoothing) fit (span = 0.2) to the number of alerts sent during each four-week bin, normalized to a weekly rate. These results demonstrate that (1) the process has been under control for the past 4 years since the BP gap detection threshold was reduced to 6 minutes; (2) there is periodicity among four-week bins in both the proportion of cases with BP gaps and the number of alerts sent; and (3) the numbers of alerts sent weekly have not decreased substantially over time. This finding suggests a lack of long-term learning by providers to avoid interruptions in BP measurement and a need to continue to provide the alert system.


One aspect of implementing a CDS system is the potential for unintended consequences. For example, a personalized quarterly reporting system to anesthesia providers about their compliance with administration of prophylactic antiemetic medications to high-risk patients resulted in improved compliance with the protocol but also substantial overtreatment.14 An accompanying editorial by one of the authors of this review (R.H.E.) raised concern about potential US Food and Drug Administration (FDA) medical device regulatory issues related to implementation of real-time decision support making clinical recommendations.88

The FDA recently published an update to its September 2013 Guidance for Industry and Drug Administration Staff for Mobile Medical Applications.h This document indicates that regulatory oversight will be applied to mobile devices used as accessories to regulated medical devices or which transform a mobile platform into a regulated medical device. The later includes devices that control the operation of medical devices such as BP monitors or insulin infusion pumps or have attached or embedded sensors that perform medical functions (e.g., accelerometers to monitor sleep apnea or electrodes to electrocardiogram signals).

US regulation of CDS software is currently in a state of transition. In April 2014, the FDA, the US Federal Commerce Commission, and the Office of the National Coordinator for Health Information Technology issued a jointly prepared report that outlined a “proposed strategy and recommendations on an appropriate, risk-based regulatory framework pertaining to health information technology, including mobile medical applications, that promotes innovation, protects patient safety, and avoids regulatory duplication.”i Under the proposed framework, most CDS software would be subject to nonregulatory oversight by Office of the National Coordinator for Health Information Technology. Quality management principles, adoption of standards and best practices, and establishment of a culture of safety and quality would be stressed. Examples of functions that would fall in this category would be computerized order sets tailored to specific conditions, diseases, or clinician preference; detection of drug−drug and drug−allergy interactions; drug calculators; calculation of prediction rules; duplicate testing alerts; and suggestions of possible diagnoses based on information retrieved from the electronic health record of a patient. However, other categories of decision support that represent higher risk would continue to fall under FDA regulation. Examples particularly relevant to anesthesia and intensive care include computerized detection/diagnostic software, remote display or notification of real-time alarms (physiological, technical, or advisory) from bedside monitors, and electrocardiography analytical software.

Further confusing the US regulatory framework, the FDA published a Draft Guidance on June 20, 2014, indicating its intention not to enforce compliance with regulatory controls applicable to certain categories of Medical Device Data Systems and certain medical image storage and communications devices.j The guidance, adopted as of February 9, 2015, is nonbinding upon the FDA. Essentially, software whose function is to redisplay medical device data (including on portable devices) will not be regulated, but software in which active patient monitoring is involved will remain under regulation. By active patient monitoring, the FDA means assessments are made in real time using remotely acquired data and that “immediate clinical action” may be needed. From this, it appears that regulation of software displaying data in near real time from OR patient monitors on smartphones89 might not be enforced by the FDA, even though such software falls under its “Guidance for Industry and Drug Administration Staff for Mobile Medical Applications.”h

It is unclear at present how the European Union (EU) Medical Device Directive 2007/47/EC will be modified to reflect the proposed changes in FDA policy, but both the European Union and FDA appear not to be changing their oversight regulations regarding devices performing active patient monitoring.k

As an example of the impact of the potential US regulatory framework on anesthesia, we consider the VigiVU system, developed at Vanderbilt University to remotely display intraoperative vital signs from patient monitors on either a desktop or iOS device (e.g., iPhone, iPad). VigiVu would likely fall outside of regulatory compliance under the new guidance. However, if the remote system triggers an alert (to which the anesthesiologists might respond by returning immediately to the OR or calling the room to institute a therapeutic change), perhaps that would mean that the software is performing active patient monitoring and thus subject to regulation. Clearer interpretation can be made for displaying real-time popup messages on the AIMS workstation to reduce phenylephrine infusions and lower volatile anesthetic concentrations based on BPs.25 That likely would be subject to FDA regulation as a medical device because active patient monitoring is involved.

In contrast, sending reports after the fact to providers regarding their past performance raises no regulatory concerns because the report cannot influence clinical behavior on the cases that have already concluded.88 Organizations will need to decide whether the benefits of an immediate feedback alert system justifies the potential burden of a regulatory submission. As a potential alternative, for the period of clinical trials of a device representing a low risk of patient harm, each local US IRB can issue an investigational device exemption, allowing use under the context of the normal human studies oversight process.

Recent action by the FDA indicates that that the agency intends to intervene when CDS software may produce a significant risk of patient harm. For example, on March 14, 2014, the FDA advised McKesson that a Class I recall was indicated for its anesthesia care software (which provides decision support involving notification of potential adverse drug reactions) due to a reported substitution error of 1 patient’s data for another.l Although the index patient was not injured, this incident was interpreted by the FDA as a serious safety issue because incorrect decisions could be made based on erroneous information being provided. For example, if a patient has a history of an anaphylactic reaction to cephalosporins and “no drug allergies” is transmitted due to a substitution error, the provider might give cefazolin as the prophylactic antibiotic, with serious consequences.

Case Study 1: FGF Reduction

The FGF system we implemented was designed to avoid concerns related to the potential creation of a medical device. Sending individualized reports of historical FGF by e-mail to providers is simply an administrative function designed to enhance overall compliance with a departmental quality practice objective, not an effort to dictate care for individual patients. In the monthly FGF e-mails, we emphasized the flow-rate restrictions in the package labeling related to sevoflurane and emphasized the need to pay attention to end-tidal volatile agent gas concentrations when running low flows to avoid the potential for inadvertently providing an inadequate depth of anesthesia.

We noted a few instances of providers running low FGF during administration of total IV anesthesia in the presence of partial exhaustion of the carbon dioxide adsorbent. In Rasmussen’s48 taxonomy of cognitive theory applied to the design of human−machine interface systems, this example would be classified as a rule-based mistakef because FGF reduction is not relevant for such cases, and misapplication of the rule resulted in unintended rebreathing of carbon dioxide.

Case Study 2: BP Gap Detection

It is unclear to us what level of regulatory oversight (if any) that would be required under the proposed risk-based regulatory framework for our BP gap detection system. On one hand, we are not remotely displaying values or alerts from bedside patient monitors, a process covered by regulation within the framework. Rather, we are detecting the absence of an expected value in the database. On the other hand, we are sending an alert to the AIMS workstation, and regulatory bodies might be concerned about undelivered messages resulting from software failure at multiple points (e.g., the workstation application that looks for messages to display or the query on the server that processes the data and generates the message). However, because there is no other alarm in the OR that will inform the anesthesia provider that the BP monitor has stopped automatic cycling, we have elected to keep the process running until there is further clarification. Figure 3 shows that there are scores of events weekly for which the BP is not being recorded for longer than the ASA-mandated monitoring interval of 5 minutes.


Implementation of a real-time decision support system using AIMS data is an important process that many organizations may find challenging to undertake. For example, multiple US organizations that have implemented the EPIC enterprise-wide electronic health record (EPIC, Madison, WI) have thus far found accessing the real-time AIMS data in the system’s proprietary Caché database to be difficult. As summarized in the introduction to this review, at the time of submission of this paper, there had not been a single AIMS study published in a journal indexed by PubMed from an enterprise-wide system.c Because data warehouses (e.g., EPIC’s Clarity database) are populated infrequently (usually once a day), near real-time decision support requires access to the primary data tables in the AIMS. Cross-platform database access from multiple systems may need to be accessed if the organization does not have an enterprise-wide product (e.g., the OR information system and the AIMS are from separate vendors).

Decision support systems also need to be maintained throughout version upgrades of the AIMS and changes to message delivery infrastructure addressed. When new processes are added to the decision support system or changes are made to the underlying system (e.g., software upgrades), regression testing is required to ensure that database schema changes do not break existing CDS routines. Underlying CDS rules need to be curated and modified, as indicated by changes in medical knowledge.27 Greenes26 describes 3 “intersecting and interacting life cycles” relevant to CDS: (1) knowledge generation and validation; (2) CDS method development and refinement; and (3) knowledge management and dissemination.

Systems providing near real-time support also, by definition, need to have technical support available 24 hours a day, 7 days a week, with knowledgeable analysts and developers available on-call to fix or mitigate any problems that occur. If messages are sent to alphanumeric pagers or other devices, those systems need to be monitored and supported, which will often require coordination with external vendors’ technical support teams. Help desk and technical support agreements should be written into the contracts for such systems to ensure rapid resolution of problems. The complexities and expense of maintaining near real-time decision support systems should not be underestimated.m

In contrast, individualized feedback to providers after the case is completed is much less labor intensive because it only involves creating a report and enabling a delivery method. There is no need for 24 × 7 support because it does not matter if a performance report goes out a few days later than scheduled (e.g., if a software bug needs to be corrected by a data analyst or there is a temporary problem with the e-mail system). Because the reporting system would typically be executed against a data warehouse, version updates to the AIMS would be expected to deal with continuing to populate that database. Even if the underlying AIMS or other data sources changed, the field names accessed by the reporting system would not need to change; thus, reporting systems are generally portable between software upgrades and even system changes.

Case Study 1: FGF Reduction

Our e-mail notification system is automatic and relies on the hospital’s outbound mail server, which is monitored and maintained by the Information Systems department. A stored procedure written in Transact-SQL (Microsoft, Redmond, WA) on our AIMS server pulls the data for all providers and computes the average FGF for each volatile agent. The job is scheduled to run every fourth Monday at 5 AM. Hospital e-mail addresses for our providers are stored in the staff table in our AIMS database. An account was established with permission to use the hospital’s outbound mail server, which is called from the SQL code. If for some reason the procedure fails, an e-mail alert and text page are sent to the system administrators. When new providers join the department, their correct e-mail address must be entered into the AIMS database, which is specified in the AIMS procedure manual. Otherwise, they will not receive the reports.

If we were to move from our existing AIMS platform, we would need to rewrite the FGF stored procedure to access the new data source. We would also need to rewrite our code for all of our other feedback reports.

Case Study 2: BP Gap Detection

Maintenance of this system is much more involved than the FGF notification system because messages are provided in near real time. We have had to solve issues related to database record locking due to multiple simultaneous data monitoring processes running on our AIMS server, software interactions on the AIMS workstations that result in a failure to receive messages, and providers changing the names of labels on the patient monitor from the expected values, resulting in false alerts that the BP was not being measured. If the monitoring procedure in the database fails, an alert is sent automatically to system administrators.


Decision support systems providing near real-time alerts and/or post hoc reports (e.g., via e-mail) based on AIMS data typically result in better adherence to protocols and increase financial performance through facilitation of billing, regulatory, and compliance documentation. In the 21 studies summarized in Table 1 that were evaluable for measured improvement, a statistically significant benefit was shown in 20.

When considering implementing or adding to decision support systems, developers should consider steps to reduce potential deleterious impacts due to inopportune interruptions during the course of clinical care. Modalities to deliver messages should be appropriate to the content of the message and the speed with which an intervention might need to be made. Timing of messages should be considered, and if message delivery can be safely deferred, attempts should be made to avoid interruptions to clinical care. Support hours and maintainability of systems need to be incorporated in the process implemented, with near real-time systems requiring considerably more resources than report-based systems. As a general recommendation, if the same goal can be accomplished through periodic, individualized reporting than can be accomplished via a real-time feedback system, the former is nearly always preferable. Nonetheless, many processes related to perioperative care do require near real-time feedback. In order for processes requiring immediate feedback to become widely used by anesthesia providers, such processes need to become a core function of electronic health systems that are vendor supported and facilitated by intelligent database and software design. In the meantime, individualized feedback to anesthesia providers’ performance delivered the next day or later remains a viable method of improving many perioperative measures.


Sample of the FGF E-Mail Sent to Providers

To: William Morton

We are trying to reduce the environmental impact of excessive fresh gas flows, as such flows result in large amounts of volatile anesthetics being vented into the atmosphere. Another benefit of this effort will be a modest reduction in our volatile agent cost.

We are therefore asking everyone to reduce the total fresh gas flows during the interval from Surgery Begin to Surgery End to result in the following average FGF:

  1. Desflurane 1.00 L/min
  2. Sevoflurane 2.00 L/min
  3. Isoflurane 1.00 L/min

Please note that these targets exclude induction and emergence times when higher flows are likely. Also excluded are cases where the duration of volatile agent administration was 15 minutes or less.

As you know, DES can be administered at much lower flows than 1 L/min. SEV can be administered safely at 1 L/min for 2 MAC hours, then the FGF increased to 2 L/min.

When running low flows, be sure to pay attention to the vaporizer setting and the ET gas concentration, as the latter may be substantially below the vaporizer setting until equilibration occurs. This is affected by agent solubility and the internal mechanics of the Apollo anesthesia machine.


Personalized FGF report for William Morton for Sevoflurane

During the last 10 cases you performed with SEV, your average FGF during the interval from Surgery Begin to End was 2.49 L/min

Your flow rate was 24.3% higher than our target of 2.00 L/min*

Please try to reduce the flow rates during maintenance for your subsequent SEV cases.


Rich Epstein

Dave Maguire

* If the provider’s FGF was ≤101% of the target, this line was replaced with:

“Congratulations on reaching the goal line. Keep up the good work!”

If the provider’s FGF was ≤110% of the target the line, this line was replaced with:

“You are almost at the goal line. Thank you for your efforts so far.”

(FGF = fresh gas flow; MAC = minimal alveolar concentration.)


Dr. Franklin Dexter is the Statistical Editor and Section Editor for Economics, Education, and Policy for Anesthesia & Analgesia. This manuscript was handled by Dr. Maxime Cannesson, Section Editor for Technology, Computing, and Simulation for the Journal, and Dr. Dexter was not involved in any way with the editorial process or decision.


Name: Richard H. Epstein, MD.

Contribution: This author helped design the study, conduct the study, analyze the data, and write the manuscript.

Attestation: Richard H. Epstein approved the final manuscript.

Name: Franklin Dexter, MD, PhD.

Contribution: This author helped design the study, conduct the study, analyze the data, and write the manuscript.

Attestation: Franklin Dexter approved the final manuscript.

Name: Neil Patel, MD.

Contribution: This author helped design the study, analyze the data, and write the manuscript.

Attestation: Neil Patel approved the final manuscript.


a A complete, annotated AIMS bibliography is maintained at Accessed March 12, 2015.
Cited Here

b Prahl A, Dexter F, Van Swol L, Braun MT, Epstein RH. E-mail as the appropriate method of communication for the decision maker when soliciting advice for an intellective decision task. Submitted for publication. 14-01111R1.
Cited Here

c On July 28, 2014, the following PubMed search was executed for publications potentially related to the topic of the anesthesia module in EPIC, Cerner, or McKesson and decision support: (EPIC[All Fields] OR Cerner[All Fields] OR McKesson[All Fields]) AND (“anaesthesia”[All Fields] OR “anesthesia”[MeSH Terms] OR “anesthesia”[All Fields] OR “anaesthesia”[All Fields] OR “anesthesia”[MeSH Terms] OR “anesthesia”[All Fields] OR “CDS”[All Fields] OR “Decision Support”[All Fields] OR “Decision-Support”[All Fields]) Of the 47 articles found, none were relevant to CDS.
Cited Here

d On July 28, 2014, the following PubMed search was executed for publications potentially related topic of the EPIC, Cerner, or McKesson and decision support: (EPIC[All Fields] OR Cerner[All Fields] OR McKesson[All Fields]) AND (“CDS”[All Fields] OR “Decision Support”[All Fields] OR “Decision-Support”[All Fields]) Of the 32 articles found, only 5 were relevant. Antibiotic stewardship was discussed in 2, 1 related to computerized provider order entry (with very limited CDS), 1 was an editorial related to the lack of ability of current EHRs to support personalized medicine, and 1 related to sharing CDS rules in a Web 2.0 accessible framework.
Cited Here

e American Society of Anesthesiologists. Standards of the American Society of Anesthesiologists: standards for basic anesthetic monitoring. Available at: Accessed March 12, 2015.
Cited Here

f Duke University Medical Center. Human Performance. Available at: Accessed March 12, 2015.
Cited Here

g Wikipedia Foundation. p-chart. Available at: Accessed March 12, 2015.
Cited Here

h Mobile Medical Applications. Guidance for Industry and Food and Drug Administration Staff. Available at: Accessed March 12, 2015.
Cited Here

i FDASIA Health IT Report. Proposed Strategy and Recommendations for a Risk-Based Framework. Available at: Accessed March 12, 2015.
Cited Here

j Food and Drug Administration. Medical Device Data Systems, Medical Image Storage Devices, and Medical Image Communications Devices— Guidance for Industry and Food and Drug Administration Staff. Available at: Accessed March 12, 2015.
Cited Here

k Klümper M, Vollebregt E. Navigating the new EU rules for medical device software. Available at: Accessed March 12, 2015.
Cited Here

l Lessons from McKesson: Insight into FDA’s Stance on Clinical Decision-Support Software. Available at: Accessed March 12, 2015.
Cited Here

m A brief support scenario is provided. A dashboard application (written in Microsoft VB.Net) is used by our anesthesia OR directors to make operational decisions related to running the OR, moving cases, and assigning staff. They are highly dependent on this application, as they have used it daily since 2007. Early in Summer, 2014, OR locations stopped being updated, prompting an urgent call to RHE. The software first had to be debugged to determine which of the multiple stored procedures called by the application had broken. Then, that stored procedure needed to be tested to find the source of the problem. The issue was that a patient had been admitted with a 3-word first name and a 3-word last name, and this exceeded the length of the patient name field in one of the temporary tables created by the stored procedure. This was corrected, and the application began working properly.
Cited Here


1. Stol IS, Ehrenfeld JM, Epstein RH. Technology diffusion of anesthesia information management systems into academic anesthesia departments in the United States. Anesth Analg. 2014;118:644–50
2. Lubarsky DA, Sanderson IC, Gilbert WC, King KP, Ginsberg B, Dear GL, Coleman RL, Pafford TD, Reves JG. Using an anesthesia information management system as a cost containment tool. Description and validation. Anesthesiology. 1997;86:1161–9
3. Spring SF, Sandberg WS, Anupama S, Walsh JL, Driscoll WD, Raines DE. Automated documentation error detection and notification improves anesthesia billing performance. Anesthesiology. 2007;106:157–63
4. Kheterpal S, Gupta R, Blum JM, Tremper KK, O’Reilly M, Kazanjian PE. Electronic reminders improve procedure documentation compliance and professional fee reimbursement. Anesth Analg. 2007;104:592–7
5. Nair BG, Peterson GN, Neradilek MB, Newman SF, Huang EY, Schwid HA. Reducing wastage of inhalation anesthetics using real-time decision support to notify of excessive fresh gas flow. Anesthesiology. 2013;118:874–84
6. Freundlich RE, Barnet CS, Mathis MR, Shanks AM, Tremper KK, Kheterpal S. A randomized trial of automated electronic alerts demonstrating improved reimbursable anesthesia time documentation. J Clin Anesth. 2013;25:110–4
7. O’Reilly M, Talsma A, VanRiper S, Kheterpal S, Burney R. An anesthesia information system designed to provide physician-specific feedback improves timely administration of prophylactic antibiotics. Anesth Analg. 2006;103:908–12
8. Wax DB, Beilin Y, Levin M, Chadha N, Krol M, Reich DL. The effect of an interactive visual reminder in an anesthesia information management system on timeliness of prophylactic antibiotic administration. Anesth Analg. 2007;104:1462–6
9. Nair BG, Newman SF, Peterson GN, Wu WY, Schwid HA. Feedback mechanisms including real-time electronic alerts to achieve near 100% timely prophylactic antibiotic administration in surgical cases. Anesth Analg. 2010;111:1293–300
10. Nair BG, Peterson GN, Newman SF, Wu WY, Kolios-Morris V, Schwid HA. Improving documentation of a beta-blocker quality measure through an anesthesia information management system and real-time notification of documentation errors. Jt Comm J Qual Patient Saf. 2012;38:283–8
11. Kooij FO, Vos N, Siebenga P, Klok T, Hollmann MW, Kal JE. Automated reminders decrease postoperative nausea and vomiting incidence in a general surgical population. Br J Anaesth. 2012;108:961–5
12. Kooij FO, Klok T, Hollmann MW, Kal JE. Decision support increases guideline adherence for prescribing postoperative nausea and vomiting prophylaxis. Anesth Analg. 2008;106:893–8
13. Schwann NM, Bretz KA, Eid S, Burger T, Fry D, Ackler F, Evans P, Romancheck D, Beck M, Ardire AJ, Lukens H, McLoughlin TM. Point-of-care electronic prompts: an effective means of increasing compliance, demonstrating quality, and improving outcome. Anesth Analg. 2011;113:869–76
14. Frenzel JC, Kee SS, Ensor JE, Riedel BJ, Ruiz JR. Ongoing provision of individual clinician performance data improves practice behavior. Anesth Analg. 2010;111:515–9
15. Kappen TH, Moons KG, van Wolfswinkel L, Kalkman CJ, Vergouwe Y, van Klei WA. Impact of risk assessments on prophylactic antiemetic prescription and the incidence of postoperative nausea and vomiting: a cluster-randomized trial. Anesthesiology. 2014;120:343–54
16. Vigoda MM, Lubarsky DA. The medicolegal importance of enhancing timeliness of documentation when using an anesthesia information system and the response to automated feedback in an academic practice. Anesth Analg. 2006;103:131–6
17. Vigoda MM, Gencorelli F, Lubarsky DA. Changing medical group behaviors: increasing the rate of documentation of quality assurance events using an anesthesia information system. Anesth Analg. 2006;103:390–5
18. Sandberg WS, Sandberg EH, Seim AR, Anupama S, Ehrenfeld JM, Spring SF, Walsh JL. Real-time checking of electronic anesthesia records for documentation errors and automatically text messaging clinicians improves quality of documentation. Anesth Analg. 2008;106:192–201
19. Peterfreund RA, Driscoll WD, Walsh JL, Subramanian A, Anupama S, Weaver M, Morris T, Arnholz S, Zheng H, Pierce ET, Spring SF. Evaluation of a mandatory quality assurance data capture in anesthesia: a secure electronic system to capture quality assurance information linked to an automated anesthesia record. Anesth Analg. 2011;112:1218–25
20. Ehrenfeld JM, Epstein RH, Bader S, Kheterpal S, Sandberg WS. Automatic notifications mediated by anesthesia information management systems reduce the frequency of prolonged gaps in blood pressure documentation. Anesth Analg. 2011;113:356–63
21. Blum JM, Stentz MJ, Maile MD, Jewell E, Raghavendran K, Engoren M, Ehrenfeld JM. Automated alerting and recommendations for the management of patients with preexisting hypoxia and potential acute lung injury: a pilot study. Anesthesiology. 2013;119:295–302
22. Nair BG, Newman SF, Peterson GN, Schwid HA. Smart Anesthesia Manager™ (SAM)—a real-time decision support system for anesthesia care during surgery. IEEE Trans Biomed Eng. 2013;60:207–10
23. Eden A, Pizov R, Toderis L, Kantor G, Perel A. The impact of an electronic reminder on the use of alarms after separation from cardiopulmonary bypass. Anesth Analg. 2009;108:1203–8
24. Nair BG, Horibe M, Newman SF, Wu WY, Schwid HA. Near real-time notification of gaps in cuff blood pressure recordings for improved patient monitoring. J Clin Monit Comput. 2013;27:265–71
25. Nair BG, Horibe M, Newman SF, Wu WY, Peterson GN, Schwid HA. Anesthesia information management system-based near real-time decision support to manage intraoperative hypotension and hypertension. Anesth Analg. 2014;118:206–14
26. Greenes RA Clinical Decision Support: The Road to Broad Adoption. 20142nd ed. Amsterdam, The Netherlands Elsevier
27. Greenes RA Clinical Decision Support: The Road Ahead. 20071st ed. Amsterdam, The Netherlands Elsevier
28. Kain ZN, Vakharia S, Garson L, Engwall S, Schwarzkopf R, Gupta R, Cannesson M. The perioperative surgical home as a future perioperative practice model. Anesth Analg. 2014;118:1126–30
29. Dexter F, Wachtel RE. Strategies for net cost reductions with the expanded role and expertise of anesthesiologists in the perioperative surgical home. Anesth Analg. 2014;118:1062–71
30. Zanetti G, Flanagan HL Jr, Cohn LH, Giardina R, Platt R. Improvement of intraoperative antibiotic prophylaxis in prolonged cardiac surgery by automated alerts in the operating room. Infect Control Hosp Epidemiol. 2003;24:13–6
31. St Jacques P, Sanders N, Patel N, Talbot TR, Deshpande JK, Higgins M. Improving timely surgical antibiotic prophylaxis redosing administration using computerized record prompts. Surg Infect (Larchmt). 2005;6:215–21
32. Wax DB, Beilin Y, Levin M, Chadha N, Krol M, Reich DL. The effect of an interactive visual reminder in an anesthesia information management system on timeliness of prophylactic antibiotic administration. Anesth Analg. 2007;104:1462–6
33. Wanderer JP, Charnin J, Driscoll WD, Bailin MT, Baker K. Decision support using anesthesia information management system records and accreditation council for graduate medical education case logs for resident operating room assignments. Anesth Analg. 2013;117:494–9
34. Wright A, Bates DW, Middleton B, Hongsermeier T, Kashyap V, Thomas SM, Sittig DF. Creating and sharing clinical decision support content with Web 2.0: issues and examples. J Biomed Inform. 2009;42:334–46
35. Wagner MM, Eisenstadt SA, Hogan WR, Pankaskie MC. Preferences of interns and residents for e-mail, paging, or traditional methods for the delivery of different types of clinical information. Proc AMIA Symp. 1998:140–4
36. Wu RC, Lo V, Morra D, Wong BM, Sargeant R, Locke K, Cavalcanti R, Quan SD, Rossos P, Tran K, Cheung M. The intended and unintended consequences of communication systems on general internal medicine inpatient care delivery: a prospective observational case study of five teaching hospitals. J Am Med Inform Assoc. 2013;20:766–77
37. Ledolter J, Dexter F, Epstein RH. Analysis of variance of communication latencies in anesthesia: comparing means of multiple log-normal distributions. Anesth Analg. 2011;113:888–96
38. Rothman BS, Dexter F, Epstein RH. Communication latencies of Apple push notification messages relevant for delivery of time-critical information to anesthesia providers. Anesth Analg. 2013;117:398–404
39. Epstein RH, Dexter F, Rothman B. Communication latencies of wireless devices suitable for time-critical messaging to anesthesia providers. Anesth Analg. 2013;116:911–8
40. Ledolter J, Dexter F, Epstein RH. Analysis of variance of communication latencies in anesthesia: comparing means of multiple log-normal distributions. Anesth Analg. 2011;113:888–96
41. Epstein RH, Dexter F, Ehrenfeld JM, Sandberg WS. Implications of event entry latency on anesthesia information management decision support systems. Anesth Analg. 2009;108:941–7
42. Dexter F, Epstein RH, Lee JD, Ledolter J. Automatic updating of times remaining in surgical cases using Bayesian analysis of historical case duration data and “instant messaging” updates from anesthesia providers. Anesth Analg. 2009;108:929–40
43. Epstein RH, Dexter F. Mediated interruptions of anaesthesia providers using predictions of workload from anaesthesia information management system data. Anaesth Intensive Care. 2012;40:803–12
44. de Gil PR, Chen Y, Kim ES, Nguyen D, Kellermann A, Bellara A, Kromrey J. Parametric test for two population means: an empirical comparison of type I error control and statistical power. In: JSM Proceedings, Social Statistics Section. 2013 Alexandria, Egypt American Statistical Association:2388–402
45. Sherman J, Le C, Lamers V, Eckelman M. Life cycle greenhouse gas emissions of anesthetic drugs. Anesth Analg. 2012;114:1086–90
46. McGain F, Story D, Kayak E, Kashima Y, McAlister S. Workplace sustainability: the “cradle to grave” view of what we do. Anesth Analg. 2012;114:1134–9
47. Dexter F, Maguire D, Epstein RH. Observational study of anaesthetists’ fresh gas flow rates during anaesthesia with desflurane, isoflurane and sevoflurane. Anaesth Intensive Care. 2011;39:460–4
48. Rasmussen J. Skills, rules, and knowledge: signals, signs, and symbols, and other distinctions in human performance models. IEEE Trans Syst Man Cybern. 1983;13:257–66
49. Epstein RH, Dexter F, Piotrowski E. Automated correction of room location errors in anesthesia information management systems. Anesth Analg. 2008;107:965–71
50. Xiao Y, Hu P, Hu H, Ho D, Dexter F, Mackenzie CF, Seagull FJ, Dutton RP. An algorithm for processing vital sign monitoring data to remotely identify operating room occupancy in real-time. Anesth Analg. 2005;101:823–9
51. Shabot MM, LoBue M. Real-time wireless decision support alerts on a Palmtop PDA. Proc SCAMC (JAMIA Suppl). 1995;13:54–7
52. Shabot MM, LoBue M, Chen J. Wireless clinical alerts for physiologic, laboratory and medication data. Proc AMIA Symp. 2000:789–93
53. Eisenstadt SA, Wagner MM, Hogan WR, Pankaskie MC, Tsui F-C, Wilgright W. Mobile workers in healthcare and their information needs: are 2-way pagers the answer? Proc AMIA Symp. 1998:135–9
54. Lubarsky DA, Glass PS, Ginsberg B, Dear GL, Dentz ME, Gan TJ, Sanderson IC, Mythen MG, Dufore S, Pressley CC, Gilbert WC, White WD, Alexander ML, Coleman RL, Rogers M, Reves JG. The successful implementation of pharmaceutical practice guidelines. Analysis of associated outcomes and cost savings. Anesthesiology. 1997;86:1145–60
55. Body SC, Fanikos J, DePeiro D, Philip JH, Segal BS. Individualized feedback of volatile agent use reduces fresh gas flow rate, but fails to favorably affect agent choice. Anesthesiology. 1999;90:1171–5
56. Epstein RH, Dexter F. Implications of resolved hypoxemia on the utility of desaturation alerts sent from an anesthesia decision support system to supervising anesthesiologists. Anesth Analg. 2012;115:929–33
57. Wickens WD, Dixon S. The benefits of imperfect diagnostic automation: a synthesis of the literature. Theor Issues Ergon Sci. 2007;8:201–12
58. Ehrenfeld JM, Dexter F, Rothman BS, Minton BS, Johnson D, Sandberg WS, Epstein RH. Lack of utility of a decision support system to mitigate delays in admission from the operating room to the postanesthesia care unit. Anesth Analg. 2013;117:1444–52
59. Healey AN, Sevdalis N, Vincent CA. Measuring intra-operative interference from distraction and interruption observed in the operating theatre. Ergonomics. 2006;49:589–604
60. Healey AN, Primus CP, Koutantji M. Quantifying distraction and interruption in urological surgery. Qual Saf Health Care. 2007;16:135–9
61. Fargen KM, O’Connor T, Raymond S, Sporrer JM, Friedman WA. An observational study of hospital paging practices and workflow interruption among on-call junior neurological surgery residents. J Grad Med Educ. 2012;4:467–71
62. Bailey BP, Konstan JA, Carlis JV. Measuring the effects of interruptions on task performance in the user interface. In: Proceedings of 2000 IEEE International Conference on Systems, Man, and Cybernetics. 2000;2 Nashville, Tennessee Institute of Electrical and Electronics Engineers:757–62
63. Cutrell EB, Czerwinski M, Horvitz E. Notification, disruption, and memory: effects of messaging interruptions on memory and performance. In: Proceedings of INTERACT 2001. 2001 Tokyo, Japan IOS Press:263–9
64. Pluyter JR, Buzink SN, Rutkowski AF, Jakimowicz JJ. Do absorption and realistic distraction influence performance of component task surgical procedure? Surg Endosc. 2010;24:902–7
65. Persoon MC, van Putten K, Muijtjens AM, Witjes JA, Hendrikx AJ, Scherpbier AJ. Effect of distraction on the performance of endourological tasks: a randomized controlled trial. BJU Int. 2011;107:1653–7
66. Sevdalis N, Undre S, McDermott J, Giddie J, Diner L, Smith G. Impact of intraoperative distractions on patient safety: a prospective descriptive study using validated instruments. World J Surg. 2014;38:751–8
67. Gillespie BM, Chaboyer W, Fairweather N. Interruptions and miscommunications in surgery: an observational study. AORN J. 2012;95:576–90
68. Espino S, Cox D, Kaplan B. Alphanumeric paging: a potential source of problems in patient care and communication. J Surg Educ. 2011;68:447–51
69. Blum NJ, Lieu TA. Interrupted care. The effects of paging on pediatric resident activities. Am J Dis Child. 1992;146:806–8
70. Weinger MB, Herndon OW, Zornow MH, Paulus MP, Gaba DM, Dallen LT. An objective methodology for task analysis and workload assessment in anesthesia providers. Anesthesiology. 1994;80:77–92
71. Manser T, Wehner T. Analysing action sequences: variations in action density in the administration of anaesthesia. Cogn Technol Work. 2002;4:71–81
72. Phipps D, Meakin GH, Beatty PC, Nsoedo C, Parker D. Human factors in anaesthetic practice: insights from a task analysis. Br J Anaesth. 2008;100:333–43
73. Leedal JM, Smith AF. Methodological approaches to anaesthetists’ workload in the operating theatre. Br J Anaesth. 2005;94:702–9
74. Aronsky D, Chan KJ, Haug PJ. Evaluation of a computerized diagnostic decision support system for patients with pneumonia: study design considerations. J Am Med Inform Assoc. 2001;8:473–85
75. Dexter F, Epstein RH. Holiday and weekend operating room on-call staffing requirements. Anesth Analg. 2006;103:1494–8
76. Moore IC, Strum DP, Vargas LG, Thomson DJ. Observations on surgical demand time series: detection and resolution of holiday variance. Anesthesiology. 2008;109:408–16
77. Dexter F, Wachtel RE, Epstein RH, Ledolter J, Todd MM. Analysis of operating room allocations to optimize scheduling of specialty rotations for anesthesia trainees. Anesth Analg. 2010;111:520–4
78. Wachtel RE, Dexter F. Influence of the operating room schedule on tardiness from scheduled start times. Anesth Analg. 2009;108:1889–901
79. Dexter F, Marcon E, Aker J, Epstein RH. Numbers of simultaneous turnovers calculated from anesthesia or operating room information management system data. Anesth Analg. 2009;109:900–5
80. Wang J, Dexter F, Yang K. A behavioral study of daily mean turnover times and first case of the day start tardiness. Anesth Analg. 2013;116:1333–41
81. Dexter F, Epstein RH, Marcon E, Ledolter J. Estimating the incidence of prolonged turnover times and delays by time of day. Anesthesiology. 2005;102:1242–8
82. Dexter F, Epstein RH. Increased mean time from end of surgery to operating room exit in a historical cohort of cases with prolonged time to extubation. Anesth Analg. 2013;117:1453–9
83. Dexter F, Marcon E, Epstein RH, Ledolter J. Validation of statistical methods to compare cancellation rates on the day of surgery. Anesth Analg. 2005;101:465–73
84. Dexter F. High-quality operating room management research. J Clin Anesth. 2014;26:341–2
85. Austin TM, Lam HV, Shin NS, Daily BJ, Dunn PF, Sandberg WS. Elective change of surgeon during the OR day has an operationally negligible impact on turnover time. J Clin Anesth. 2014;26:343–9
86. Ryan SE, Porth LS. A tutorial on the piecewise regression approach applied to bedload transport data. US Department of Agriculture, Forest Service, Rocky Mountain Research Station. 2007 RMRS-GTR-189
87. Wagner AK, Soumerai SB, Zhang F, Ross-Degnan D. Segmented regression analysis of interrupted time series studies in medication use research. J Clin Pharm Ther. 2002;27:299–309
88. Epstein RH. Postoperative nausea and vomiting, decision support, and regulatory oversight. Anesth Analg. 2010;111:270–1
89. Lane JS, Sandberg WS, Rothman B. Development and implementation of an integrated mobile situational awareness iPhone application VigiVU™ at an academic medical center. Int J Comput Assist Radiol Surg. 2012;7:721–35
© 2015 International Anesthesia Research Society