This article is accompanied by the following Invited Commentary:
Schneider G. Bispectral index aware or minimum alveolar concentration aware? Alerting thresholds for prevention of awareness. Eur J Anaesthesiol 2015; 32:301–302.
Intraoperative awareness with explicit recall (AWR) of surgical events can be a devastating complication for patients, with significant psychological sequelae.1–6 The incidence of definite AWR in patients undergoing general anaesthesia is reported to be between 1 and 2 out of 1000 cases and as high as 3 to 4 out of 1000 cases for both possible and definite AWR events;7–9 in patients at a high risk for AWR, the incidence approaches 1%.10 It has been posited that the primary cause of AWR is insufficient anaesthesia (rather than, for example, genetic factors),5,11 suggesting that alerting protocols could prevent AWR if a specific threshold was identified.
Two common surrogates for anaesthetic depth are minimum alveolar concentration (MAC) measured by end-tidal anaesthetic concentration (ETAC), and the bispectral index (BIS). Alerting algorithms based on either MAC or BIS values can be implemented easily to notify the provider of potentially insufficient anaesthesia. The rapid expansion of electronic Anaesthesia Information Management Systems (AIMS) allows for enhanced use of alerting algorithms with the potential to combine demographic, comorbidity, physiological and anaesthetic concentration variables. In addition, the AIMS allow the provider to be notified via pager of potentially insufficient anaesthesia even when the alarms on the primary monitoring system have been silenced.
Clinical trials investigating the prevention of AWR9,10,12–14 used specific thresholds for potentially insufficient anaesthesia, with the provider being instructed to keep the BIS value between 40 and 60 or with audible alarms if the BIS or MAC values fell outside defined ranges. The MAC and BIS values chosen were based on previously published work, but to date, there has been no systematic study of the appropriate threshold for MAC or BIS alarms for the prevention of AWR based on prospectively collected data.
The parent trial for this study9 investigated whether the use of alerting algorithms in cases randomised to either anaesthetic concentration or BIS values decreased the incidence of AWR. It did not investigate discrete MAC or BIS data elements to determine whether there is a specific value that would maximise sensitivity and specificity in the prevention of AWR or explore any changes in provider behaviour when alerts are generated. Therefore, the objective of this study was to test the hypothesis that there is an evidence-based alerting threshold for MAC or BIS values that would maximise the sensitivity and specificity of alarms aimed at preventing AWR. In addition, we sought to determine whether alerting the provider changes behaviour with respect to anaesthetic management in the prevention of AWR.
Materials and methods
This study is a prespecified secondary analysis of the Michigan Awareness Control Study (MACS) (ClinicalTrials.gov No. NCT00689091).9 The parent trial and this secondary analysis were approved by the Institutional Review Board (IRB HUM 13626, initial study approval 14 August 2007) of the University of Michigan (2800 Plymouth Road, Building 520 Room 3214, Ann Arbor, Michigan, USA; Chairs, Drs Michael Geisser and John Weg). In brief, we screened all adult patients between May 2008 and May 2010 presenting to a multihospital healthcare system for surgery with general anaesthesia using an inhaled or intravenous anaesthetic. A detailed discussion with each patient took place and verbal informed consent was obtained and documented in our AIMS. Patients were excluded if the use of a BIS monitor was impractical (e.g. intracranial procedures, adhesive allergy, surgery involving the forehead) or an underlying brain disorder rendered the BIS a questionable measure of consciousness (e.g. history of traumatic brain injury). The BIS Quatro sensor (Covidien, Boulder, Colorado, USA) was attached preoperatively in all patients by a member of the research staff. Alerts to notify the provider of potentially insufficient anaesthetic dosing were based on either the age-adjusted MAC (aaMAC)15 or BIS values. For the parent trial, aaMAC was calculated on the basis of prespecified age groups.16
A detailed description of the randomisation and blinding is explained elsewhere9,17 and is briefly summarised here. The study was divided into year quarters (eight 3-month periods over 2 years), with MAC and BIS alerting algorithms assigned randomly for each quarter. For the MAC alerting rooms, the real-time BIS values were hidden from the provider's view. In addition, if the median ETAC for a 5-min epoch was less than 0.5 aaMAC, an alphanumeric paging alert was sent to the provider in the room. For the BIS alerting group, the BIS values were visible to the provider. In addition, if the median BIS value for a 5-min epoch was greater than 60, an alphanumeric paging alert was generated. A study team member contacted each patient and administered the modified Brice interview 28 to 30 days after surgery.18,19 As described previously,9 after the modified Brice interview and additional interviews of potential AWR patients were performed, each event was categorised as no AWR, possible AWR or definite AWR. Data from other trials in which the authors were involved12,13 could not be included due to differences in data acquisition systems and incomplete records of alarm delivery.
For the MACS trial, Centricity (GE Healthcare, Madison, Wisconsin, USA) was the AIMS used for programming alerts and notifying providers via alphanumeric text paging. Centricity interfaces with the haemodynamic monitors (GE Marquette Solar 9500, Milwaukee, Wisconsin, USA) and also with the anaesthesia machine (AISYS Anaesthesia Machine; GE Healthcare). ETAC values were automatically calculated in real time from the expired volatile anaesthetic concentrations that were collected by the AISYS anaesthesia machine and transmitted to Centricity. BIS and ETAC data elements were electronically captured for every patient by the AIMS every 60 s and were available for later study extraction and analysis.
For this secondary analysis, we included cases in which inhaled volatile agents were used as the primary anaesthetic. We excluded total intravenous anaesthetic (TIVA) cases, any case for which a propofol infusion was used in conjunction with a volatile anaesthetic and any case with missing volatile anaesthetic data due to infrequent AIMS data interface issues.
Secondary analysis methodology
All cases were reviewed to ensure complete data for ETAC and BIS values from an electronically documented time of ‘anaesthesia induction end’ to the time of ‘request for postanaesthesia care unit (PACU) bed’ or ‘transport to the ICU’. For this secondary analysis, aaMAC was calculated on the basis of the age documented in the AIMS at the time of operation and not on the prespecified age groups as in the parent trial.15 The surgery was divided into 5-min epochs (during the AIMS timestamps listed previously) and the median ETAC was calculated for each of those 5-min epochs. The overall median ETAC was also calculated for each case. The same technique was used for patients with valid BIS monitoring data.
Data were analysed using two different methods to determine a single threshold for the prevention of AWR for ETAC and BIS values. First, the data were dichotomised by whether any 5-min epoch was below (aaMAC) or above (BIS) a set value throughout the case. The maximum sensitivity, specificity and Youden's index were calculated. Second, the median ETAC and BIS values for the case were analysed using a receiver operating characteristic curve c-statistic. If the c-statistic demonstrated adequate discriminating capacity (>0.70), then the value with the maximum sensitivity and specificity would be computed and retrospectively applied to the database to determine a single threshold for the prevention of AWR.
To investigate whether the alerting algorithm changed provider behaviour, we first calculated the percentage of the case during which the anaesthetic concentrations triggered the alarm. This percentage was calculated by dividing the number of 5-min epochs that met the ETAC alerting threshold by the number of 5-min epochs overall. We chose to present the data as the percentage of the case duration instead of total minutes of the case to account for the variance in length of procedures. A control group that had no real-time alerting interventions, distinct from the ETAC and BIS groups, was used to explore whether there is a behavioural effect by retrospectively applying the ETAC alerting algorithm. This group received neither BIS nor MAC alerts yet was still assessed for AWR; the anaesthetic was delivered on the basis of routine clinical and haemodynamic variables. The control group resulted from technical interface issues from the parent trial and was not prespecified.9 However, as there was a ‘no intervention’ group that resulted from MACS, we could assess whether there was a behavioural effect attributable to having alerts generated throughout the case. This was accomplished by retrospectively applying the ETAC alerting algorithm to the ‘no intervention’ group and calculating what percentage of time the case would have triggered an alarm if an algorithm had actually been active in real time. The percentage of the case during which the anaesthetic triggered the ETAC alerting algorithm was then compared between the original ETAC intervention arm and the post hoc control group. The mean percentage of the case that triggered an alarm was also examined for a change in behaviour across the study period for both the ETAC and BIS arms. If the mean percentage changed across the quarters, this would indicate that anaesthetic delivery behaviour had changed.
Due to the low incidence of definite AWR in the parent trial and the potential psychological impact of possible AWR, we combined definite and possible AWR events into one category for this secondary analysis. To determine whether there was a single threshold that maximises sensitivity and specificity for the prevention of AWR, aaMAC for all cases was dichotomised by whether the case did or did not have any 5-min median epochs in which the aaMAC was less than 0.4, less than 0.5, less than 0.6, less than 0.7, less than 0.8 or less than 0.9. The same dichotomising technique was used for cases with valid BIS data that had at least one 5-min median epoch with BIS more than 60, more than 70, more than 80 or more than 90. The baseline BIS threshold was set at 60 because this generally represents the threshold between general anaesthesia (<60) and sedation or wakefulness (>60).20 Sensitivity, specificity and Youden's index were then calculated to determine whether there was an optimal threshold for the prevention of AWR for either aaMAC or BIS. The Youden's index was calculated as (sensitivity + specificity – 1).21 A Youden's index of 1 would indicate that the threshold is perfect and a Youden's index of 0 would indicate that the threshold has no diagnostic value in the prevention of AWR.21 Next, a c-statistic was calculated from a receiver operating characteristic curve to determine whether there was a single diagnostic threshold for either ETAC or BIS that can be quantified for prevention of AWR. If the c-statistic was deemed adequate (>0.70), then the continuous data for both aaMAC and BIS would be analysed to determine the specific threshold in the prevention of AWR.
A Mann–Whitney U test was performed to assess whether providers receiving the original MAC alerts differed statistically when compared with the ‘no intervention’ (i.e. no alert) group for the entire time period and by quarter of the study. To determine whether there was a behavioural change, as documented by a significant difference in the percentage of the case that triggered an alert for potentially insufficient anaesthesia, a Kruskal–Wallis test was performed for both MAC and BIS arms across the study period. Data are displayed as the mean percentage of the case to trigger an alert ± 2 x SEM.
SPSS version 20 (IBM Corp, Armonk, New York, USA) was used for all analyses. Data extraction from AIMS was completed using structured query language. A P value of less than 0.05 was considered statistically significant throughout.
The parent trial had a total of 18 836 patients with complete information on the AWR outcome. We excluded 231 cases because of the use of TIVA, 297 for agent analyser device technical issues and 2240 for use of an adjunct intraoperative propofol infusion. This resulted in a dataset of 16 068 patients, with a total of 32 definite or possible AWR events (Fig. 1).
Youden's index did not demonstrate a single threshold for aaMAC or BIS values in the prevention of AWR (Table 1). The c-statistic for median aaMAC was 0.431 ± 0.046 and 0.491 ± 0.056 for BIS, indicating that there is not a specific threshold that can be calculated for the prevention of AWR when using either aaMAC or BIS values. There were 10 patients who experienced an AWR event with BIS values <60 (median for 5-min epoch) for the entire case.
When applying the ETAC alerting algorithm retrospectively to the ‘no intervention’ (i.e. no alert) cases, we determined that cases randomised to MAC alerting had a statistically shorter mean percentage of the case that generated an alert for potentially insufficient anaesthesia than the ‘no intervention’ cases (2.4% ± 7.5 vs. 3.1% ± 8.5%, P = 0.009). Four of the eight year quarters demonstrated these findings, although the remaining quarters did not reach statistical significance (Fig. 2). In the trend analysis by study period, the mean percentage of the case that triggered a potentially insufficient anaesthetic alert in the ETAC arm increased significantly (P < 0.001) (Fig. 3). However, the mean percentage of the case that generated a BIS alert via the alerting algorithm did not change across the study period (P = 0.38) (Fig. 4).
Population-based alerting in the prevention of AWR is important to consider because retrospective evidence shows that approximately 87% of all AWR cases are attributable to insufficient anaesthesia.5 An editorial by Nickalls and Mahajan11 presented a parsimonious approach by stating that all cases of AWR are attributable to insufficient anaesthesia unless there is compelling evidence to the contrary. They suggested that a population-based adequate dose for anaesthetics should be identified and implemented for the prevention of AWR.11 In the present study, we analysed discrete surrogate metrics of anaesthetic depth (ETAC and BIS) in order to identify a single diagnostic threshold for the systematic prevention of AWR. We have demonstrated that, in patients undergoing general anaesthesia in which only volatile anaesthetics were used, there is no population-based threshold that could be used as an alert in the prevention of definite or possible AWR using either ETAC or BIS. The population data from Youden's index suggest that the thresholds studied would not result in the eradication of AWR.
The thresholds that we selected in the current study were representative of what would be considered standard of care aaMAC for patients under general anaesthesia. Concentrations higher than 1.0 aaMAC were excluded from this analysis because of very high false positive alarms at thresholds of at least 0.7 MAC.16 It is important to note that Youden's index incorporates data on both sensitivity and specificity. As Table 1 indicates, lower alerting thresholds (e.g. higher anaesthetic concentrations) increase sensitivity at the cost of decreasing specificity. Thus, including thresholds of AWR alerting beyond 1 MAC would further increase sensitivity, but would probably not increase Youden's index due to the concomitant decrease in specificity. In addition, the c-statistics indicated that, for both aaMAC and BIS, there was approximately a 50% chance (essentially random) of determining the correct threshold in the prevention of AWR using a population-based approach.21
To date, multiple randomised controlled trials have been conducted using a population-based alerting (or monitoring) strategy in which a specific alarm or range for ETAC or BIS was established for the prevention of AWR.9,10,12,13,22 Myles et al.10 found that BIS monitoring (maintained between 40 and 60) was associated with a relative risk reduction of 82% in surgical patients at a high risk for AWR when compared with routine monitoring. The next two trials used audible alerts to notify the provider of potentially insufficient anaesthesia in patients at a high risk of AWR. Avidan et al.12,13 found that both BIS and ETAC alarms reduced observed awareness events compared with the expected incidence. Mashour et al.9 included unselected surgical patients requiring general anaesthesia and demonstrated that BIS alerting protocols reduced the incidence of definite or possible AWR compared with no intervention. The persistence of AWR in previous trials suggests that population-based alerting approaches for insufficient anaesthesia will not eradicate AWR. The current study adds to the literature by suggesting that it was not simply the empirical or arbitrary determination of alerting thresholds in past trials that failed to eradicate AWR because our systematic analysis and comparison suggest that no one threshold exists. Rather than arguing that BIS or ETAC alerts are not useful, we would instead encourage clinicians to choose some threshold to prevent egregious causes of AWR (such as an empty vaporiser). Past randomised controlled trials demonstrate that alerts are, in fact, associated with reduced AWR.
At an individual level, we know that there is a specific threshold at which a given patient will be likely to experience AWR. The identification of such a threshold will, in the future, probably be guided by risk factors for AWR coupled with more sophisticated monitoring. Previous work has reported specific patient-based risk factors for AWR.5 Further identification of risk factors, in conjunction with developments in monitoring the neural substrates of consciousness, must ultimately be incorporated into prevention strategies for AWR at an individualised level.
Fundamentally, individualised alerting strategies would not be beneficial without establishing evidence that real-time alarms are capable of changing behaviour. We have demonstrated that real-time alerting alters the administration of anaesthetics. When retrospectively applying the MAC alerting algorithm to the ‘no intervention’ cases in the parent trial, the providers who were not alerted to potentially insufficient anaesthesia had lower MAC values throughout the case than those who were alerted in real time. This indicates that the providers receiving alerts were statistically more likely to keep the anaesthetic concentration within the stated range for the trial and, we infer, changed their behaviour to do so. These results confirm other previously published studies that providing clinicians with alerts can drive a change in clinical care.23–30
When alerting is used to notify providers to change behaviour, there is the potential for alert fatigue, which is defined as the provider becoming less responsive due to an alarm becoming bothersome or ineffective. Alert fatigue is an increasingly prevalent phenomenon in the medical community, especially with the rapid advancement of electronic medical records.31–33 Block et al.34 surveyed anaesthesiologists practising in the USA and found that 70% of the times that alarms were silenced were due to the perception of a false alarm. Only 16% of providers stated that they never turn off alarms.34 Therefore, it is imperative to minimise false alarms – and, consequently, alert fatigue – in developing an alerting strategy for AWR, a difficult task given the rarity of the outcome. This is one reason why the current study focused on both sensitivity and specificity. We found evidence of alert fatigue as well as possible desensitisation to ETAC alerts across the study period. The mean percentage of the case that generated an ETAC alert changed significantly during the last quarter of the trial, with alerts having increased approximately 1.2% from the previous quarter and 1.5% from the beginning of the study. These data could indicate that the providers thought the alerts were false alarms and therefore were becoming desensitised and fatigued as the study continued (delayed alert fatigue). However, these findings were only found in the last quarter of the study and could also be an outlier. The BIS alerting rate was consistent throughout the study and showed no evidence of generating desensitisation or alert fatigue.
The main limitation of this study was the small number of AWR outcomes in our population (n = 32). To move towards an individualised alerting approach, a multinomial logistic regression model must be developed that incorporates patient-specific risk factors (such as history of AWR)35 along with anaesthetic concentrations and, if possible, neurophysiological indices. Furthermore, we used the median from a 5-min period and must acknowledge that the individual BIS or MAC values could have nonetheless fluctuated in a way that might not be detected with our methodology. Finally, the current study only investigated a threshold using general anaesthesia with inhaled anaesthetics and is therefore not generalisable to cases performed using TIVA.
In conclusion, we could not identify a single practical threshold of ETAC or BIS values that can be chosen for the eradication of definite and possible AWR in a broad surgical population. Although alerts have been demonstrated to prevent AWR, future work must move towards an individualised patient-based approach incorporating specific risk factors as well as monitoring the neural substrates of consciousness. Indeed, recent studies have demonstrated promise in identifying correlates of anaesthetic-induced unconsciousness based on the neurobiology of consciousness.36,37 Finally, we were able to demonstrate that providing alerts via an AIMS can influence intraoperative care, but ETAC alerting has the potential for desensitisation and alert fatigue.
Acknowledgements relating to this article
Assistance with the study: none.
Financial support: the parent trial, entitled ‘Prevention of intraoperative awareness in an unselected surgical population: a randomized compared effectiveness trial’, was funded by the Cerebral Function Monitoring grant (to GAM) from the Foundation for Anesthesia Education and Research (Rochester, Minnesota, USA), American Society of Anesthesiologists (Park Ridge, Illinois, USA), the National Institutes of Health (Bethesda, Maryland, USA) (KL2 RR024987-01) (to GAM); Department of Anesthesiology, University of Michigan Medical School. This secondary analysis was funded by the Department of Anesthesiology, University of Michigan Medical School.
Conflicts of interest: GAM has a patent pending (through the University of Michigan) on the use of connectivity measures to assess levels of consciousness.
Presentation: some results of this study were presented at the annual meeting of the International Anesthesia Research Society in San Diego, California, USA, 2013.
1. Moerman N, Bonke B, Oosting J. Awareness and recall during general anesthesia. Facts and feelings. Anesthesiology
2. Schwender D, Kunze-Kronawitter H, Dietrich P, et al. Conscious awareness during general anaesthesia: patients’ perceptions, emotions, cognition and reactions. Br J Anaesth
3. Domino KB, Posner KL, Caplan RA, Cheney FW. Awareness during anesthesia: a closed claims analysis. Anesthesiology
4. Osterman JE, Hopper J, Heran WJ, et al. Awareness under anesthesia and the development of posttraumatic stress disorder. Gen Hosp Psychiatry
5. Ghoneim MM, Block RI, Haffarnan M, Mathews MJ. Awareness during anesthesia: risk factors, causes and sequelae: a review of reported cases in the literature. Anesth Analg
6. Leslie K, Chan MT, Myles PS, et al. Posttraumatic stress disorder in aware patients from the B-aware trial. Anesth Analg
7. Sandin RH, Enlund G, Samuelsson P, Lennmarken C. Awareness during anaesthesia: a prospective case study. Lancet
8. Sebel PS, Bowdle TA, Ghoneim MM, et al. The incidence of awareness during anesthesia: a multicenter United States study. Anesth Analg
9. Mashour GA, Shanks A, Tremper KK, et al. Prevention of intraoperative awareness with explicit recall in an unselected surgical population: a randomized comparative effectiveness trial. Anesthesiology
10. Myles PS, Leslie K, McNeil J, et al. Bispectral index monitoring to prevent awareness during anaesthesia: the B-Aware randomised controlled trial. Lancet
11. Nickalls RW, Mahajan RP. Awareness and anaesthesia: think dose, think data. Br J Anaesth
12. Avidan MS, Zhang L, Burnside BA, et al. Anesthesia awareness and the bispectral index. N Engl J Med
13. Avidan MS, Jacobsohn E, Glick D, et al. Prevention of intraoperative awareness in a high-risk surgical population. N Engl J Med
14. Zhang C, Xu L, Ma YQ, et al. Bispectral index monitoring prevent awareness during total intravenous anesthesia: a prospective, randomized, double-blinded, multicenter controlled trial. Chin Med J (Engl)
15. Nickalls RW, Mapleson WW. Age-related iso-MAC charts for isoflurane, sevoflurane and desflurane in man. Br J Anaesth
16. Mashour GA, Esaki RK, Vandervest JC, et al. A novel electronic algorithm for detecting potentially insufficient anesthesia: implications for the prevention of intraoperative awareness. J Clin Monit Comput
17. Mashour GA, Tremper KK, Avidan MS. Protocol for the ‘Michigan Awareness Control Study’: a prospective, randomized, controlled trial comparing electronic alerts based on bispectral index monitoring or minimum alveolar concentration for the prevention of intraoperative awareness. BMC Anesthesiol
18. Brice DD, Hetherington RR, Utting JE. A simple study of awareness and dreaming during anaesthesia. Br J Anaesth
19. Abouleish E, Taylor FH. Effect of morphine-diazepam on signs of anesthesia, awareness, and dreams of patients under N2
O for cesarean section. Anesth Analg
20. Glass PS, Bloom M, Kearse L, et al. Bispectral analysis measures sedation and memory effects of propofol, midazolam, isoflurane, and alfentanil in healthy volunteers. Anesthesiology
21. Bewick V, Cheek L, Ball J. Statistics review 13: receiver operating characteristic curves. Crit Care
22. Avidan MS, Mashour GA. Prevention of intraoperative awareness with explicit recall: making sense of the evidence. Anesthesiology
23. Bates DW, Cohen M, Leape LL, et al. Reducing the frequency of errors in medicine using information technology. J Am Med Inform Assoc
24. St Jacques P, Sanders N, Patel N, et al. Improving timely surgical antibiotic prophylaxis redosing administration using computerized record prompts. Surg Infect (Larchmt)
25. Kucher N, Koo S, Quiroz R, et al. Electronic alerts to prevent venous thromboembolism among hospitalized patients. N Engl J Med
26. O’Reilly M, Talsma A, VanRiper S, et al. An anesthesia information system designed to provide physician-specific feedback improves timely administration of prophylactic antibiotics. Anesth Analg
27. Kheterpal S, Gupta R, Blum JM, et al. Electronic reminders improve procedure documentation compliance and professional fee reimbursement. Anesth Analg
28. Wax DB, Beilin Y, Levin M, et al. The effect of an interactive visual reminder in an anesthesia information management system on timeliness of prophylactic antibiotic administration. Anesth Analg
29. Eden A, Pizov R, Toderis L, et al. The impact of an electronic reminder on the use of alarms after separation from cardiopulmonary bypass. Anesth Analg
30. Kooij FO, Klok T, Hollmann MW, Kal JE. Decision support increases guideline adherence for prescribing postoperative nausea and vomiting prophylaxis. Anesth Analg
31. Baker DE. Medication alert fatigue: the potential for compromised patient safety. Hosp Pharm
32. Lee EK, Mejia AF, Senior T, Jose J. Improving patient safety through medical alert management: an automated decision tool to reduce alert fatigue. AMIA Annu Symp Proc
33. Kesselheim AS, Cresswell K, Phansalkar S, et al. Clinical decision support systems could be modified to reduce ’alert fatigue’ while still minimizing the risk of litigation. Health Aff (Millwood)
34. Block FE Jr, Nuutinen L, Ballast B. Optimization of alarms: a study on alarm limits, alarm sounds, and false alarms, intended to reduce annoyance. J Clin Monit Comput
35. Aranake A, Gradwohl S, Ben-Abdallah A, et al. Increased risk of intraoperative awareness in patients with a history of awareness. Anesthesiology
36. Lee U, Ku S, Noh G, et al. Disruption of frontal-parietal communication by ketamine, propofol, and sevoflurane. Anesthesiology
37. Casali AG, Gosseries O, Rosanova M, et al. A theoretically based index of consciousness independent of sensory processing and behavior. Sci Transl Med