Anesthesia & Analgesia:
Patient Safety: Research Reports
A Novel Classification Instrument for Intraoperative Awareness Events
Mashour, George A. MD, PhD*†; Esaki, Roy K. MS, MD*; Tremper, Kevin K. PhD, MD*; Glick, David B. MD‡; O'Connor, Michael MD‡; Avidan, Michael S. MBBCh§∥
Section Editor(s): Brull, Sorin J.
From the Departments of *Anesthesiology, and †Neurosurgery, University of Michigan Medical School, Ann Arbor, Michigan; ‡Department of Anesthesia and Critical Care, University of Chicago, Chicago, Illinois; and Departments of §Anesthesiology, and ∥Surgery, Washington University School of Medicine, St. Louis, Missouri.
Accepted for publication June 16, 2009.
Published ahead of print August 27, 2009
Supported by departmental and institutional funds.
GAM and RKE contributed equally to this manuscript.
Address correspondence and reprint requests to George A. Mashour, MD, PhD, Department of Anesthesiology, University of Michigan Medical School, 1H247 UH/SPC-5048, 1500 East Medical Center Drive, Ann Arbor, MI 48109-5048. Address e-mail to email@example.com.
BACKGROUND: Intraoperative awareness with explicit recall occurs in approximately 1–2 cases per 1000. Given the rarity of the event, a better understanding of awareness and its sequelae will likely require the compilation of data from numerous studies. As such, a standard description and expression of awareness events would be of value.
METHODS: We developed a novel classification instrument for intraoperative awareness events: Class 0: no awareness; Class 1: isolated auditory perceptions; Class 2: tactile perceptions (e.g., surgical manipulation or endotracheal tube); Class 3: pain; Class 4: paralysis (e.g., feeling one cannot move, speak, or breathe); and Class 5: paralysis and pain. An additional designation of “D” for distress was also included for patient reports of fear, anxiety, suffocation, sense of doom, sense of impending death, or other explicit descriptions. We reviewed 15 studies of the incidence of awareness that provided specific information about awareness reports. Five anesthesiologists at three institutions who developed the categories independently classified the events. An additional 20 individuals (attending anesthesiologists, anesthesiology residents, nurse anesthetists, medical students, and ancillary staff) not involved in the development of the categories also independently classified the events. Fleiss's kappa statistic was used to evaluate inter-observer agreement.
RESULTS: One hundred fifty-one cases of intraoperative awareness in adults were identified as valid for analysis. The overall kappa value was 0.851 (0.847–0.856, 95% confidence interval) for the basic Classes 1–5. Including additional designations of emotional distress, the overall kappa value was 0.779 (0.776–0.783, 95% confidence interval).
CONCLUSION: We report a novel classification instrument for intraoperative awareness events that has excellent inter-observer agreement and that may facilitate the study of intraoperative awareness.
Intraoperative awareness with explicit recall is a complication that is concerning to patients, anesthesia providers, and the general public. A multicenter study in the United States estimated an incidence of awareness with explicit recall of approximately 0.13%,1 which is consistent with a prospective European study demonstrating awareness in 1–2 of 1000 cases.2 Other studies have reported a much lower3 or much higher4 incidence. Thus, the magnitude of the problem is as yet unclear. Awareness is associated with long-term psychological consequences, including posttraumatic stress disorder.5–8
Because intraoperative awareness is rare, a full understanding of its incidence and sequelae will likely require the compilation and analysis of data from multiple patient populations, institutions, and studies. As such, a standard method for describing and reporting awareness events would be of value to investigators. Here, we report a novel classification instrument for intraoperative awareness events with excellent inter-observer agreement.
We reviewed articles from 1961 (the first incidence study of awareness) to the present time using “intraoperative awareness,” “anesthesia awareness,” and “awareness during general anesthesia” as key words for a PubMed literature search. Based on the awareness reports in the literature, classification categories for intraoperative awareness events were developed (Table 1). All types of awareness events reported in the large studies by Sebel et al.1 and Errando et al.4 were included, supporting content validity of the instrument.
Class 0 was constructed to reflect patient complaints about events that have a high likelihood of occurring in the preoperative or postoperative period, unconfirmed awareness, or dreaming. Because the vast majority of the reports in the literature only describe cases that have already been determined to be definite awareness, Class 0 was not a focus of this study. Classes 1–4 reflect commonly reported awareness experiences, whereas Class 5 reflects a combination of experiences (paralysis and pain) that may potentially be more distressing to the patient than either event alone. Because emotional distress was the only known predictor of late psychological symptoms in a study of awareness patients,7 we created a separate designation (“D” for Distress) that is independent of the reported experience.
As noted by Sebel et al.,1 more than one description of an awareness event may occur for a given case. In this situation, we graded an event based on the feature associated with the highest class. For example, if a patient described only an auditory perception, the event would be a Class 1. If, however, a patient reported an auditory perception and the feeling of paralysis, then the event was considered Class 4.
The reports of intraoperative awareness identified in the literature were reproduced exactly in a spreadsheet format and were distributed to evaluators. Five anesthesiologists at three different academic medical centers developed the categories (GAM, KKT, DBG, MOC, and MSA) and independently classified all cases valid for analysis. Additionally, 20 other individuals (attending anesthesiologists, anesthesiology residents, nurse anesthetists, medical students, and ancillary staff) at the three institutions who were not involved in the development of the categories also independently classified the events.
SAS v.9.1 (SAS Institute, Cary, NC) was used for statistical analysis. The magree SAS macro (available at: http://support.sas.com/kb/25/006.html) was used to test the agreement among multiple evaluators, using methodologies described by Fleiss et al.9,10 The algorithm outputs a generalized kappa coefficient, which is an adaptation of Scott's pi for multiple raters. The algorithm also calculates kappa coefficients for individual evaluations, based on a consideration of the proportion of all assignments, which went to a particular category. The overall kappa represents the degree to which the observed agreement exceeds chance agreement. Fleiss's kappa values were obtained for both the basic instrument (Classes 1–5) and the one including emotional distress (Classes 1, 1D, 2, 2D, 3, 3D, 4, 4D, 5, and 5D). The scale from Landis and Koch11 was used to assess agreement based on the kappa value.
We identified 15 studies that provided specific information about cases of intraoperative awareness.1–4,7,12–21 From these studies, we identified 151 cases valid for analysis. The 25 independent classifications for 151 cases had an overall kappa value of 0.851 for the basic Classes 1–5; the expanded instrument including the distress modifier had an overall kappa value of 0.779 (Tables 2 and 3). Determination of emotional distress decreased agreement in all five classes.
The development of classification systems has been beneficial in the study of rare perioperative events. For example, Han et al.22 first reported a scale that was subsequently used to elucidate the incidence and predictors of difficult and impossible mask ventilation.23 We have developed a novel classification of intraoperative awareness events in the hope of facilitating the future study of awareness incidences, risk factors, prevention strategies, and predictors of long-term psychological consequences. To our knowledge, this is the first classification instrument for intraoperative awareness events reported in the literature.
We acknowledge that it is difficult to standardize the individual and subjective experiences of patients who have suffered intraoperative awareness. Despite this inherent limitation, the inter-observer agreement of our instrument was consistently excellent for the basic categories of intraoperative awareness events. The determination of emotional distress was associated with less agreement, perhaps because of the difficulty in distinguishing the explicit patient report of distress from what the evaluator presumed to be distressing. It is important to note, however, that the event descriptions in the literature were not intended to be classified; based on the “almost perfect agreement” for the five classes, we anticipate that the prospective use of our classification will have improved agreement. The current analysis also suggests that patients should be explicitly asked about associated “emotional distress,” especially given its value in predicting long-term psychological sequelae.7
Although classes 0–5 would appear to be a rating scale, validation was based solely on categories. Until future validation is established, the instrument should be used and interpreted as categories rather than a ranked end point. We cannot assert that the classification is necessarily predictive of adverse outcomes in its current form; its predictive validity cannot be established based on awareness reports found in the literature dating back to 1961 with incomplete follow-up. Prospective study is ongoing in patients at high risk for awareness (MSA, MOC, and DBG—clinical trial NCT00682825) and the general surgical population (GAM and KKT—clinical trial NCT00689091). The reliability of the instrument determined in this study is an important first step.
In conclusion, we have developed a novel classification instrument for intraoperative awareness events with excellent inter-observer agreement. We believe that future studies of intraoperative awareness and its psychological sequelae will benefit from a common and standard description of awareness cases.
We thank the many individuals from the University of Michigan, University of Chicago, and Washington University who participated as evaluators in this study.
1.Sebel PS, Bowdle TA, Ghoneim MM, Rampil IJ, Padilla RE, Gan TJ, Domino KB. The incidence of awareness during anesthesia: a multicenter United States study. Anesth Analg 2004;99:833–9
2.Sandin RH, Enlund G, Samuelsson P, Lennmarken C. Awareness during anaesthesia: a prospective case study. Lancet 2000; 355:707–11
3.Pollard RJ, Coyle JP, Gilbert RL, Beck JE. Intraoperative awareness in a regional medical system: a review of 3 years' data. Anesthesiology 2007;106:269–74
4.Errando CL, Sigl JC, Robles M, Calabuig E, García J, Arocas F, Higueras R, Del Rosario E, López D, Peiró CM, Soriano JL, Chaves S, Gil F, García-Aguado R. Awareness with recall during general anaesthesia: a prospective observational evaluation of 4001 patients. Br J Anaesth 2008;101:178–85
5.Osterman JE, Hopper J, Heran WJ, Keane TM, van der Kolk BA. Awareness under anesthesia and the development of posttraumatic stress disorder. Gen Hosp Psychiatry 2001;23:198–204
6.Osterman JE, van der Kolk BA. Awareness during anesthesia and posttraumatic stress disorder. Gen Hosp Psychiatry 1998; 20:274–81
7.Samuelsson P, Brudin L, Sandin RH. Late psychological symptoms after awareness among consecutively included surgical patients. Anesthesiology 2007;106:26–32
8.Lennmarken C, Sydsjo G. Psychological consequences of awareness and their treatment. Best Pract Res Clin Anaesthesiol 2007;21:357–67
9.Fleiss J, Levin B, Paik M. Statistical methods for rates and proportions. 3rd ed. New York: John Wiley & Sons, Inc., 2003
10.Fleiss J, Nee J, Landis J. Large sample variance of kappa in the case of different sets of raters. Psychol Bull 1979;86:974–7
11.Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics 1977;33:159–74
12.Avidan MS, Zhang L, Burnside BA, Finkel KJ, Searleman AC, Selvidge JA, Saager L, Turner MS, Rao S, Bottros M, Hantler C, Jacobsohn E, Evers AS. Anesthesia awareness and the bispectral index. N Engl J Med 2008;358:1097–108
13.Chaiwat O, Vacharaksa K, Prakanrattana U. Awareness under general anesthesia. J Med Assoc Thai 2002;85:S948–S955
14.Hutchinson R. Awareness during surgery. A study of its incidence. Br J Anaesth 1961;33:463–9
15.Liu WH, Thorp TA, Graham SG, Aitkenhead AR. Incidence of awareness with recall during general anaesthesia. Anaesthesia 1991;46:435–7
16.Mashour GA, Wang LYJ, Turner CR, Shanks A, Vandervest J, Tremper KK. A retrospective study of intraoperative awareness with methodological implications. Anesth Analg 2009;108:521–6
17.Myles PS, Leslie K, McNeil J, Forbes A, Chan MT. Bispectral index monitoring to prevent awareness during anaesthesia: the B-Aware randomised controlled trial. Lancet 2004;363:1757–63
18.Nordstrom O, Englstrom AM, Persson S, Sandin R. Incidence of awareness in total i.v. anaesthesia based on propofol, alfentanil and neuromuscular blockade. Acta Anaesthesiol Scand 1997;41:978–84
19.Ranta SO, Laurila R, Saario J, Ali-Melkkila T, Hynynen M. Awareness with recall during general anesthesia: incidence and risk factors. Anesth Analg 1998;86:1084–9
20.Sandin R, Norstrom O. Awareness during total i.v. anaesthesia. Br J Anaesth 1993;71:782–7
21.Wilson SL, Vaughan RW, Stephen CR. Awareness, dreams, and hallucinations associated with general anesthesia. Anesth Analg 1975;54:609–17
22.Han R, Tremper KK, Kheterpal S, O'Reilly M. Grading scale for mask ventilation. Anesthesiology 2004;101:267
23.Kheterpal S, Han R, Tremper KK, Shanks A, Tait AR, O'Reilly M, Ludwig TA. Incidence and predictors of difficult and impossible mask ventilation. Anesthesiology 2006;105:885–91
This article has been cited 2 time(s).
British Journal of AnaesthesiaPsychological impact of unexpected explicit recall of events occurring during surgery performed under sedation, regional anaesthesia, and general anaesthesia: data from the Anesthesia Awareness RegistryBritish Journal of Anaesthesia
Journal of Psychosocial Nursing and Mental Health ServicesAnesthesia Awareness-Induced Posttraumatic Stress DisorderJournal of Psychosocial Nursing and Mental Health Services
© 2010 International Anesthesia Research Society