Secondary Logo

Journal Logo

Validation of 3-minute diagnostic interview for CAM-defined Delirium to detect postoperative delirium in the recovery room

A prospective diagnostic study

Olbert, Maria; Eckert, Sophie; Mörgeli, Rudolf; Kruppa, Jochen; Spies, Claudia D.

European Journal of Anaesthesiology (EJA): September 2019 - Volume 36 - Issue 9 - p 683–687
doi: 10.1097/EJA.0000000000001048
Diagnostic and prediction
Free
SDC

BACKGROUND Recent guidelines on postoperative delirium (POD) recommend POD screening in all patients, using a validated tool, starting in the recovery room. An operationalisation of the Confusion Assessment Method (CAM) criteria, the 3-Minute Diagnostic Interview for CAM-defined Delirium (3D-CAM), has been developed for use in general medical units.

OBJECTIVES The aim of this study was to evaluate 3D-CAM performance in an adult patient population to detect POD in the recovery room.

DESIGN A prospective diagnostic study.

SETTING Recovery room of a tertiary care university hospital in Berlin, Germany, in 2017.

PATIENTS Patients at least 18 years of age undergoing elective surgery.

MAIN OUTCOME MEASURES Patients were subjected to evaluation by blinded investigators using the 3D-CAM and the Diagnostic and Statistical Manual of Mental Disorders 5 (DSM-5, reference standard). Sensitivity, specificity and positive predictive value (PPV) and negative predictive value (NPV) were analysed for 3D-CAM, in addition to test–retest and inter-rater reliability analyses.

RESULTS Sixteen out of 176 patients (9.1%) developed POD. The 3D-CAM demonstrated strong test performance (specificity 0.88, sensitivity 1.0, area under the curve 0.94, PPV 0.44 and NPV 1.0), with a test–retest reliability of 90% (n = 10) and inter-rater reliability of 80% (n = 10).

CONCLUSION In this diagnostic study, 3D-CAM showed strong performance for detection of POD in the recovery room. Due to the low training requirements, fast application and high sensitivity, it might be particularly appropriate for clinical staff with limited experience in the assessment of POD.

TRIAL REGISTRATION ClinicalTrials.gov Identifier: NCT02992717

From the Department of Anesthesiology and Operative Intensive Care Medicine (MO, SE, RM, CDS) and Institute of Biometry and Clinical Epidemiology (JK), Charité – Universitätsmedizin Berlin, corporate member of Freie Universität Berlin, Humboldt-Universität zu Berlin, and Berlin Institute of Health; Berlin Institute of Health (BIH), Berlin, Germany (MO, JK)

Correspondence to Claudia D. Spies, MD, Department of Anesthesiology and Operative Intensive Care Medicine (CCM, CVK), Charité – Universitätsmedizin Berlin, corporate member of Freie Universität Berlin, Humboldt-Universität zu Berlin, and Berlin Institute of Health, Augustenburger Platz 1, Berlin D-13353, Germany Tel: +49 30 450 551102/+49 30 450 531012; e-mail: claudia.spies@charite.de

Published online 11 July 2019

Supplemental digital content is available for this article. Direct URL citations appear in the printed text and are provided in the HTML and PDF versions of this article on the journal's Website (www.ejanaesthesiology.com).

Back to Top | Article Outline

Introduction

Postoperative delirium (POD) is a common and serious neurocognitive complication.1 According to the Diagnostic and Statistical Manual of Mental Disorders 5 (DSM-5) criteria, delirium is defined as an acute (and fluctuating) disturbance in attention and cognition, which is not based on a pre-existing neurocognitive disorder.2 The incidence of POD depends on predisposing factors (e.g. age, cognitive impairment, comorbidity or impaired functional status) and precipitating risk factors (e.g. major surgery, cardiac surgery or ICU stay) and ranges from 5 up to 80%.1,3 POD is associated with increased length of hospitalisation,4 impairment of functional status5, long-term cognitive impairments,6–8 as well as short-term and long-term increased mortality.1

The 2017 guideline on POD from the European Society of Anaesthesiology recommends POD screening with a validated tool in all patients, starting in the recovery room, and once per shift up to 5 days after surgery.1 Over time, several test instruments were developed for screening POD, differing from each other in terms of training requirements, duration of application and setting. An optimal test instrument should have a high sensitivity and specificity, can be performed quickly and should not require extensive training.

In 2014, Marcantonio et al.9 published an instrument that operationalised the four diagnostic features of the Confusion Assessment Method (CAM), calling it the 3-min Diagnostic Interview for CAM-defined delirium (3D-CAM). The authors showed that the 3D-CAM could demonstrate strong performance with a sensitivity of 95% and a specificity of 94% compared with the reference standard, requiring approximately 3 min to apply, and high inter-rater agreement with low training requirements. In previous work, the 3D-CAM was translated into the German language and was adapted for use in the recovery room.10

Diagnostic studies for detection of POD in the recovery room are rare. 3D-CAM has been validated for patients at least 75 years of age in general medical units. The aim of this study was to evaluate 3D-CAM performance in an adult patient population in the recovery room.

Back to Top | Article Outline

Materials and methods

This prospective diagnostic study was conducted at the Department of Anesthesiology and Operative Intensive Care Medicine, Charité – Universitätsmedizin Berlin, Germany, a tertiary care university hospital. The study was approved by the local independent Charité Ethics Committee, Charité – Universitätsmedizin Berlin, Germany (ref.: EA1/261/16, Chairman Dr. K. Orzechowski) in September 2016, and was conducted in accordance with the declaration of Helsinki (ClinicalTrials.gov: NCT02992717). Written informed consent was obtained from all patients, and all local data privacy regulations were followed.

Inclusion criteria were age at least 18 years, admission to the recovery room or postanaesthesia care unit (PACU) following elective general anaesthesia, and an expected hospital stay of at least 24 h following surgery. We excluded patients with a previously diagnosed cognitive or psychiatric disorder, speech, visual or hearing impairments, and patients who were illiterate or not fluent in German.

Delirium assessment was conducted independently from the routine hospital procedures by a research team that was trained and supervised by a psychiatrist and delirium expert, and with access to the 3D-CAM Training Manual. According to the study protocol, each enrolled patient was subjected to pre-operative 3D-CAM and reference standard assessments in order to obtain a baseline. Delirium assessment was conducted in the recovery room within 30 min of extubation. The assessment was conducted only if the Richmond Agitation Sedation Scale (RASS)11 score was at least -2 (corresponding to light sedation – patient briefly awakens to voice, eye opening and contact <10 s). If RASS score was insufficient, the delirium assessment was postponed to a later time. Each patient was evaluated independently by two investigators, who were blinded to each other's results. One investigator evaluated DSM-5 criteria as a reference standard, while the other performed the adapted German version of 3D-CAM for the recovery room as described by Olbert et al.10 (the adaptation simply replaces the phrase ‘during the past day’ with ‘since the surgery’). Twenty patients took part in a reliability analysis: in addition to the reference standard, each of these patients was tested twice with the 3D-CAM. For test–retest reliability, 10 patients were tested independently by two different investigators, who were blinded to each other's results. For inter-rater reliability, 10 patients were tested and simultaneously evaluated by two different investigators. If there was a disagreement between the investigators, a field meeting was carried out to discuss the assessment in detail and to make a final decision for the validation analysis.

If POD was detected at any time by our research team, members of the hospital staff were immediately notified, and treatment initiated in accordance to the Standard Operation Procedure for POD of the Department of Anesthesiology and Operative Intensive Care Medicine.

The following baseline measurements were collected from routine data to describe the study population: age, sex, physical status according to the American Society of Anesthesiologists (ASA PS), surgical procedure, duration of anaesthesia, length of stay in the recovery room and total hospitalisation period.

Back to Top | Article Outline

Statistical analysis

Sample size was calculated for receiver operating characteristics (ROC) analysis. For an expected area under the curve (AUC) from 3D-CAM of 0.75 and an assumed minimal relevant difference between AUC of 0.1 (power = 80%, test significance level α = 0.05), a sample size of 200 patients was calculated. Baseline characteristics were expressed as median [IQR] or frequencies with percentages. Differences between groups were tested using Mann–Whitney U-test or Fisher's exact test. Sensitivity, specificity and positive (PPV) and negative predictive values (NPV) compared with the reference standard (DSM-5 criteria) were determined. Diagnostic test performance of 3D-CAM was evaluated by ROC analysis. Descriptive analysis and Kappa (κ) statistics are presented in an exploratory approach for test–retest reliability and inter-rater reliability. All calculations were performed with IBM SPSS Statistics, Version 23 (Copyright 1989, 2015 by SPSS Inc., Chicago, Illinois, USA).

Back to Top | Article Outline

Results

In total, 279 patients were screened between 23 January 2017 and 3 January 2018. Of these, 200 patients were included in the study. In all, 176 patients completed both the 3D-CAM and reference standard assessments in the recovery room (Fig. 1), and 9.1% of them were diagnosed with POD by the reference standard. Patients with delirium were significantly older, had higher ASA physical status scores and had a longer duration of anaesthesia and length of hospital stay (Table 1). No sex differences between patients with and without delirium were observed. An overview of surgical procedures related to this study is provided in the supplementary data, http://links.lww.com/EJA/A212.

Fig. 1

Fig. 1

Table 1

Table 1

Test results of 3D-CAM assessment are summarised in Table 2. Sensitivity, specificity, PPV and NPV were calculated. The 3D-CAM was positive in 36 out of 176 patients and demonstrated strong test performance. It showed a high specificity (0.88) at the highest sensitivity (1), and an AUC of 0.94 (95% CI 0.90 to 0.97, P < 0.001). PPV and NPV were 0.44 and 1, respectively.

Table 2

Table 2

For test–retest reliability, 10 patients received the 3D-CAM test twice. Nine out of the 10 assessments showed identical results, while a single assessment showed differing results (κ = 0.68, P = 0.35). For inter-rater reliability, 10 patients receiving the 3D-CAM were evaluated simultaneously by two different observers. Eight of these 10 assessments showed identical results, while two assessments differed between observers (κ = 0.60, P = 0.58).

The median time required to apply the 3D-CAM was 3.05 [IQR 2.27 to 4.00] min, and the 3D-CAM assessment was performed at a median of 33 [26 to 44] min after extubation. An overview of distribution of the sedation scoring during 3D-CAM assessment is listed in the supplementary data, http://links.lww.com/EJA/A212. Nearly 90% of the assessments were performed at a RASS score of 0 or −1. The median time between the assessments (3D-CAM and DSM-5) was 4 [3 to 6] min.

An analysis of the causes for false-positive results demonstrated that in 40% of these cases (eight cases), the positive assessment was due to questions 21 and 22 of the 3D-CAM. In addition, 30% of the assessments (six cases) showed positive results due to pre-existing deficiencies.

Back to Top | Article Outline

Discussion

In this study, 9.1% of patients developed POD in the recovery room. Our goal was to evaluate the performance of 3D-CAM in the recovery room with an adult patient population. A recent guideline-conform translation of the 3D-CAM into the German language was performed, enabling German-speaking countries to utilise the tool.10 In addition, the content of the 3D-CAM was adapted for use in the recovery room.10 The adopted German version demonstrated a strong test performance, with a high sensitivity and high specificity compared with the reference standard DSM-5 criteria. The results are similar to those of Marcantonio et al.,9 who validated this tool for the general ward. We were also able to corroborate the short duration of the test application.

Due to a relatively high number of false-positive results, the PPV of 3D-CAM was modest. An analysis of causes for false-positive results demonstrated that in 40% of these cases, the assessments become positive due to questions 21 and 22 of the 3D-CAM.9 Questions 21 and 22 are optional items, which must be completed only if the test indicates inattention and either disorganised thinking or altered level of consciousness without acute onset/fluctuation. These items serve to re-evaluate the feature acute onset/fluctuation, so as to not overlook an acute change and prevent false-negative results. Question 21 requires a review of the medical records, while question 22 requires a comparison with previous 3D-CAM assessments to help determine acute change. Question 22 will indicate an acute change if the patient displays any deviation from a previous 3D-CAM assessment (baseline assessments were available for all patients), regardless of what this change might be, possibly a single mistake or miscalculation. Therefore, question 22 has the potential to cause an overdiagnosis of POD. Furthermore, 30% of the false-positive assessments showed positive results due to pre-existing deficiencies. For example, if a patient is unable to count backwards in the pre-operative 3D-CAM assessment, it is unsurprising that the patient remains unable to do so in the postoperative assessment. Such situations increase false-positive measurements, as a pre-existing deficiency is registered as an acute onset of inattention. This is in line with results of Marcantonio et al.,9 which showed that the specificity of 3D-CAM for the detection of delirium decreases for patients with cognitive impairment. In addition, a persistent influence of anaesthesia may be considered as a possible reason for false-positive results. An analysis of the RASS scores of the false-positive assessments showed that the RASS scores were between −1 and 1 in 95% of these assessments. As this tool is intended for use as a screening instrument for clinical staff (e.g. nurses) in the recovery room, from our point of view, the high sensitivity is much more important than the lower specificity.

A part of the assessments was performed or evaluated twice for reliability. It must be noted that the sample size for reliability analysis was very low, so that the κ coefficient is presented for completeness and must be interpreted carefully. Nevertheless, the analysis provides valuable input. First, an investigation of test–retest performance was important due to the fluctuating character of the delirium. Here, a descriptive analysis showed 90% agreement. This may possibly reflect the responsiveness of the 3D-CAM assessment to a fluctuating condition. Second, an investigation of inter-rater performance could provide valuable information about training requirements. For the inter-rater assessment, one 3D-CAM evaluation was performed by a delirium expert trained on the 3D-CAM Training Manual, while the other evaluation was performed by a psychology student trained on the 3D-CAM Training Manual, but lacking additional expertise in POD. An agreement of 80% suggests that less experienced staff can safely perform the 3D-CAM assessment, especially considering that the 20% disagreement was entirely composed of false positive results from the student. The exploratory analysis can only provide an estimation, as a robust reliability analysis would require a larger sample size. In addition, further studies are needed to adequately evaluate fluctuation issues and training requirements.

For an experienced recovery room nurse, the test duration of 3 min may appear to be too long. However, for clinical staff with relatively little experience with delirium, the structure of the items makes the 3D-CAM a nearly ideal tool for use in clinical routine.

Back to Top | Article Outline

Strengths and limitations

With this study, we were able to validate the 3D-CAM for detection of POD in the recovery room in an adult population over a broad spectrum of operative disciplines. As the assessments were blinded, high-quality standards could be achieved.

There are, however, several limitations to our study. The low number of delirious patients is a limitation to be considered in future studies, due in part to a decline in delirium rates since the publication and implementation of the guideline on POD. In addition, although the sample size calculation required 200 patients, only 176 patients could ultimately be included in the final analysis (see Fig. 1). Another important limitation of this validation analysis is the low number of patients for the reliability measurements. Due to the small number of measurements (only 10), only exploratory analysis could be provided for inter-rater and test–retest reliability.

Furthermore, the validation was performed in a research setting in a tertiary care university hospital. Transferability into routine care, community or nontertiary care university hospitals must be investigated in the future.

In conclusion, in this diagnostic study, 3D-CAM showed strong performance for detection of POD in the recovery room. Due to the low training requirements, fast application and high sensitivity, it might be particularly appropriate for clinical staff with limited experience in the assessment of POD.

Back to Top | Article Outline

Acknowledgements relating to this article

Assistance with the study: the authors would like to thank Julia Börner, Rayan El-Maj-Mohamad, Martin Findler, Juliane Friedrich, Brais Gonzales Sousa, Annika Kratzsch, Dieka Ohling, Anna Palatini, Lukas Roediger, Julia Schaefer and Jule Schröder for their assistance in data collection. Furthermore, the authors thank Kathrin Scholtz for her assistance in project management and quality assurance.

Financial support and sponsorship: the study was developed and conducted as an internal quality management project. No external funding was required.

Conflicts of interest: none.

Presentation: none.

Back to Top | Article Outline

References

1. Aldecoa C, Bettelli G, Bilotta F, et al. European Society of Anaesthesiology evidence-based and consensus-based guideline on postoperative delirium. Eur J Anaesthesiol 2017; 34:192–214.
2. American Psychiatric Association. Diagnostic and Statistical Manual of Mental Disorders. 5th ed.Washington, DC: American Psychiatric Association; 2013.
3. Inouye SK, Charpentier PA. Precipitating factors for delirium in hospitalized elderly persons. Predictive model and interrelationship with baseline vulnerability. JAMA 1996; 275:852–857.
4. Radtke FM, Franck M, MacGuill M, et al. Duration of fluid fasting and choice of analgesic are modifiable factors for early postoperative delirium. Eur J Anaesthesiol 2010; 27:411–416.
5. Liang CK, Chu CL, Chou MY, et al. Interrelationship of postoperative delirium and cognitive impairment and their impact on the functional status in older patients undergoing orthopaedic surgery: a prospective cohort study. PLoS One 2014; 9:e110339.
6. Drews T, Franck M, Radtke FM, et al. Postoperative delirium is an independent risk factor for posttraumatic stress disorder in the elderly patient: a prospective observational study. Eur J Anaesthesiol 2015; 32:147–151.
7. Inouye SK, Marcantonio ER, Kosar CM, et al. The short-term and long-term relationship between delirium and cognitive trajectory in older surgical patients. Alzheimers Dement 2016; 12:766–775.
8. Lingehall HC, Smulter NS, Lindahl E, et al. Preoperative cognitive performance and postoperative delirium are independently associated with future dementia in older people who have undergone cardiac surgery: a longitudinal cohort study. Crit Care Med 2017; 45:1295–1303.
9. Marcantonio ER, Ngo LH, O’Connor M, et al. 3D-CAM: derivation and validation of a 3-min diagnostic interview for CAM-defined delirium: a cross-sectional diagnostic test study. Ann Int Med 2014; 161:554–561.
10. Olbert M, Eckert S, Morgeli R, et al. 3D-CAM guideline-conform translation for German-speaking countries. Anasthesiol Intensivmed Notfallmed Schmerzther 2018; 53:793–796.
11. Sessler CN, Gosnell MS, Grap MJ, et al. The Richmond Agitation-Sedation Scale: validity and reliability in adult intensive care unit patients. Am J Respir Crit Care Med 2002; 166:1338–1344.

Supplemental Digital Content

Back to Top | Article Outline
© 2019 European Society of Anaesthesiology