The total number of behaviors observed (with neutral expression being recoded to “present = zero”) was summed to calculate the total BPAT scores, with the highest possible score being 8. Significant positive correlations were found between the mean number of behaviors and the mean pain intensity and pain distress NRS scores. At rest before the procedure, low positive Pearson correlations were obtained (0.28 and 0.24) for pain intensity and pain distress, respectively (P < 0.001). Pain intensity and pain distress self-reports were highly correlated (r = 0.71, P < 0.001). During the procedure, moderate positive Pearson correlations were found between mean behavior scores and mean pain intensity and pain distress scores (0.54 and 0.49, respectively [P < 0.001]). Again, pain intensity and pain distress self-reports were highly correlated (r = 0.79, P < 0.001).
Considering that moderate positive correlations were found between total BPAT scores and pain intensity and pain distress scores during the procedure, receiver operating characteristic curve analyses were performed to establish the ability of the BPAT to discriminate between patients who reported experiencing pain at specific NRS thresholds. Mild (≥1), moderate (≥5), and severe (≥8) levels of pain intensity and pain distress were used as criteria.41 As can be seen in Table 10, low values of AUC (≤0.60) were found at mild and at moderate levels of pain intensity and pain distress (ie, NRS score ≥1, and ≥5, respectively), indicating poor discrimination of the BPAT, ie, not better than chance.31 Yet, acceptable values of AUC (0.73-0.75) indicating good discrimination of the scale31 were found at severe levels of pain intensity and pain distress (ie, NRS score ≥8). Therefore, the BPAT showed a moderate ability to discriminate patients who reported a severe level of pain (ie, ≥8) from those who reported a pain level <8.
Although a BPAT score >2.5 was found to be the cut-point score that maximized both sensitivity (ie, the true-positive rate) and specificity (ie, the true-negative rate) at mild levels of pain intensity and pain distress (ie, NRS score ≥1), sensitivity and specificity findings were very low (<42%) (Table 10). At moderate and severe levels of pain intensity and pain distress (ie, NRS score ≥5), a BPAT cut-point score >2.5 led to higher sensitivity findings (ie, >71%), but specificity findings remained low (ie, <46%). The best BPAT cut-point score that maximized both sensitivity and specificity was found to be >3.5 at moderate levels of pain intensity and pain distress with sensitivity findings ≥52%, and specificity findings ≥61%. At severe pain levels, a BPAT cut-point score >2.5 was associated with higher sensitivity findings (ie, >90%) but with lower specificity findings (ie, <46%). On the other hand, using a BPAT cut-point score >3.5, sensitivity and specificity findings were higher (61.8%-75.1%) at severe levels of pain intensity and pain distress (ie, NRS score ≥8). In summary and according to AUC sensitivity and specificity findings altogether, the BPAT showed better results in classifying patients with or without severe levels of pain intensity and pain distress and the best BPAT cut-point score was found to be >3.5.
We conducted a detailed psychometric analysis of a new behavioral pain assessment tool used when adult ICU patients were at rest and during procedures. The BPAT was found to be reliable and valid in this population and context. Its validation in heterogeneous patients in several ICUs from several countries provides a high degree of generalizability to our findings.
Concordance was high between the 2 clinician raters, ie, at least 89%-82% before and during the procedure, respectively. Thus, considering that it is not feasible to have 2 nurses available for pain behavior observations in the clinical context, BPAT pain assessments can be done by one rater after appropriate training. The highest kappas were for eyes closed which represent a neutral item easily identified by raters. During the procedure, high kappas were obtained for facial expression items (ie, neutral, grimace, wince) and 2 verbal sounds (ie, moaning and verbal complaints). The lowest kappas, at both times, were for muscle rigidity. In this study, muscle rigidity was identified by observation. Although the BPS also directs clinicians to observe upper extremity rigidity,43 with the CPOT,22 muscle rigidity is tested by the clinician while moving the patient's arm. Intensive care unit clinicians need to be trained on testing muscle rigidity because it appears more difficult to assess visually.
Observation of behaviors was influenced by whether the assessment was preprocedure or procedure, and on the patient's sedation level. At rest, neutral expression was by far the most frequently observed behavior, and almost one-third of patients had their eyes closed during this time. Conversely, most behaviors (except for neutral expression and closed eyes) were more frequently seen during the procedure when an acute reflexive pain response could be expected to occur. Behaviors such as grimace, wince, and muscle rigidity have been demonstrated to be pain-related behaviors in previous studies.20,47
Observation of most behaviors (except for closed eyes) was dependent on the patient's level of sedation, with increase in behavior presence being in patients who were less sedated. The finding about closed eyes being more frequently observed when patients were more heavily sedated is reasonable because sedation often leads to closed eyes. In previous research,24 conscious ICU patients had higher CPOT scores during turning than did unconscious patients, and those who received sedatives had lower CPOT scores. Patients who are sedated are less likely to exhibit behaviors, but this may not mean that they do not have pain. When there is a reason to suspect pain, an intervention should be trialed.29
We demonstrated that behavior items of the BPAT did not discriminate between patients able vs unable to self-report because the proportion of behaviors observed both before and during the procedure was similar for both groups. This finding provides support for the validity of the tool use in nonreporting patients, a situation in which a behavioral pain assessment tool is needed the most.29 There were some differences; however, closed eyes were seen more often in patients unable to self-report and who were more likely to be sedated, and moaning and verbal complaints were more frequently observed during the procedure in patients able to self-report. These findings confirm discriminant validation and make clinical sense. However, the BPAT directions for use may need more specificity. For instance, while “closing eyes” or “eye closure” has been related to the facial expression of pain,45 the terminology “eyes closed” was included in the BPAT. Thus, “closing eyes” should be considered in further research with the BPAT.
Most BPAT behaviors significantly predicted the degree of the patient's self-report of pre-procedural pain intensity. Furthermore, all behaviors significantly predicted the degree of the patient's self-report of procedural pain intensity as well as pain distress. In addition, we found moderate positive Pearson correlations between mean BPAT scores and mean pain intensity and distress scores during the procedure. Similar findings were found for the CPOT and BPS.1,8,16,22,24 Correlations between the behavioral and self-report scale scores were moderate but not high. In actuality, these scales focus on different, but interrelated, pain dimensions: pain intensity (ie, sensory dimension), distress (ie, affective dimension), and behaviors (ie, behavioral dimension; in this case, motor responses to a noxious event).38,39 Although previous research established the relationship between pain intensity and pain behaviors,20,35,47 only one recent study has established the relationship between pain distress (unpleasantness) and behaviors (per CPOT scores), with a positive mild Pearson correlation of 0.31 (P < 0.01) in postoperative cardiac surgery during mediastinal tube removal.6 Our study confirms a moderate correlation between pain distress and behavioral scores during common procedures performed in ICU patients. This moderate correlation demonstrates that affective and behavioral dimensions of pain are interrelated.
Significant positive correlations between the BPAT and NRS mean that their scores move in a linear fashion. However, they are on different scales, so their numbers do not have the same meaning; for example, a score of 7 on the BPAT is not the same as a score of 7 on the NRS. Thus, in clinical practice, the BPAT, like the BPS43 and CPOT,22 must be interpreted differently than the NRS.29 More specifically, NRS scores can be used to identify mild, moderate, and severe levels of pain. With behavioral pain assessment tools, only cut-point scores have been established for the presence of pain (yes/no), In addition, the CPOT was found to better classify the presence of pain when it was reported as moderate to severe by ICU patients.23
Despite positive correlations between the BPAT and pain intensity and distress scores, the total explained variance of the BPAT was no higher than 35%. There may be other behaviors that could be indicative of pain intensity and distress such as moving the hand toward the pain site, touching the pain site, and removing the painful stimulus.20,33 Future research is warranted that would explore other relevant behaviors as possible additions to the BPAT. However, pain is a multidimensional experience,38,39 and observing behaviors does not fully capture all dimensions of pain, although they remain the best option for the assessment of pain in patients unable to self-report.4,29 As also highlighted by Hadjistavropoulos and Craig,28 automatic expressive behaviors included in observational measures are subject to less purposeful distortion than are behaviors dependent upon higher mental processes necessary to self-report pain. Consequently, observational measures of behaviors have clinical utility as indices of pain when the patient's self-report is not available.
Using a sum score for the BPAT, with the possible range of 0 to 8, we found that the highest degree of sensitivity and specificity was associated with a BPAT score of 3.5; at this cut-point score, a patient could be having severe pain. That the BPAT appears to better discriminate patients with severe levels of pain intensity and pain distress (AUC > 0.70) is consistent with findings for the CPOT23,24 and BPS.11 Sensitivity was higher than specificity, meaning that the BPAT seems to have a better true-positive rate than true-negative rate. Thus, the BPAT has the ability to adequately classify patients with severe levels of pain, but the BPAT's ability to adequately classify patients not reaching severe levels of pain is not much better than chance (60%). Clinically, when a BPAT score >3.5 is achieved, it is highly likely that the patient is experiencing severe pain. Administration of an opioid, with the possible addition of nonopioid and nonpharmacological strategies, could benefit the patient. Patients with BPAT scores <3.5 may still experience some level of pain, and the use of nonopioid or nonpharmacological strategies could be trialed.
Unlike the BPS and CPOT, the BPAT does not have an item that relates to patient compliance with their ventilator. However, compliance with the ventilator may mean other things than pain such as the presence of secretions and the need for suctioning. Additional items related to body movements, added to the BPAT and tested, may increase the tool's sensitivity. Also, “closed eyes” might have been confused with the action of closing eyes by the observers, and “closing eyes” could be used in further validation of the BPAT.
Another limitation of our study is that we were unable to differentiate patients according to their specific injury and, thus, do not know if mechanism of injury affects validity of the BPAT. Further testing of the BPAT will be required in different ICU patient groups. Finally, this validation was conducted for research purposes, and further validation in daily ICU practice needs to be performed.
The BPAT was found to be reliable and valid for use in critically ill patients able or not to self-report. Valid behavioral scales are necessary to ensure appropriate assessment of pain and to guide decisions for pain management in this vulnerable population. The use of such scales is recommended in practice guidelines,4,52 and their clinical implementation in the ICU setting has led to improved pain management practices and better patient outcomes.10,12,21,49,51 Implementation studies with the BPAT are needed to determine the effects of its use on ICU pain management practices and patient outcomes. The differences in behavioral expression of pain in an international population are also a relevant topic for further exploration.
The authors have no conflicts of interest to declare.
The study, named Europain, was funded by the 2009 Established Investigator Award from the European Society of Intensive Care Medicine/European Critical Care Research Network and by an academic grant from Hôpital Saint-Louis, Paris, France. The authors gratefully acknowledge all of the national coordinators for the Europain study. They are listed in appendix A as supplemental digital content.
The authors also thank all ICU coordinators who were responsible for the data collection in their units, and all clinicians who were involved in data collection. Finally, the authors thank the ICU patients who agreed to participate in this study.
. Ahlers SJGM, van der Veen AM, van Dijk M, Tibboel D, Knibbe CAJ. The use of the behavioral pain scale to assess pain in conscious sedated patients. Anest Analg 2010;110:127–33.
. Arbour C, Choinière M, Topolovec-Vranic J, Loiselle C, Puntillo K, Gélinas C. Detecting pain in traumatic brain injured patients exposed to common procedures in the ICU: typical or atypical behaviors? Clin J Pain 2014;30:960–9.
. Arroyo-Novoa CM, Figueroa-Ramos MI, Puntillo KA, Stanik-Hutt J, Thompson CL, White C, Wild LR. Pain related to tracheal suctioning in awake acutely and critically ill adults: a descriptive study. Intensive Crit Care Nurs 2008;24:20–7.
. Barr J, Fraser GL, Puntillo KA, Ely W, Gélinas C, Dasta JF, Davidson JE, Devlin JW, Kress JP, Joffe AM, Coursin DB, Herr DL, Tung A, Robinson BRH, Fontaine DK, Ramsay MA, Riker RR, Sessler CN, Pun B, Skrobik Y, Jaeschke R. Clinical practice guidelines for the management of pain, agitation, and delirium in adult patients in the intensive care unit. Crit Care Med 2013;41:263–306.
. Bates D, Maechler M, Bolker BM, Walker SC. Fitting linear mixed-effects models using lme4. J Statistical Software 2015;67:1–48.
. Boitor M, Fiola Lachance J, Gélinas C. Validation of the critical-care pain observation tool and vital signs in relation to the sensory and affective components of pain during mediastinal tube removal in postoperative cardiac surgery intensive care unit adults. J Cardiovas Nurs 2016;31:425–32.
. Brislin RW. Back-translation for cross-cultural research. J Cross Cult Psychol 1970;1:187–216.
. Buttes P, Keal G, Cronin SN, Stocks L, Stout C. Validation of the critical-care pain observation tool in adult critically ill patients. Dimens Crit Care Nurs 2014;33:78–81.
. Carlsson K, Andersson J, Petrovic P, Petersson KM, Ohman A, Ingvar O. Predictability modulates the affective and sensory-discriminative neural processing of pain. Neuroimage 2006;32:1804–14.
. Chanques G, Jaber S, Barbotte E, Violet S, Sebbane M, Perrigault PF, Mann C, Lefrant JF, Eledjam JJ. Impact of systematic evaluation of pain and agitation in an intensive care unit. Crit Care Med 2006;34:1691–9.
. Chen YY, Lai YH, Shun SC, Chi NH, Tsai PS, Liao YM. The chinese behavior pain scale for critically ill patients: translation and psychometric testing. Int J Nurs Stud 2011;48:438–48.
. De Jong A, Molinari N, De Lattre S, Gniadek C, Carr J, Conseil M, Susbielles MP, Jung B, Jaber S, Chanques G. Decreasing severe pain and serious adverse events while moving intensive care unit patients: a prospective interventional study (the NURSE-DO project). Crit Care 2013;17:R74.
. Deogaonkar A, Gupta R, DeGeorgia M, Sabharwal V, Gopakumaran B, Schubert A, Provencio JJ. Bispectral Index monitoring correlates with sedation scales in brain-injured patients. Crit Care Med 2004;32:2403–6.
. Downie WW, Leatham PA, Rhind PM, Wright V, Branco JA, Anderson JA. Studies with pain rating scales. Ann Rheum Dis 1978;37:378–81.
. Dworkin RH, Turk DC, Farrar JT, Haythornthwaite JA, Jensen MP, Katz NP, Kerns RD, Stucki G, Allen RR, Bellamy N, Carr DB, Chandlen J, Cowan P, Dionne R, Galer BS, Hertz S, Jadad AR, Kramer LD, Manning DC, Martin S, McCormick CG, McDermott MP, McGrath P, Quessy S, Rappaport BA, Robbins W, Robinson JP, Rothman M, Royal MA, Simon L, Stauffer JW, Stein W, Tollet J, Wernicke J, Witter J. Core outcome measures for chronic pain clinical trials: IMMPACT recommendations. PAIN 2005;113:9–19.
. Echegaray-Benites C, Kapoustina O, Gélinas C. Validation of the critical-care pain observation tool in brain surgery patients during common procedures in the intensive care unit. Intensive Crit Care Nurs 2014;30:257–65.
. Ely EW, Truman B, Shintani A, Thomason JWW, Wheeler AP, Gordon S, Francis J, Speroff T, Gantam S, Margolin R, Sessler CN, Dittus RS, Bernard GR. Monitoring sedation status over time in ICU patients: reliability and validity of the richmond agitation-sedation scale (RASS). JAMA 2003;289:2983–91.
. Ekman P, Friesen EP. Manual for the facial action coding system. Palo Alto, CA, USA: Consulting Psychologists Press, 1978.
. Fischer JE, Bachmann LM, Jaeschke R. A readers' guide to the interpretation of diagnostic test properties: clinical example of sepsis. Intensive Care Med 2003;29:1043–51.
. Gélinas C, Arbour C. Behavioral and physiological indicators during a nociceptive procedure in conscious and unconscious mechanically ventilated adults: similar or different? J Crit Care 2009;24:628. e7–628. e17.
. Gélinas C, Arbour C, Michaud C, Vaillant F, Desjardins S. Implementation of the critical-care pain observation tool on pain assessment/management nursing practices in an intensive care unit with nonverbal critically ill adults: a before and after study. Int J Nurs Stud 2011;48:1495–504.
. Gélinas C, Fillion L, Puntillo K, Viens C, Fortier M. Validation of the critical-care pain observation tool in adult patients. Am J Crit Care 2006;15:420–7.
. Gélinas C, Harel F, Fillion L, Puntillo KA, Johnston C. Sensitivity and specificity of the critical-care pain observation tool for the detection of pain in intubated adults after cardiac surgery. J Pain Symptom Manage 2009;37:58–67.
. Gélinas C, Johnston C. Pain assessment in the critically ill ventilated adult: validation of the critical-care pain observation tool and physiologic indicators. Clin J Pain 2007;23:497–505.
. Gélinas C, Loiselle CG, Le May S, Ranger M, Bouchard E, McCormack D. Theoretical, psychometric, and pragmatic issues in pain measurement. Pain Manag Nurs 2008;9:120–30.
. Gélinas C, Puntillo KA, Joffe A, Barr JA. A validated approach to evaluating psychometric properties of pain assessment tools for use in nonverbal critically ill adults. Semin Respir Crit Care Med 2013;34:153–68.
. Gelman A, Hill J. Data analysis using regression and multilevel/hierarchical models. Cambridge, United Kingdom: Cambridge University Press, 2006.
. Hadjistavropoulos T, Craig KD. A theoretical framework for understanding self-report and observational measures of pain: a communication model. Behav Res Ther 2002;40:551–70.
. Herr K, Coyne PJ, McCaffery M, Manworren R, Merkel S. Pain assessment in the patient unable to self-report: position statement with clinical practice recommendations. Pain Manag Nurs 2011;12:230–50.
. Hofbauer RK, Rainville P, Duncan GH, Bushnell C. Cortical representation of the sensory dimension of pain. J Neurophysiol 2001;86:402–11.
. Hosmer DW, Lemeshow S. Applied logistic regression. 2nd ed. New York: John Wiley and Sons, 2000.
. Jensen MP, Karoly P, Braver S. The measurement of clinical pain intensity: a comparison of six methods. PAIN 1986;27:117–26.
. Joffe A, McNulty B, Boitor M, Marsh R, Gélinas C. Validation of the critical-care pain observation tool in brain-injured critically ill adults. J Cri Care 2016;36:76–80.
. Johnson JE, Rice VH. Sensory and distress components of pain: implications for the study of clinical pain. Nurs Res 1974;23:203–9.
. Kapoustina O, Echegaray-Benites C, Gélinas C. Fluctuations in vital signs and behavioral responses of brain surgery patients in the intensive care unit: are they valid indicators of pain? J Adv Nurs 2014;70:2562–76.
. Landis RJ, Koch GG. The measurement of observer agreement for categorical data. Biometrics 1977;33:159–74.
. Li D, Miaskowski C, Burkhardt D, Puntillo K. Evaluations of physiologic reactivity and reflexive behaviors during noxious procedures in sedated critically ill patients. J Crit Care 2009;24:472. e9–472. e13.
. McGuire DB. Comprehensive and multidimensional assessment and measurement of pain. J Pain Symptom Manage 1992;7:312–19.
. Melzack R, Casey KL. Sensory, motivational, and central control determinants of pain: a new conceptual model. In: Kenshalo D, editor. The skin senses: Springfield: Chas. C. Thomas, 1968. p. 423–443.
. Ogilvie MP, Pereira BMT, Ryan ML, Gomez-Rodriguez JC, Pierre EJ, Livingstone AS, Proctor KG. Bispectral index to monitor propofol sedation in trauma patients. J Trauma Inj Infect Crit Care 2011;71:1415–21.
. Paul SM, Zelman DC, Smith M, Miaskowski C. Categorizing the severity of cancer pain: further exploration of the establishment of cutpoints. PAIN 2005;113:37–44.
. Payen JF, Bosson JL, Chanques G, Mantz J, Labarere J. Pain assessment is associated with decreased duration of mechanical ventilation in the intensive care unit: a post Hoc analysis of the DOLOREA study. Anesthesiology 2009;111:1308–16.
. Payen JF, Bru O, Bosson JL, Lagrasta A, Novel E, Deschaux I, Lavagne P, Jacquet C. Assessing pain in critically ill sedated patients by using a behavioral pain scale. Crit Care Med 2001;29:2258–63.
. Price DD, Harkins SW, Baker C. Sensory-affective relationships among different types of clinical and experimental pain. PAIN 1987;28:297–307.
. Prkachin KM. The consistency of facial expressions of pain: a comparison across modalities. PAIN 1992;51:297–306.
. Puntillo KA, Max A, Timsit JF, Vignoud L, Chanques G, Robleda G, Roche-Campo F, Mancebo J, Divatia JV, Soares M, Ionescu DC, Grintescu IM, Vasiliu IL, Maggiore SM, Rusinova K, Owczuk R, Egerod I, Papathanassoglou EDE, Kyranou M, Joynt GM, Burghi G, Freebairn RC, Ho KM, Kaarlola A, Gerritsen RT, Kesecioglu J, Sulaj MMS, Norrenberg M, Benoit DD, Seha MSG, Hennein A, Periera FJ, Benbenishty JS, Abroug F, Aquilina A, Monte JRC, An Y, Azoulay E. Determinants of procedural pain intensity in the intensive care unit. The Europain(R) study. Am J Respir Crit Care Med 2014;189:39–47.
. Puntillo KA, Morris AB, Thompson CL, Stanik-Hutt J, White CA, Wild LR. Pain behaviors observed during six common procedures: results from thunder project II. Crit Care Med 2004;32:421–27.
. R Core Team. R: a language and environment for statistical computing. Vienna: R Foundation for Statistical Computing, 2012.
. Radtke FM, Heymann A, Franck M, Maechler F, Drews T, Luetz A, Nachtigall I, Wernecke KD, Spies CD. How to implement monitoring tools for sedation, pain and delirium in the intensive care unit: an experimental cohort study. Intensive Care Med 2012;38:1974–81.
. Reading AE. A comparison of pain rating scales. J Psychosom Res 1980;24:119–24.
. Rose L, Haslam L, Dale C, Knechtel L, McGillion M. Behavioral pain assessment tool for critically ill adults unable to self-report pain. Am J Crit Care 2013;22:246–55.
. Sauder P, Andreoletti M, Cambonie G, Capellier G, Feissel M, Gall O, Goldran-Toledano D, Kierzek G, Mateo J, Mentec H, Mion H, Rigaud JP, Seguin P. Conférence de consensus commune (SFAR-SRLF) en reanimation: Sédation-analgésie en reanimation (nouveau-né exclu). Réanimation 2008;17:600–12.
. Sessler CN, Gosnell MS, Grap MJ, Brophy GM, O'Neal PV, Keane KA, Tesoro EP, Elswick RK. The richmond agitation-sedation scale: validity and reliability in adult intensive care unit patients. Am J Respir Crit Care Med 2002;166:1338–44.
. Yaman F, Ozcan N, Ozcan A, Kaymak C, Basar H. Assesment of correlation between bispectral index and four common sedation scales used in mechanically ventilated patients in ICU. Eur Rev Med Pharmacol Sci 2012;16:660–6.
. Zweig MH, Campbell G. Receiver-operating characteristic (ROC) plots: a fundamental evaluation tool in clinical medicine. Clin Chem 1993;39:561–77.