Perioperative Medicine: Clinical Science
Prospective External Validation of a Predictive Score for Postoperative Pulmonary Complications
Mazo, Valentín M.D.; Sabaté, Sergi M.D., Ph.D.; Canet, Jaume M.D., Ph.D.; Gallart, Lluís M.D., Ph.D.; de Abreu, Marcelo Gama M.D., Ph.D.; Belda, Javier M.D., Ph.D.; Langeron, Olivier M.D., Ph.D.; Hoeft, Andreas M.D., Ph.D.; Pelosi, Paolo M.D.
Background: No externally validated risk score for postoperative pulmonary complications (PPCs) is currently available. The authors tested the generalizability of the Assess Respiratory Risk in Surgical Patients in Catalonia risk score for PPCs in a large European cohort (Prospective Evaluation of a RIsk Score for postoperative pulmonary COmPlications in Europe).
Methods: Sixty-three centers recruited 5,859 surgical patients receiving general, neuraxial, or plexus block anesthesia. The Assess Respiratory Risk in Surgical Patients in Catalonia factors (age, preoperative arterial oxygen saturation in air, acute respiratory infection during the previous month, preoperative anemia, upper abdominal or intrathoracic surgery, surgical duration, and emergency surgery) were recorded, along with PPC occurrence (respiratory infection or failure, bronchospasm, atelectasis, pleural effusion, pneumothorax, or aspiration pneumonitis). Discrimination, calibration, and diagnostic accuracy measures of the Assess Respiratory Risk in Surgical Patients in Catalonia score’s performance were calculated for the Prospective Evaluation of a RIsk Score for postoperative pulmonary COmPlications in Europe cohort and three subsamples: Spain, Western Europe, and Eastern Europe.
Results: The full Prospective Evaluation of a RIsk Score for postoperative pulmonary COmPlications in Europe data set included 5,099 patients; 725 PPCs were recorded for 404 patients (7.9%). The score’s discrimination was good: c-statistic (95% CI), 0.80 (0.78 to 0.82). Predicted versus observed PPC rates for low, intermediate, and high risk were 0.87 and 3.39% (score <26), 7.82 and 12.98% (≥26 and <45), and 38.13 and 38.01% (≥45), respectively; the positive likelihood ratio for a score of 45 or greater was 7.12 (5.93 to 8.56). The score performed best in the Western Europe subsample—c-statistic, 0.87 (0.83 to 0.90) and positive likelihood ratio, 11.56 (8.63 to 15.47)—and worst in the Eastern Europe subsample. The predicted (5.5%) and observed (5.7%) PPC rates were most similar in the Spain subsample.
Conclusions: The Assess Respiratory Risk in Surgical Patients in Catalonia score predicts three levels of PPC risk in hospitals outside the development setting. Performance differs between geographic areas.
What We Already Know about This Topic
* There is no externally validated and replicated risk assessment tool for postoperative pulmonary complications
What This Article Tells Us That Is New
* The Assess Respiratory Risk in Surgical Patients in Catalonia risk assessment tool was replicated and externally validated in over 5,000 patients across Europe
POSTOPERATIVE pulmonary complications (PPCs) are a major contributor to the overall risk of surgery.1–4
They are associated with substantially longer time spent in hospital5
and higher in-hospital postoperative mortality.6
In the United States, the reported annual economic burden of PPCs is approximately $3.42 billion (USD).7
A wide range of patient, anesthetic, and surgical factors are associated with PPCs.2
To date, only a few studies have developed predictive models for PPCs in settings that reached beyond very specific disease or surgical contexts. Two of these studies, with pneumonia10
and respiratory failure11
as outcomes, were performed in a population of American veterans; as 90% of the patients were men, the generalizability of the findings may be limited. Six others used retrospective data sets to develop a score to predict single PPC outcomes: unplanned reintubation,12–14
postoperative pulmonary failure,15
and adult respiratory distress syndrome.16
were prospective studies in patients undergoing a wide range of surgeries and only one was internally validated.1
To our knowledge, none of these studies have been replicated in other settings to externally validate the scores in new prospectively collected samples of patients, and for this reason, none can be confidently generalized.18
The lack of validated models affects the clinician’s ability to predict and plan strategies to prevent PPCs in high-risk groups.
The recent Assess Respiratory Risk in Surgical Patients in Catalonia (ARISCAT) study1
addressed the problem of differences across surgical contexts by using a population-based approach that was representative of a wide range of procedures and patients in a geographically defined, mixed urban-rural practice setting. A clinically practical seven-factor scoring system to assess the risk of a composite PPC—the likelihood of developing any complication in a list of well-defined events—was internally validated.
To test the hypothesis of the geographic transportability of the ARISCAT score to different but plausibly related19
surgical populations, 63 European centers in 21 countries prospectively recruited a new patient cohort for the Prospective Evaluation of a RIsk Score for postoperative pulmonary COmPlications in Europe (PERISCOPE) study. The aim of this study was to measure the accuracy of ARISCAT score predictions of PPCs in the PERISCOPE cohort overall and in three subsamples of that cohort. Cohort splitting was intended to reflect possible case-mix differences that might appear with increased geographic distance from the setting where the ARISCAT model was developed.
Materials and Methods
Study Design and Participants
The PERISCOPE cohort was established following a prospective, observational multicenter design in which 63 European hospitals (appendix) volunteered to recruit surgical patients during continuous 7-day periods. Recruitment within a center started on a date between May 2, 2011, and August 15, 2011, chosen on the basis of the local researchers’ convenience, and follow-up ended in November 2011. The hospitals were identified through membership in the European Society of Anaesthesiology and approached directly by national study coordinators. The study was registered at www.clinicalTrials.gov
Consecutive patients undergoing a nonobstetric in-hospital elective or emergency surgical procedure under general, neuraxial, or plexus block anesthesia were recruited. Exclusion criteria were age under 18 yr, obstetric procedures or any intervention during pregnancy, procedures in which only local or peripheral nerve anesthesia would be used, procedures outside an operating theater, procedures related to a previous postoperative complication, transplantation, patients with preoperatively intubated trachea, and outpatient procedures (hospital stay of <24 h).
Three numerically comparable subsamples were defined, based on their geographic distance from the development population as follows: Spain, Western Europe (WE), and Eastern Europe (EE).
Ethics requirements differed in the 21 countries, but formal approval from a research ethics review board was applied for and given in each. The locally responsible investigator also applied for and obtained approval from the ethics committee of each participating hospital. Each center investigator sent a scanned copy of the ethics committee approval to the European Society of Anaesthesiology secretariat, where files were centralized. Written informed consent to use the data was obtained from each enrolled patient in all centers.
Organization, Data Collection, Variables of Interest, and Quality Assurance
The international research team consisted of a steering committee and nationally and locally responsible investigators on behalf of the European Society of Anaesthesiology.
Postoperative pulmonary complications were recorded by the investigators throughout the postoperative hospital stay up to a maximum of 5 weeks. Patients with PPCs were identified by data collectors who consulted medical records in real time, daily while they were being created, to find events that fulfilled any PPC definition; they did not modify a center’s customary management of patients. A structured paper questionnaire was filled in for each patient; later, information that could identify the patients was removed before transfer to secure online record forms (OpenClinic Optimized Cloud Hosting, Boston, MA), an electronic system with quality control algorithms to validate data entry and identify missing data. A central data manager checked entries to confirm completeness of records and asked the designated local contact person to provide additional information if necessary.
Data for the seven risk factors described in the ARISCAT model (table 1
) were collected preoperatively by the anesthesiologist in charge of the patient after signed informed consent had been given, as follows: age in years; peripheral oxyhemoglobin saturation measured by pulse oximetry (SpO2
) breathing air in supine position after resting 1 min or, in patients on oxygen, SpO2
after 10 min without oxygen; respiratory infection in the last month; hemoglobin concentration; surgical incision site; surgical duration in hours; type of surgery (scheduled or emergency).
Postoperative pulmonary complication was defined as the occurrence of at least one event on a list of in-hospital fatal or nonfatal PPCs (table 2
). Thus, a patient was considered to have had a PPC when at least one of these events was recorded. This outcome was therefore considered as a binary categorical variable (yes/no) for the purposes of statistical analysis.
To compare subsamples, we also recorded administrative data (dates of surgery and discharge and vital status at discharge), general information (sex, height, and weight), preoperative variables (chronic pulmonary disease, smoking status, hypertension, cerebrovascular disease, coronary artery disease, chronic heart failure, liver disease, chronic kidney disease, and physical status using the American Society of Anesthesiologists’ classification), and intraoperative variables (anesthetic technique and surgical specialty). Postoperative hospital length of stay and in-hospital postoperative mortality were followed up to a maximum of 90 days.
The sample size was calculated considering that at least 100 PPCs were needed in each of the three external validation subsamples.21
The incidence of each PPC was calculated in the PERISCOPE cohort (and its three subsamples: Spain, WE, and EE), and the associations between number of PPCs per patient and both length of hospital stay and in-hospital mortality were assessed. Comparative analysis of demographic and clinical characteristics between the ARISCAT development sample versus
the whole PERISCOPE cohort and each subsample was performed.
In the ARISCAT regression model (risk of PPC = 1/(1 + e−linear predictor
), the linear predictor (lpARISCAT) was built using β coefficients derived from the original regression model. A risk score was also calculated for each patient by assigning points derived by multiplying the regression coefficients by 10, rounding to the nearest integer and adding the integers (table 1). Three predicted risk groups were then defined according to the cutoffs identified in the ARISCAT study1
by means of the minimum description length principle: <26 (low), ≥26 and <45 (intermediate), and ≥45 (high risk).
The model’s performance was then assessed by studying discrimination, calibration, and clinical usefulness in the PERISCOPE data set overall and the three subsamples (Spain, WE, and EE). For measures of discrimination and diagnostic accuracy, the ARISCAT clinical score was used, whereas for calibration the linear predictor equation was used because it gives a more accurate mathematical assessment of each patient’s outcome risk.
Accurate predictions discriminate between patients with the outcome and those without. The ability of the ARISCAT score to rank patients with and without at least one PPC was quantified with the c-statistic, which is the equivalent of the area under receiver operating characteristic curve for a dichotomous outcome variable.
Calibration refers to the agreement between observed outcomes and predictions. It can be broken down into two components: a constant (a
) and a coefficient (b
), which represent, respectively, the intercept and the calibration slope of a line plotting observed frequencies against predictions.22
These two components can be calculated by logistic regression with the linear predictor as the only risk factor for the outcome; this regression defines a new linear predictor: lpCALIBRATED = a
The calibration slope (b
) also reflects the average effect of predictors in the outcome; the adjusted value of the intercept (a
) after setting the value of b
to 1 (assuming hypothetical optimal performance of predictors) reveals a systematic deviation (bias) of predictions. Following these statistical premises, under the hypothesis that the ARISCAT linear predictor (lpARISCAT) was the only valid PPC predictor, calibration was verified by performing logistic regression in the complete PERISCOPE cohort and in each geographic subsample, with lpARISCAT as the only independent variable and the observed PPC composite outcome as the dependent variable. From each of these regressions, we obtained a new linear predictor (lpCALIBRATED) as lpCALIBRATED = a
× lpARISCAT. The PPC risk in each PERISCOPE subsample was then expressed as follows (using a different lpCALIBRATED for each): Risk of PPC = 1/(1 + e−lp
CALIBRATED). Finally, to reflect the clinical implications of calibration, we also calculated the predicted and observed PPC frequencies in each of the PERISCOPE data sets according to the cutoffs for three levels of risk.
The utility of a predictive model can be assessed by means of measures of accuracy of outcome diagnosis (sensitivity, specificity, positive and negative likelihood ratios, and positive and negative predictive values). These measures of PPC diagnostic accuracy were analyzed for intermediate and high-risk scores compared to lower risk scores, respectively.
To adjust the ARISCAT score for the influence of European regional influence, we performed a logistic regression with PPC occurrence as the dependent variable and the ARISCAT score (three levels of risk) and geographic area (Spain, WE, and EE) as independent variables. We also carried out a supplementary exploration of the performance of the ARISCAT model’s ability to predict single components of the composite and alternative composite outcomes in the PERISCOPE sample by calculating adjusted odds ratios for each predictor and c-statistics for the each individual PPC outcome and alternative composites.
The Mann–Whitney U test was used to compare means and the chi-square test or Fisher exact test to compare percentages. The Kruskal–Wallis test was used to compare postoperative length of stay between subgroups formed according to the number of PPCs found. The Mantel–Haenszel test was used to analyze trends in mortality rates between those subgroups. Statistical analyses were performed using the SPSS Software package (IBM SPSS Statistics 19.0, Armonk, NY). Categorical variables were expressed as number of cases, and percentage and continuous variables were expressed as the median and interquartile range. All performance measures were expressed with 95% CIs.
A total of 5,859 surgical patients were recruited by the participating hospitals (fig. 1
); 475 (8.1%) were lost because of recruitment or protocol violations or missing follow-up data, and 285 (4.9%) were lost due to missing data in candidate risk factors (see table 1, Supplemental Digital Content 1, http://links.lww.com/ALN/B55
). The most important missing variables lost were SpO2
, preoperative anemia (<10 g/dl), and respiratory infection in the last month (see table 2, Supplemental Digital Content 1, http://links.lww.com/ALN/B55
). In the PERISCOPE cohort overall, 725 PPCs were recorded in 404 patients (7.9% of the 5,099 patients studied). Respiratory failure was the most frequent complication (241 patients, 4.7%), followed by pleural effusion (159, 3.1%), atelectasis (122, 2.4%), pulmonary infection (120, 2.4%), bronchospasm (42, 0.8%), pneumothorax (29, 0.6%), and aspiration pneumonitis (12, 0.2%). Among patients with PPCs, 263 (65%) had more than one complication and 141 (35%) had three or more. The time between surgery and the first PPC recorded was 3 (2 to 6) days. In-hospital mortality in the group of patients with at least one PPC (8.3%) was significantly higher than in patients with no PPC (0.2%; P
Comparison of demographic and clinical characteristics, PPC incidence, length of hospital stay, and mortality between the ARISCAT development cohort (1,627 patients) and the overall PERISCOPE cohort and subsamples are shown in table 3
. The PPC incidence was higher in the overall PERISCOPE cohort and subsamples than in the ARISCAT sample, but the in-hospital PPC-associated mortality rate and postoperative length of stay were similar. Cardiac and cerebrovascular comorbidities were higher in the PERISCOPE cohort than in the ARISCAT development sample. The relationships between length of stay and in-hospital mortality and number of PPCs are shown in table 4
The performance measures describing discrimination, calibration, and clinical usefulness in each of the PERISCOPE samples in which the ARISCAT model was tested are shown in table 5
. Predicted probabilities and observed PPC frequencies in each of the PERISCOPE data sets are shown in table 6
, according to the ARISCAT score cutoffs for three levels of risk.
The adjusted odds ratios for predictors in the ARISCAT score and the c
-statistics of the ARISCAT model for each component of the composite outcome with over 100 events in the overall PERISCOPE cohort (respiratory failure, suspected pulmonary infection, pleural effusion, and atelectasis) and for possible combinations of two, three, or four PPC outcomes are shown in supplemental tables (see Supplemental Digital Content 2, http://links.lww.com/ALN/B56
). This supplemental analysis suggested that the variables in the model with the highest odds ratios might also be good predictors in refitted predictive models for any component of the PPC composite outcome. The adjustment of the ARISCAT score for interaction with geographic area is also shown (see table 1, Supplemental Digital Content 3, http://links.lww.com/ALN/B57
). This analysis confirmed first, that EE region was an independent risk factor for the outcome and second, that there was an interactive positive effect between WE region and the ARISCAT score’s prediction of risk for the outcome.
Before prognostic scores are adopted for clinical use outside the development setting, they should be studied in new populations19
; yet to our knowledge, this is the first study in which a PPC risk score has been validated externally.
Our study of the performance of the seven-factor ARISCAT score (table 1) in patient samples that were progressively distant from the development setting provides an intermediate level of evidence according to the definition of Justice et al.
supporting the use of this model for PPC risk prediction in a broad European surgical population in which the observed incidence of PPCs fell within the ranges reported for similar settings.4
The results in each external sample additionally illustrate the degree to which external validation has potential clinical significance, as it suggests that even a validated score may need further adjustments for populations with characteristics that diverge from those previously studied.23
The ARISCAT score, which had shown very good ability to discriminate PPC risk in the development sample (c
also showed good discrimination in the overall PERISCOPE sample (c
-statistic, 0.80). Discrimination was even better or equally good in the WE and Spain samples. The c
-statistic in the EE subsample (0.76) was only moderately good.
While discrimination values check that the model has good ability to stratify patients according to risk, calibration provides information on the degree to which observed frequencies of outcomes deviate from predicted in a particular population. The calibration slope b
was significantly lower than the ideal (b
= 1) in all PERISCOPE subsamples studied (table 5). Although this finding is potentially attributable to the optimism inherent in nearly every model,29
it may also be a consequence of a real difference in the effects of predictors in the new populations. In the WE subsample, where the score performed best, the calibration slope was over 0.8, while in the EE subsample, the slope was under 0.6, suggesting that the coefficients of the ARISCAT predictors might require recalibration in a population represented by this subsample. Intercept values significantly different from 0 in the EE and the WE subsamples in this analysis suggest that factors other than the ARISCAT score’s predictors probably play a role. Specifically, differences in case mix between the PERISCOPE and development samples (table 3)—such as different rates of underlying cardiovascular disease or different distributions of surgical procedures and anesthesia techniques—could explain the significant differences in the PPC incidences, as well as differences in the intercepts between PERISCOPE subsamples. This hypothesis seems strong based on our supplemental analysis (table 1, Supplemental Digital Content 3, http://links.lww.com/ALN/B57
) to explore interaction between the ARISCAT score’s prediction of PPC risk and geographic area. Significant risk in the EE region is evident even after adjustment, but the odds ratio implies that there are additional unknown risk factors in this area. The analysis also revealed a positive interaction between the score’s prediction of risk and WE geographic zone; put another way, the ARISCAT score performed best in this region, even better than in the Spanish cohort. It has been noted that external validation studies should be undertaken in patient samples that are different but plausibly related to the development sample,17
but the scientific community has unfortunately not come to agreement on a definition for this combination of difference and relation in case mix. In other words, the number, or proportion, of candidate predictors not finally included in the model that should be similar in an external sample for it to serve for validation purposes has not been made explicit.
Likelihood ratios (table 5) are the measures that best summarize the usefulness of a prognostic test. Thus, the risk for PPC of a patient with a score of 45 or more (high) is 5 to 11 times higher than the risk of a patient with a lower score; a patient with a score below 26 has a level of risk from two to four times lower than others with higher scores. These results allow us to define the score as a tool with moderate to good clinical utility to estimate the risk of complications. It should be noted that apparently modest predictive values that would not be acceptable in diagnostic tests, where accuracy is essential, may still be very helpful in prognostic models, which are used in preoperative visits to predict a complication risk higher than average.
This first study to externally validate a PPC risk score has several strengths. First, we followed a prospective design calling for careful data collection in an appropriately large sample of patients from a wide spectrum of European countries and surgical settings. Second, by dividing the PERISCOPE cohort into three subsamples that were progressively distant from the development setting, we were able to illustrate the degree to which a model might behave somewhat differently in each one, a reminder of the importance of external validation generally speaking and of the demanding calibration step in particular. This calibration differentiates between miscalibration attributable to potential predictors not included in the model (when the intercept a differs significantly from 0) and miscalibration attributable to different weights of the predictors (when the slope b differs significantly from 1).
Concerning limitations, we are aware of the low representativeness of the samples with respect to the geographic areas in which they were obtained. This limitation prevents us from extrapolating definite conclusions as to the extent to which recalibration might be required in each of the areas we studied. Such recalibration might be useful, with a view to optimizing the usefulness of the model in particular settings. A second possible limitation is that some PPCs might not have been detected in patients discharged early, but this hypothetical bias is inherent to every study with in-hospital follow-up. We think it would have had only a minor effect in this study, given that the candidates for early discharge would have been patients undergoing less invasive procedures and those showing a more favorable course. Finally, a composite outcome, of the type the ARISCAT score predicts, might be considered a controversial choice. However, such composites mimic clinicians’ weighting of several risk factors at once or in series; thus, we think this approach reflects the real conditions of clinical practice during the period when decisions affecting perioperative management are taken. Our confirmation that the length of hospital stay and in-hospital mortality increase as PPCs rise in number in the external samples (table 4) underlines the importance of a patient’s development of any single respiratory event included in the composite. Moreover, the results of the exploratory analysis of the ARISCAT score’s prediction of single components of the composite outcome (Supplemental Digital Content 3, http://links.lww.com/ALN/B57
) suggested that the most powerful predictors in the ARISCAT model would also be good predictors in a refitted predictive model for any component of the composite.
We conclude that assessing the seven easily recordable and clinically accessible factors identified by the ARISCAT score (age, preoperative SpO2 in air, respiratory infection in the last month, preoperative anemia, upper abdominal or intrathoracic surgical incision, duration of surgery, and emergency procedure) is useful for differentiating three levels of PPC risk in hospitals outside the development setting, although performance differs significantly between geographic areas. That ability to distinguish risk makes the score a validated starting point for controlled trials and audits of risk-reduction strategies. Even so, we advise clinicians to use this scale cautiously when predicting risk for an individual patient, given that the calibration of the model is suboptimal in some geographic contexts. Recalibration can optimize performance of the score for use in homogeneous populations with well-defined characteristics.
The authors thank all collaborators for their recruitment of patients (listed in appendix). The authors thank Brigitte Leva (European Society of Anaesthesiology, Brussels, Belgium) for her assistance as data manager, and Mary Ellen Kerans, M.A., for article editing and advice on the English usage in some versions of the article (European Society of Anaesthesiology funding).
The Prospective Evaluation of a RIsk Score for postoperative pulmonary COmPlications in Europe (PERISCOPE) study was funded and supported by the European Society of Anaesthesiology (Brussels, Belgium) through the Clinical Trials Network. This study also received support provided by Grant 041610-2003 from Fundació La Marató de Televisió de Catalunya, Barcelona, Spain.
The authors declare no competing interests.
1. Canet J, Gallart L, Gomar C, Paluzie G, Vallès J, Castillo J, Sabaté S, Mazo V, Briones Z, Sanchis JARISCAT Group. . Prediction of postoperative pulmonary complications in a population-based surgical cohort. ANESTHESIOLOGY. 2010;113:1338–50
2. Smetana GW, Lawrence VA, Cornell JEAmerican College of Physicians. . Preoperative pulmonary risk stratification for noncardiothoracic surgery: Systematic review for the American College of Physicians. Ann Intern Med. 2006;144:581–95
3. McAlister FA, Bertsch K, Man J, Bradley J, Jacka M. Incidence of and risk factors for pulmonary complications after nonthoracic surgery. Am J Respir Crit Care Med. 2005;171:514–7
4. Shander A, Fleisher LA, Barie PS, Bigatello LM, Sladen RN, Watson CB. Clinical and economic burden of postoperative pulmonary complications: Patient safety summit on definition, risk-reducing interventions, and preventive strategies. Crit Care Med. 2011;39:2163–72
5. Khan NA, Quan H, Bugar JM, Lemaire JB, Brant R, Ghali WA. Association of postoperative complications with hospital costs and length of stay in a tertiary care center. J Gen Intern Med. 2006;21:177–80
6. Khuri SF, Henderson WG, DePalma RG, Mosca C, Healey NA, Kumbhani DJParticipants in the VA National Surgical Quality Improvement Program. . Determinants of long-term survival after major surgery and the adverse effect of postoperative complications. Ann Surg. 2005;242:326–41; discussion 341
7. Linde-Zwirble WL, Bloom JD, Mecca RS, Hansell DM. Postoperative pulmonary complications in adult elective surgery patients in the US: Severity, outcomes, and resource use. Crit Care. 2010;14(suppl 1):P210
8. Canet J, Gallart L. Predicting postoperative pulmonary complications in the general population. Curr Opin Anaesthesiol. 2013;26:107–15
9. Fisher BW, Majumdar SR, McAlister FA. Predicting pulmonary complications after nonthoracic surgery: A systematic review of blinded studies. Am J Med. 2002;112:219–25
10. Arozullah AM, Khuri SF, Henderson WG, Daley JParticipants in the National Veterans Affairs Surgical Quality Improvement Program. . Development and validation of a multifactorial risk index for predicting postoperative pneumonia after major noncardiac surgery. Ann Intern Med. 2001;135:847–57
11. Arozullah AM, Daley J, Henderson WG, Khuri SF. Multifactorial risk index for predicting postoperative respiratory failure in men after major noncardiac surgery. The National Veterans Administration Surgical Quality Improvement Program. Ann Surg. 2000;232:242–53
12. Ramachandran SK, Nafiu OO, Ghaferi A, Tremper KK, Shanks A, Kheterpal S. Independent predictors and outcomes of unanticipated early postoperative tracheal intubation after nonemergent, noncardiac surgery. ANESTHESIOLOGY. 2011;115:44–53
13. Hua M, Brady JE, Li G. A scoring system to predict unplanned intubation in patients having undergone major surgical procedures. Anesth Analg. 2012;115:88–94
14. Brueckmann B, Villa-Uribe JL, Bateman BT, Grosse-Sundrup M, Hess DR, Schlett CL, Eikermann M. Development and validation of a score for prediction of postoperative respiratory complications. ANESTHESIOLOGY. 2013;118:1276–85
15. Gupta H, Gupta PK, Fang X, Miller WJ, Cemaj S, Forse RA, Morrow LE. Development and validation of a risk calculator predicting postoperative respiratory failure. Chest. 2011;140:1207–15
16. Kor DJ, Warner DO, Alsara A, Fernández-Pérez ER, Malinchoc M, Kashyap R, Li G, Gajic O. Derivation and diagnostic accuracy of the surgical lung injury prediction model. ANESTHESIOLOGY. 2011;115:117–28
17. Blum JM, Stentz MJ, Dechert R, Jewell E, Engoren M, Rosenberg AL, Park PK. Preoperative and intraoperative predictors of postoperative acute respiratory distress syndrome in a general surgical population. ANESTHESIOLOGY. 2013;118:19–29
18. Moons KG, Kengne AP, Grobbee DE, Royston P, Vergouwe Y, Altman DG, Woodward M. Risk prediction models: II. External validation, model updating, and impact assessment. Heart. 2012;98:691–8
19. Justice AC, Covinsky KE, Berlin JA. Assessing the generalizability of prognostic information. Ann Intern Med. 1999;130:515–24
20. Altman DG, Royston P. What do we mean by validating a prognostic model? Stat Med. 2000;19:453–73
21. Vergouwe Y, Steyerberg EW, Eijkemans MJ, Habbema JD. Substantial effective sample sizes were required for external validation studies of predictive logistic regression models. J Clin Epidemiol. 2005;58:475–83
22. Steyerberg EW Clinical Prediction Models: A Practical Approach to Development, Validation, and Updating. 2009 New York Springer Publishing Company
23. Vergouwe Y, Moons KG, Steyerberg EW. External validity of risk models: Use of benchmark values to disentangle a case-mix effect from incorrect coefficients. Am J Epidemiol. 2010;172:971–80
24. Steyerberg EW, Borsboom GJ, van Houwelingen HC, Eijkemans MJ, Habbema JD. Validation and updating of predictive logistic regression models: A study on sample size and shrinkage. Stat Med. 2004;23:2567–86
25. Moons KG, Altman DG, Vergouwe Y, Royston P. Prognosis and prognostic research: Application and impact of prognostic models in clinical practice. BMJ. 2009;338:b606
26. Altman DG, Vergouwe Y, Royston P, Moons KG. Prognosis and prognostic research: Validating a prognostic model. BMJ. 2009;338:b605
27. Toll DB, Janssen KJ, Vergouwe Y, Moons KG. Validation, updating and impact of clinical prediction rules: A review. J Clin Epidemiol. 2008;61:1085–94
28. Janssen KJ, Moons KG, Kalkman CJ, Grobbee DE, Vergouwe Y. Updating methods improved the performance of a clinical prediction model in new patients. J Clin Epidemiol. 2008;61:76–86
29. Harrell FE Jr Regression Modeling Strategies: With Applications to Linear Models, Logistic Regression, and Survival Analysis. 2001 New York Springer-Verlag New York
Prospective Evaluation of a RIsk Score for postoperative pulmonary COmPlications in Europe —List of Participating Centers and Collaborators (*Site Lead Investigators)
Jaume Canet, Hospital Universitari Germans Trias i Pujol, Barcelona, Spain
Steering Committee Members
Jaume Canet, Spain
Sergi Sabaté, Spain
Olivier Langeron, France
Marcelo Gama de Abreu, Germany
Lluís Gallart, Spain
F. Javier Belda, Spain
Paolo Pelosi, Italy
Andreas Hoeft, Germany
Valentin Mazo, Spain
Brigitte Leva, European Society of Anaesthesiology, Brussels, Belgium
1. University Hospital centre Mother Theresa (Tirana), Jonela Burimi, M.D., Toma Halefi, M.D., Aleksander Hoxha, M.D.,* Kliti Pilika, M.D., Imelda Selmani, M.D.
1. Cliniques Universitaires Saint Luc A.S.B.L. Université Catholique de Louvain (Brussels), Véronique Daout, M.D., Caroline Gauthier, M.D., David Kahn, M.D., Mona Momeni, M.D.,* Christine Watremez, M.D.
Bosnia and Herzegovina
1. Clinical Centre University Sarajevo Heart Center (Sarajevo), Slavenka Straus, M.D.*
2. General Hospital Prim.dr Abdulah Nakas(Sarajevo), Dejana Djonovic-Manovic, M.D., Marina Juros-Zovko, M.D.*
1. University Hospital Rijeka (Rijka), Helga Komen-Ušljebrka, M.D.,* Vlasta Orlić, M.D., Ivana Stuck, M.D.
1. Faculty Hospital Brno (Brno), Lenka Baláková, M.D., Martina Kosinová, M.D., Ivo Křikava, M.D., Roman Štoudek, M.D., Petr Štourač, M.D.,* Katarina Zadražilová, M.D.
2. Masaryks Hospital Usti nad labem (Usti Nad Labem), Sanober Janvekar, M.D.*
1. Tartu University Hospital (Tartu), Juri Karjagin, M.D., Kadri Rõivassepp, M.D., Alar Sõrmus, M.D.*
1. Hôpital Pitié-Salpêtrière (Paris), Philippe Cuvillon, M.D., Cristina Ibáñez-Esteve, M.D., Olivier Langeron, M.D.,* Mathieu Raux, M.D., Armelle Nicolas-Robin, M.D.
1. Klinikum Darmstadt GmbH (Darmstadt), André Winter, M.D.*
2. Medical Center of the Johannes Gutenberg University Mainz (Mainz), Malte Brunier, M.D., Kristin Engelhard, M.D., Rita Laufenberg Feldmann, M.D.,* Raphaele Lindemann, M.D., Susanne Mauff, M.D., Anne Sebastiani, M.D., Camila Zamperoni, M.D.
3. University Hospital Bonn (Bonn), Andreas Hoeft, M.D.,* Florian Kessler, M.D., Maria Wittmann, M.D.
4. University Hospital Carl Gustav Carus, Dresden University of Technology (Dresden), Thomas Bluth, M.D., Marcelo Gama de Abreu, M.D.,* Andreas Güldner, M.D., Thomas Kiss, M.D.
1. MISEK Kft. (Miskolc), Kristina Bráz, M.D., Csilla Ruszkai, M.D.*
1. Azienda Ospedaliera (Padova), Massimo Micaglio, M.D., Carlo Ori, M.D., Matteo Parotto, M.D.,* Paolo Persona, M.D.
2. Azienda Ospedaliera S. Croce e Carle (Cuneo), Coletta Giuseppe, M.D.*
3. Azienda USL n. 5 di Pisa Ospedale F. Lotti (Pontedera), Paolo Carnesecchi, M.D., Denise Lazzeroni, M.D., Irene Lorenzi, M.D.*
4. European Institute of Oncology (Milano), Gianluca Castellani, M.D., Daniele Sances, M.D.,* Gianluca Spano, M.D., Stefano Tredici, M.D., Dario Vezzoli, M.D.
5. Ospedale San Martino (Genova), Iole Brunetti, M.D., Anna Di Noto, M.D., Angelo Gratarola, M.D., Alexandre Molin, M.D.,* Luca Montagnani, M.D., Giulia Pellerano, M.D., Paolo Pelosi, M.D.
6. Ospedale Sant'Orsola, Malpighi (Bologna), Maurizio Fusari, M.D.*
7. University of Insubria (Varese), Laura Camici, M.D., Luca Guzzetti, M.D., Fabio Marangoni, M.D., Paolo Severgnini, M.D.*
8. University of Milano, Ospedale San Paolo (Milano), Piero Di Mauro, M.D., Francesca Rapido, M.D., Concezione Tommasino, M.D.*
1. Pauls Stradins Clinical University Hospital (Riga), Ieva Nemme, M.D., Janis Nemme, M.D.*
1. Kaunas Medical University Hospital (Kaunas), Justinas Blieka, M.D., Jurgita Borodičienė, M.D., Brigita Budrytė, M.D., Aurika Karbonskiene, M.D.,* Inga Kiudulaitė, M.D., Eglė Milieškaitė, M.D., Renata Rasimavičiūtė, M.D., Ugnė Sirevičienė, M.D., Ramunė Stašaitytė, M.D., Edgaras Ūsas, M.D., Giedrė Žarskienė, M.D.
2. Vilnius University Hospital Santariskiu Clinics (Vilnius), Egle Kontrimaviciute, M.D., Jurate Sipylaite, M.D.,* Gabija Tomkutė, M.D.
1. ZithaKlinik (Luxembourg), Petra Bardea, M.D., Marco Klop, M.D., Marc Koch, M.D.*
1. 10 Wojskowy Szpital Kliniczny z Polikliniką w Bydgoszczy (Bydgoszcz) Dominika Bożiłow, M.D., Robert Goch, M.D.*
1. Hospitais da Universidade de Coimbra, EPE (Coimbra), João Bonifácio, M.D., Sofia Marques, M.D., Tânia Teresa dos Santos Ralha, M.D.*
2. Centro Hospitalar de Lisboa Ocidental (Lisbon), Daniel Alves, M.D., Inês Carvalho, M.D., Josefina Suzana Da Cruz Parente, M.D.,* Sara Tomé, M.D.
3. Hospital Fernando Fonseca (Lisbon), Cristina Carmona, M.D.*
4. Instituto Português de Oncologia Do Porto (Porto), Miranda Costa, M.D.,* Maria Lina, M.D., Sofia Sierra, M.D.
1. Emergency Clinical Hospital of Constanta (Constanta), Alina Balcan, M.D., Iulia Cindea, M.D., Viorel Ionel Gherghina, M.D.,* Catalin Grasa, M.D.
2. Emergency County Hospital Clinic of Anesthesia and Intensive Care (Târgu Mures), Ruxandra Copotoiu, M.D., Sanda-Maria Copotoiu, M.D.,* Judit Kovacs, M.D.,* Janos Szederjesi, M.D., Arthur Theil, M.D.
3. Emergency Institute of Cardiovascular Diseases Prof. Dr. C. C. Iliescu (Bucharest), Daniela Filipescu, M.D.*
1. Krasnoyarsk State Medical University (Krasnoyarsk), Alexey Grytsan, M.D.,* Tatiana Kapkan, M.D., Sergey Rostovtsev, M.D., Anastasia Yushkova, M.D.
1. Clinica Universidad de Navarra (Pamplona), Ricardo Calderón, M.D., Elena Cacho, M.D., Carolina Marginet, M.D., Pablo Monedero, M.D.,* Maria José Yepes, M.D.
2. Consorcio Hospital General Universitario de Valencia (Valencia), Jose Miguel Esparza Miñana, M.D., Manuel Granell Gil, M.D.,* Gabriel Rico Portolés, M.D.
3. Corporació Sanitària Parc Taulí (Barcelona Sabadell), Alberto Lisi, M.D.,* Gisela Perez, M.D., Nuria Poch, M.D.
4. Fundacio Althaia (Manresa); Mauricio Roberto Argañaraz Quinteros, M.D., Carme Font Bosch, M.D., Jordi Torrellardona Llobera, M.D.*
5. Fundació Puigvert (Barcelona) Sergi Sabaté, M.D.,* Pilar Sierra, M.D.
6. Hospital Arnau de Vilanova (Lleida), Mercedes Matute, M.D.*
7. Hospital Clinic de Barcelona (Barcelona) Amalia Alcon Dominguez, M.D.,* María José Arguis, M.D., Isabel Belda, M.D., Enrique Carrero, M.D., Jacobo Moreno, M.D., Irene Rovira, M.D., Marta Ubre, M.D., Roberto Castillo, M.D., Sílvia Herrero, M.D.
8. Hospital Clínic Universitari de Valéncia (Valencia) Maria Teresa Ballester Luján, M.D.,* F. Javier Belda, M.D., Jose Carbonell, M.D., Geri Gencheva, M.D., Andrea Gutierrez, M.D., Julio Llorens, M.D., Sofia Machado, M.D.
9. Hospital de Denia (Denia), Francisca Llobell, M.D.,* Daniel Paz Martin, M.D.
10. Hospital del Tajo Aranjuez (Madrid), Francisco Javier García-Miguel, M.D.*
11. Hospital General de La Palma Breña Alta (La Palma, Canarias), Aníbal Pérez García, M.D.*
12. Hospital General Universitario Alicante (Alicante), Roque Company, M.D.,* Aixa Ahamdanech Idrissi, M.D., Josefina del Fresno Cañaveras, M.D., Jose Alejandro Navarro Martinez, M.D., Estefania Paya Martinez, M.D., Ester Sanchez Garcia, M.D.
13. Hospital San Jorge (Huesca), Jorge Vera Bella, M.D.*
14. Hospital Sant Pau (Barcelona), Inmaculada India Aldana, M.D., J. Manuel Campos, M.D., Xavier Pelaez Vaamonde, M.D.*
15. Hospital Santa Maria (Lleida), Montserrat Torra, M.D.*
16. Hospital Universitari del Mar 'Parc de Salut Mar (Barcelona), Raquel Arroyo, M.D., Juan Carlos Cabrera, M.D., Jesús Carazo Cordobes, M.D.,* Lluís Gallart, M.D., Amelia Rojo, M.D., Francisco Javier Santiveri, M.D.
17. Hospital Universitari Germans Trias i Pujol (Badalona), Jaume Canet, M.D.,* Miriam González, M.D., Anabel Jiménez, M.D., Yolanda Jiménez, M.D., Agnès Martí, M.D., Valentin Mazo, M.D., Enrique Moret, M.D., Monica Rodriguez Nuñez, M.D.,* Joaquin Velasco, M.D.
18. Hospital Universitario 12 de Octubre (Madrid), Adriana Calderón, M.D., Matide González, M.D., Olga González, M.D., Ana Hermira Anchuelo, M.D.,* Eloisa López, M.D., Esther Sánchez, M.D.
19. Hospital Universitario de La Princesa (Móstoles-Madrid), Blanca Aznárez Zango, M.D.,* Francisco José García Corral, M.D., Esperanza Mata Mena, M.D., Antonio Planas Roca, M.D.
20. Hospital Universitario de Móstoles (Madrid), Raquel Fernández Rocío Ayala Soto, M.D.,* Borja Quintana, M.D.
21. Hospital Universitario Marques De Valdecilla (Santander), José Manuel Rabanal Llevot, M.D.,* Mónica Mercedes, M.D., Williams Camus, M.D., Alba Palacios Blanco, M.D., Angela Largo Ruiz, M.D.
22. Hospital Universitario Rio Hortega (Valladolid), Jesus Rico Feijoo, M.D.*
23. Hospital Universitario Virgen del Rocio (Sevilla), Elvira Castellano Garijo, M.D.*
24. Hospital Son Llatzer (Palma de Mallorca), Julio Belmonte Cuenca, M.D.,* Marcos José Bonet Binimelis, M.D., Ivaylo Grigorov, M.D., Josep Lluis Aguilar, M.D.
25. Vall d’Hebron University Hospital (Barcelona), Míriam De Nadal Clanchet, M.D., Encarnación Guerrero Viñas, M.D., Susana Manrique Muñiz, M.D., Víctor Martín Mora, M.D., Francisca Munar Bauzà, M.D., Sonia Núñez Aguado, M.D., Montserrat Olivé Vidal, M.D.,* María Luisa Paños Gozalo, M.D., Marcos Sánchez Marín, M.D., María Carmen Suescun López, M.D.
1. Ospedale Regionale di Lugano (Lugano), Paolo Maino, M.D.*
1. Medical Faculty of Istanbul, Istanbul University (Istanbul), Semra Kucukgoncu, M.D., Nuzhet Mert Sentürk, M.D.,* Zerrin Sungur Ulke, M.D.
1. St. Katherine Hospital of Cardiology (Odessa), Yevhen Eugene Yevstratov, M.D.*
Supplemental Digital Content
© 2014 American Society of Anesthesiologists, Inc.
Publication of an advertisement in Anesthesiology Online does not constitute endorsement by the American Society of Anesthesiologists, Inc. or Lippincott Williams & Wilkins, Inc. of the product or service being advertised.