There is increasing interest in using administrative-claims data for surveillance of surgical site infections but the performance of claims-based models for case-mix adjustment has not been well studied. In a large cohort of patients who underwent total joint replacement at one tertiary care institution, we examined the predictive performance of claims data-based comorbidities as a case-mix adjustor for surgical site infections and the incremental value of adding clinical risk factors. Our findings suggest that the claims-based risk-adjustment models for surgical site infections are well calibrated but lack predictive discrimination. Discrimination can be improved by the addition of clinical risk factors.
Our study results should be interpreted in light of some potential limitations. First, our study was confined to procedures performed at one hospital, which can restrict power and generalizability of findings to other institutions. However, this is also a strength of the study because confounding by unknown and known healthcare delivery factors, such as infection-prevention practices, is minimized. Ideally, future multicenter studies will take into account hospital-level healthcare delivery factors when assessing the performance of risk prediction models. Second, our method for ascertainment of comorbidities and surgical site infections outcomes is different than those of previous analyses. Similar to other investigators [5-7, 9, 16], we relied on electronically available administrative data to ascertain comorbidities but our time window was limited to the index hospitalization. This is in contrast to a Medicare study which ascertained comorbidities from the 1-year period before admission . Therefore, we may have under-ascertained some comorbidities. Despite this, a qualitative comparison of the prevalence of comorbidities revealed minimal differences. In addition, the types of comorbidities we included are different from other claims-based analyses [5-7, 16]. For example, the Centers for Medicare & Medicaid Services complication measure  grouped comorbidities using 29 condition categories, whereas the analyses by Bozic et al. [5-7] were based on the Elixhauser method. Other important differences in our study include the exclusion criteria, outcome definitions, and the time for ascertainment of surgical site infections. Despite these differences in methods, unique strengths of our study are complete ascertainment and chart review-based validation of all surgical site infection outcomes, which minimized measurement bias. Owing to active followup of patients through the institutional total joint registry, we were able to identify all surgical site infections during the 1 year after surgery. In addition, partly because of the availability of unique institutional resources, we were able to ascertain and examine difficult to obtain clinical risk factors known to be associated with surgical site infections. This rarely is possible in other settings. Finally, although our cohorts are large, it is questionable whether the number of surgical site infections is adequate to ensure a reliable model of this many candidate predictor variables and avoid overfitting. We assessed this using internal validation and found a modest amount of overfitting based on model C statistics which were overoptimistic by approximately 0.03 to 0.04. Since overfitting tends to produce overestimated effects, we estimated the calibration slope in the bootstrap procedure and found that model accuracy would benefit from some shrinkage in these regression coefficients (by approximately 20% to 25%) which further suggests overspecification in the modeling.
Our results indicate that the discrimination (ability to separate patients who do and do not experience surgical site infections) of claims-based models is poor (bias-corrected C statistics, 0.629 for THA and 0.585 for TKA), but substantially improved with the addition of four well-established clinical risk factors for surgical site infections, particularly in patients who underwent THA. Nevertheless, discrimination of the clinical models remained modest (bias-corrected C statistics were less than 0.7), suggesting that other clinical risk factors could be incorporated to risk-adjustment models of surgical site infections to further improve prediction.
Surgical site infections, particularly, prosthetic joint infections, cause substantial patient morbidity and mortality, and contribute to arthroplasty-related healthcare costs [21, 35]. Although the overall incidence of surgical site infections is low, the absolute number is expected to increase in the future owing to increased volume of total joint replacement procedures and growth in the proportion of high-risk patients. Numerous surgical site infections potentially are preventable through effective management of patient risk factors and implementation of hospital-based infection-prevention practices. Surveillance of surgical site infections rates is considered an important quality measure for hospital profiling and future value-based purchasing and pay-for-performance programs. However, not all patients have the same baseline risk for surgical site infections and hospitals and surgeons differ in the case mix of their patients. Therefore, risk adjustment for case-mix differences is an essential component of comparisons of surgical site infections across hospitals. Apart from issues related to case-mix adjustment, risk prediction of surgical site infections for an individual patient can define specific prevention strategies. Reliance on claims data is becoming increasingly popular because the data are readily available and offer the ability to conduct surveillance activities with minimal additional data-collection requirements. However, many of the strong predictors of surgical site infections in total joint replacement are intrinsic patient factors and/or surgical factors that are not adequately captured in claims data. Coupled with the inadequacy of claims data in terms of completeness and coding errors, there are concerns regarding the potential consequences of using claims data for case-mix adjustment in surveillance of surgical site infections in total joint replacement [15, 29]. Therefore, it is important that the findings of claims-based analyses are validated against high-quality, clinically derived data, because prior studies of various medical conditions and cardiac procedures indicate that the concordance between administrative claims data and medical-record documentation may vary substantially, and adding clinical data to claims data can significantly improve risk prediction and also alter performance ratings of hospitals [13, 17, 30].
Our analyses are an external assessment of claims-based risk models for surgical site infections in one hospital, focusing on two important measures of predictive performance . Discrimination is the ability of a model, using a set of predictor variables, to accurately separate patients who did and did not experience surgical site infections. The C statistic is the standard approach to quantifying discrimination, where a value of 0.5 indicates that discrimination is no better than chance, whereas larger values indicate better discrimination. We found the discrimination of claims-based models (especially in TKA) to be lacking, with a bias-corrected C statistic of approximately 0.6. This may be attributable to two factors: lack of prediction of the claims-based risk models and/or a more homogeneous case mix in our cohort (that is, low variability of known and unknown risk factors for surgical site infections and consequently, patients being more alike). Only a few of the claims-based comorbidities in our study were found to be significantly associated with surgical site infections, indicating lack of model prediction. Surprisingly, three of four clinically well-recognized risk factors for surgical site infections  were not found to be predictive for surgical site infections among patients who underwent TKA. Furthermore, our study cohort possibly had a more severe case mix and was more homogeneous than other large multicenter cohorts—in terms of risk factors and rate of surgical site infections—such as the Medicare population. Nevertheless, it is not fair to discard claims-based models based on the low C statistic in one external dataset, plus the C statistic is only one of many discrimination measures. The integrative discrimination improvement analysis indicates substantial improvement in risk classification among individuals with surgical site infections in the THA cohort. Although the average integrative discrimination improvement of 0.36% in patients with surgical site infections seems like a negligible amount, one must consider this is on a scale of absolute risk and generally the overall risk of surgical site infections is low (approximately 1.7%). Additional studies are warranted to document how comorbidities and clinical risk-factor profiles vary across hospitals and the implications for hospital performance profiling, if any.
To our knowledge, no previous claims-based analyses reported on the performance of the prediction models used, therefore, it is not possible to compare our findings with previously published analyses. Furthermore, study definitions of comorbidities are not alike, except in one study . Although the prevalence of some more common comorbidities and the risk estimates appear to be comparable, there are important differences. For example, the relatively low prevalence of dementia and congestive heart failure may be a reflection of the younger patients and higher proportion of women in our cohort compared with the Medicare datasets. Furthermore, residual confounding is a potential limitation with ICD-based comorbidity definitions owing to coding inaccuracy, completeness, and absence of severity information . An important unexpected finding of our study is that the risk factors for surgical site infections and performance of prediction and risk-adjustment models may differ depending on type of joint replacement surgery. This was suggested previously  and deserves further investigation. If true, analyses of outcome data for surgical site infections from combined cohorts should be stratified by surgery type.
In terms of calibration (agreement between observed and predicted outcomes of surgical site infections), it is encouraging that calibration of claims-based models appeared adequate in THA and TKA cohorts, indicating that the predicted probabilities of surgical site infections from the claims-based models were fairly agreeable with the observed rates of surgical site infections. The addition of the four clinical risk factors provided little incremental improvement. However the relatively low incidence of surgical site infections combined with a lack of predictive discrimination resulted in a rather narrow and skewed spread of the 10-decile risk groups from these models. A model that is well calibrated across the entire range of predictions is particularly important in hospital profiling. For example, if the calibration of a surgical site infections model is poor in the most at-risk patients, then a hospital with a worse case mix can be unfairly labeled a poor performer. Our study is limited to one institution and further validation studies in larger, multicenter datasets are needed to better scrutinize the implications of calibration for hospital profiling. Further validation studies are warranted to determine how well claims-based and clinical models perform as a case-mix adjustor across hospitals. The addition of clinical risk factors to claims models can significantly improve discrimination, which suggests that if used in the clinical setting, a higher degree of accuracy in predicting surgical site infections risk is achievable.
Claims-based risk-adjustment models for surgical site infections appeared to be well calibrated in our large cohort of patients who underwent total joint replacements; however, the models lacked strong predictive discrimination. Our findings underscore the importance of external validation studies to better understand the validity and generalizability of claims-based risk models for surveillance of surgical site infections in total joint replacements, and how they can be interpreted when assessing case-mix adjustment and risk prediction in the clinic.
1. Aggarwal VK, Tischler EH for Workgroup 1 (Leaders: Lautenbach C, Williams GR Jr; Delegates: Abboud JA, Altena M, Bradbury T, Calhoun J, Douglas D, Del Gaizo DJ, Font-Vizcarra L, Huotari K, Kates S, Koo KH, Mabry TM, Moucha CS, Palacio JC, Peel TN, Poolman RW, Robb WJ 3rd
, Salvagno R, Seyler T, Skaliczki G, Vasarhelyi EM, Watters WC 3rd
). Mitigation and Education. In: Parvizi J, Gehrke T, eds. Proceedings of the International Consensus Meeting on Periprosthetic Joint Infection. Available at: http://www.msis-na.org/wp-content/themes/msis-temp/pdf/ism-periprosthetic-joint-information.pdf
. Accessed November 14, 2014.
2. Anderson DJ, Chen LF, Sexton DJ, Kaye KS. Complex surgical site infections and the devilish details of risk adjustment: important implications for public reporting. Infect Control Hosp Epidemiol.
3. Berbari EF, Hanssen AD, Duffy MC, Steckelberg JM, Ilstrup DM, Harmsen WS, Osmon DR. Risk factors for prosthetic joint infection: case-control study. Clin Infect Dis.
4. Berbari EF, Osmon DR, Lahr B, Eckel-Passow JE, Tsaras G, Hanssen AD, Mabry T, Steckelberg J, Thompson R. The Mayo prosthetic joint infection risk score: implication for surgical site infection reporting and risk stratification. Infect Control Hosp Epidemiol.
5. Bozic KJ, Lau E, Kurtz S, Ong K, Berry DJ. Patient-related risk factors for postoperative mortality and periprosthetic joint infection in medicare patients undergoing TKA. Clin Orthop Relat Res.
6. Bozic KJ, Lau E, Kurtz S, Ong K, Rubash H, Vail TP, Berry DJ. Patient-related risk factors for periprosthetic joint infection and postoperative mortality following total hip arthroplasty in Medicare patients. J Bone Joint Surg Am.
7. Bozic KJ, Ong K, Lau E, Berry DJ, Vail TP, Kurtz SM, Rubash HE. Estimating risk in Medicare patients with THA: an electronic risk calculator for periprosthetic joint infection and mortality. Clin Orthop Relat Res.
8. Brandt C, Hansen S, Sohr D, Daschner F, Ruden H, Gastmeier P. Finding a method for optimizing risk adjustment when comparing surgical-site infection rates. Infect Control Hosp Epidemiol.
9. Calderwood MS, Kleinman K, Bratzler DW, Ma A, Bruce CB, Kaganov RE, Canning C, Platt R, Huang SS. Centers for Disease Control and Prevention Epicenters Program; Oklahoma Foundation for Medical Quality. Use of Medicare claims to identify US hospitals with a high rate of surgical site infection after hip arthroplasty. Infect Control Hosp Epidemiol.
10. CDC Centers for Disease Control and Prevention. National Healthcare Safety Network (NHSN). Available at: http://www.cdc.gov/nhsn/
. Accessed November 11, 2014.
11. Charlson ME, Pompei P, Ales KL, MacKenzie CR. A new method of classifying prognostic comorbidity in longitudinal studies: development and validation. J Chron Dis.
12. Dowsey MM, Choong PF. Obesity is a major risk factor for prosthetic infection after primary hip arthroplasty. Clin Orthop Relat Res.
13. Fonarow GC, Pan WQ, Saver JL, Smith EE, Reeves MJ, Broderick JP, Kleindorfer DO, Sacco RL, Olson DM, Hernandez AF, Peterson ED, Schwamm LH. Comparison of 30-day mortality models for profiling hospital performance in acute ischemic stroke with vs without adjustment for stroke severity. JAMA.
14. Gibbons C, Bruce J, Carpenter J, Wilson AP, Wilson J, Pearson A, Lamping DL, Krukowski ZH, Reeves BC. Identification of risk factors by systematic review and development of risk-adjusted models for surgical site infection Introduction to the research. Health Technol Assess.
16. Grosso LM, Curtis JP, Lin Z, Geary LL, Vellanky S, Oladele C, Ott LS, Parzynski C, Suter LG, Berhneim SM, Drye EE, Krumholz HM (Yale New Haven Health Services Corporation/Center for Outcomes Research & Evaluation (YNHHSC/CORE). Hospital-level Risk-Standardized Complication Rate Following Elective Primary Total Hip Arthroplasty (THA) And/Or Total Knee Arthroplasty (TKA): Measure Methodology Report. Prepared for the Centers for Medicare & Medicaid Services. 2012. Available at: https://staging.qualitynet.org/
. Accessed November 11, 2014.
17. Hammill BG, Curtis LH, Fonarow GC, Heidenreich PA, Yancy CW, Peterson ED, Hernandez AF. Incremental value of clinical data beyond claims data in predicting 30-day outcomes after heart failure hospitalization. Circ Cardiovasc Qual Outcomes.
18. Jackson ML, Nelson JC, Jackson LA. Why do covariates defined by International Classification of Diseases codes fail to remove confounding in pharmacoepidemiologic studies among seniors? Pharmacoepidemiol Drug Saf.
19. Jamsen E, Huhtala H, Puolakka T, Moilanen T. Risk factors for infection after knee arthroplasty: a register-based analysis of 43,149 cases. J Bone Joint Surg Am.
20. Kurtz SM, Lau E, Schmier J, Ong KL, Zhao K, Parvizi J. Infection burden for hip and knee arthroplasty in the United States. J Arthroplasty.
21. Kurtz SM, Lau E, Watson H, Schmier JK, Parvizi J. Economic burden of periprosthetic joint infection in the United States. J Arthroplasty.
22. Kurtz SM, Ong KL, Lau E, Bozic KJ, Berry D, Parvizi J. Prosthetic joint infection risk after TKA in the Medicare population. Clin Orthop Relat Res.
23. Leening MJ, Cook NR. Net reclassification improvement: a link between statistics and clinical practice. Eur J Epidemiol.
24. Mu Y, Edwards JR, Horan TC, Berrios-Torres SI, Fridkin SK. Improving risk-adjusted measures of surgical site infection for the national healthcare safety network. Infect Control Hosp Epidemiol.
25. Namba RS, Inacio MC, Paxton EW. Risk factors associated with surgical site infection in 30,491 primary total hip replacements. J Bone Joint Surg Br.
26. Ong KL, Kurtz SM, Lau E, Bozic KJ, Berry DJ, Parvizi J. Prosthetic joint infection risk after total hip arthroplasty in the Medicare population. J Arthroplasty.
2009;24:6 suppl105-109 10.1016/j.arth.2009.04.027.
27. Osmon DR, Berbari EF, Berendt AR, Lew D, Zimmerli W, Steckelberg JM, Rao N, Hanssen A, Wilson WR. Diagnosis and management of prosthetic joint infection: clinical practice guidelines by the Infectious Diseases Society of America. Clin Infect Dis.
28. Parvizi J, Berbari EF, Bauer TW, Springer BD, Della Valle CJ, Garvin KL, Mont MA, Wongworawat MD, Zalavras CG. New definition for periprosthetic joint infection: from the Workgroup of the Musculoskeletal Infection Society. Clin Orthop Relat Res.
29. Sarrazin MS, Rosenthal GE. Finding pure and simple truths with administrative data. JAMA.
30. Shahian DM, He X, Jacobs JP, Rankin JS, Peterson ED, Welke KF, Filardo G, Shewan CM, O'Brien SM. Issues in quality measurement: target population, risk adjustment, and ratings. Ann Thorac Surg.
31. Song KH, Kim ES, Kim YK, Jin HY, Jeong SY, Kwak YG, Cho YK, Sung J, Lee YS, Oh HB, Kim TK, Koo KH, Kim EC, Kim JM, Choi TY, Kim HY, Choi HJ, Kim HB. Differences in the risk factors for surgical site infection between total hip arthroplasty and total knee arthroplasty in the Korean Nosocomial Infections Surveillance System (KONIS). Infect Control Hosp Epidemiol.
32. Steiner C, Andrews R, Barrett M, Weiss A. HCUP Projections: Mobility/Orthopedic Procedures 2003 to 2012.
HCUP Projections Report # 2012-03. Online September 20, 2012. U.S. Agency for Healthcare Research and Quality. Available: http://www.hcup-us.ahrq.gov/reports/projections/2012-03.pdf
. Accessed November 11, 2014.
33. Steyerberg EW, Vickers AJ, Cook NR, Gerds T, Gonen M, Obuchowski N, Pencina MJ, Kattan MW. Assessing the performance of prediction models: a framework for traditional and novel measures. Epidemiol.
34. Urquhart DM, Hanna FS, Brennan SL, Wluka AE, Leder K, Cameron PA, Graves SE, Cicuttini FM. Incidence and risk factors for deep surgical site infection after primary total hip arthroplasty: a systematic review. J Arthroplasty.
35. Zimlichman E, Henderson D, Tamir O, Franz C, Song P, Yamin CK, Keohane C, Denham CR, Bates DW. Health care-associated infections: a meta-analysis of costs and financial impact on the US health care system. JAMA Intern Med.