Cohen, Elaine R. BA; Feinglass, Joe PhD; Barsuk, Jeffrey H. MD; Barnard, Cynthia MBA, MSJS; O'Donnell, Anna RN, BSN; McGaghie, William C. PhD; Wayne, Diane B. MD
Recent reimbursement decisions by the Centers for Medicare and Medicaid Services1 have increased national focus on reducing preventable complications in hospitalized patients. Prior work has demonstrated that interventions which reduce preventable complications such as catheter-related bloodstream infections (CRBSI) also decrease hospital costs.2–5 Although there is increased interest in cost savings associated with interventions to reduce hospital-acquired infections, little is known about the cost-effectiveness of interventions such as simulation-based education.
Simulation has become an increasingly popular technique to teach and assess the skills of trainees in invasive procedures.6,7 Several studies show that simulation-based education improves actual patient care in areas such as emergency airway management,8 laparoscopic surgery,9,10 colonoscopy,11,12 bronchoscopy,13 and Advanced Cardiac Life Support.14
Beginning in 2006, Northwestern University Internal Medicine and Emergency Medicine residents completed simulation-based mastery learning in central venous catheter (CVC) insertion before clinical rotations in the Medical Intensive Care Unit (MICU).15,16 This program resulted in improved trainee skill and reduced CVC insertion complications, including a significant decrease in the rate of CRBSI.15–17 The aim of this study was to estimate hospital cost savings related to the reduction in CRBSI and examine the extent to which these savings offset simulation training costs.
Study Design and Data Collection
This was an intervention evaluation study of cost savings related to a simulation-based intervention in CVC insertion in the MICU of Northwestern Memorial Hospital (NMH), an 897-bed tertiary-care urban teaching hospital. The NMH MICU is a 20-bed facility that treats approximately 1500 patients/yr. Five or six second and third year internal medicine and emergency medicine residents rotate in the MICU every 4 weeks.
Previous research demonstrated that the simulation-based mastery learning program in CVC insertion, as opposed to conventional hospital infection control efforts such as patient care bundles, was primarily responsible for a highly significant reduction in the MICU CRBSI rate.17 In the prior study, 98% of CVCs were inserted by residents, and there was an 84.5% decrease in CRBSIs. Importantly, this reduction was sustained for 16 months after the intervention.17 Based on concurrent comparison with an NMH surgical intensive care unit staffed by residents who had not received the simulation intervention and lengthy discussions with clinicians, infection control nurses and hospital staff, who could not identify any other mechanism for the observed highly significant reduction in CRBSI rates, we concluded that the observed decline in the rate of CRSBI was a direct result of simulation-based education (Figure 1).17
This study calculated the financial implications of the simulation-based intervention by comparing CRBSI rates for the year before the intervention (December 2005 to November 2006) to the year after simulator-trained residents began staffing the MICU (December 2006 to November 2007). Using hospital cost accounting data, we derived the estimated incremental cost and length of stay (LOS) associated with MICU patients with a CVC and a CRBSI during the study period. We then compared savings from reduced CRBSI after the simulation training intervention to the cost of the simulation training. The Northwestern University Institutional Review Board approved the study.
From December 2006 to November 2007, 69 second and third year internal medicine and emergency medicine residents completed a mandatory simulation-based education program in CVC insertion before rotating in the MICU.15–17 The training included two 2-hour education sessions featuring a lecture, ultrasound training, deliberate practice with the CVC simulator (Simulab's CentralLineMan), and instructor feedback. The CVC simulator features realistic tissue with ultrasound compatibility, an arterial pulse and self-sealing veins and skins. Education sessions used uniform didactic material on CVC indications and complications, as well as stepwise demonstrations of internal jugular and subclavian CVC insertions. Evidence-based guidelines for CRBSI reduction were emphasized (hand washing, full sterile barrier technique, chlorhexidine skin preparation, avoidance of the femoral site, and prompt CVC removal).2–4 These sessions were supervised by a senior hospitalist faculty member with expertise in CVC insertion (J.H.B.). Residents were evaluated before (pretest) and after (posttest) simulation training using a validated checklist.15 At posttest, residents were required to meet or exceed a minimum passing score set by an expert panel for both internal jugular and subclavian procedures.18 After completing simulation-based mastery learning in CVC insertion, the simulator-trained residents rotated through the MICU beginning in December 2006.
Cost Accounting Framework and CRBSI Rates
CRBSI incidence was measured by NMHs Infection Control and Prevention staff using standard criteria in accordance with protocols described by the National Healthcare Safety Network, formerly the National Nosocomial Infections Surveillance System.3,19 All positive blood cultures from the MICU were identified, and medical records were reviewed by trained infection control personnel to determine whether criteria for CRBSI were met. Infection control personnel determined CRBSI incidence in accordance with routine hospital policies and were blind to the nature of this study and the timing of the simulation-based intervention.
Using NMH administrative data with International Classification of Diseases (ICD-9)-coded procedures, we determined that 477 patients had a CVC inserted in the MICU during the study period. Patients were included in the analysis only if the CVC insertion was performed during the patients' documented MICU stay. These records were then matched to infection control CRBSI incidence data. We found that before our simulation-based intervention, the average infection rate was 4.2/100 MICU CVC admissions (11 infections in 239 CVC patients). After the intervention, when all second and third year residents (n = 69) in the MICU received simulation-based training, the rate was reduced to 0.42/100 admissions (one infection in 238 CVC patients), thus preventing an estimated 9.95 CRBSI cases in the year after the intervention. We used the estimated additional cost, and LOS of these 9.95 prevented CRBSIs to compute cost savings for the year after simulation training.
Hospital costs were obtained from the hospital's finance department. Their cost estimates, including both direct and indirect costs, are allocated to activity charge codes within each department based on either relative value units, such as labor minutes or supply costs, or ratio of cost to charges. All costs were adjusted to 2008 dollars using the Bureau of Labor Statistics Annual Consumer Price Index for inpatient hospital services.20 Simulation training costs were based on a facility rental rate of $45/h, and staff and faculty support was estimated as a salary percentage. Supply costs were compiled and also adjusted to 2008 dollars using the Bureau of Labor Statistics Annual Consumer Price Index for medical supplies.20
Two methods are customarily used to analyze the cost-effectiveness of nonrandomized quality improvement interventions.21–23 One approach uses a regression-derived propensity score to equate similar groups of patients based on their predicted probability of experiencing a specific adverse event. These results are then used to estimate cost differences between matched cases and controls, based solely on matched controls with a propensity score similar to the cases to avoid obvious selection biases that might favor the intervention.21,22 A second, more direct approach relies on estimating the attributable cost of adverse events directly from a risk-adjusted regression analysis of costs, using the coefficient on adverse event incidence in the regression model to estimate the additional costs associated with each adverse event prevented.23 We performed both of these analyses to estimate the costs and additional hospital days associated with CRBSI (SPSS Version 17, Chicago, IL).
First, we estimated the predicted probability of CRBSI for all 477 MICU patients who had a CVC inserted during the study period from a multiple logistic regression model controlling for patient age, sex, and each patient's Charlson Score, a proxy for severity of illness.24 The Charlson score is based on 17 clinically significant comorbidities, derived from secondary ICD-9 diagnosis codes based on the Deyo administrative data method,25 and predicts 1-year mortality for hospitalized medical patients. Each comorbidity is assigned a specific weight based on the degree of its relative risk, and these weights are combined to generate an overall score for each patient.24 The resulting logistic regression-predicted probability of CRBSI produces a CRBSI propensity score (ranging from 0.0 to 1.0) for each patient.
Then, we eliminated all patients whose probability of a CRBSI was lower than any observed patient with a CRBSI from our analysis; these patients were deemed to have noncomparable risk and were thus unsuitable controls. We compared mean hospital costs between “case” patients with a CRBSI and the remaining “control” patients who did not have a CRBSI across four strata (quartiles) of roughly equal CRBSI propensity scores. Summing costs across each case-control strata allowed us to make an initial estimate of true additional costs (and hospital days) related to the CRBSI incidence.
Next, we produced a duplicate analysis in which we again eliminated all patients with CRBSI propensity scores that were lower than any observed patient with a CRBSI. We estimated the association of CRBSI incidence with costs and LOS using linear regression models that controlled for MICU CVC patients' age, sex, and Charlson score. The coefficient on CRBSI incidence in those models reflects the expected costs and hospital days associated with CRBSI after controlling for the effects of case mix differences. We present linear rather than log-transformed results because of ease of interpretation and the fact that antilogged results were very similar.
During the first year of the intervention, operating costs to train 69 residents, including supplies, faculty and staff time, and space rental, totaled $111,916. These initial costs included onetime purchases of Simulab's CentralLineMan, an ultrasound machine and a supply cart. Other supply costs included CVC kits, replaceable CentralLineMan tissues, ultrasound probe covers, and sterile gowns and drapes (Table 1). The predicted annual cost to maintain the project was $89,455 in 2008 dollars.
Case Control Estimated Cost and LOS of a CRBSI
During the preintervention period, the 239 CVC patients had a mean age of 60.5 years (SD = 16.4), were 56% female, and had a mean Charlson score of 3.54. The 238 CVC patients in the postintervention period had a mean age of 61.1 years (SD = 17.1), were 55% female, and had a mean Charlson score of 3.97; these differences were nonsignificant. The regression-derived CRBSI propensity scores for predicted probability of CRBSI ranged from 0.2% to 6.9% across all 477 patients with MICU CVC insertions. The 12 actual CRBSI patients had propensity scores ranging from 1.2% to 6.8%. Sixty-eight non-CRBSI cases were thus excluded as controls because their predicted probability of infection was less than 1.2%. The remaining 409 “control” patients were divided into four strata based on nearly equal propensity scores (Table 2) and matched to “case” patients with CRBSI in the same predicted probability strata.
Using the propensity score matched case-control comparison method, we averaged the quartile results and estimated the mean costs and LOS (total hospital days and MICU days) of a patient with a CRBSI, when compared with the mean costs and LOS of a patient without a CRBSI. Mean differences between case and control patients across quartiles ranged from $45,000 to $100,000, from 5 to 22 hospital days, and 13 to 27 MICU days. Summing across these quartile differences, we attributed overall mean additional costs of $82,730, 14.2 additional patient hospital days and 12.1 patient MICU days to the incidence of a CRBSI (Table 2).
Linear Regression Estimated Cost and LOS of a CRBSI
Linear regression models of cost and LOS controlling for age, sex, and Charlson score produced very similar results for the cost associated with an incident CRBSI. The coefficients on CRBSI in these models provided an estimate of $82,005 (P < 0.001, 95% CI = $51,299–$112,791), 13.8 additional patient hospital days (P = 0.001, 95% CI = 5.9–21.6), and 12.2 additional patient MICU days (P < 0.001, 95% CI = 8.8–20.8) associated with a CRBSI.
Using these two methods and 9.95 CRBSI prevented by the simulation-based intervention, the total annual estimated savings were $823,164 and $815,950,141 and 137 patient hospital days and 120 and 121 MICU days. When compared with the cost of our intervention ($111,916), the net savings ranged from $704,034 to $711,248 (a 7–1 rate of return).
Several authors have called for studies linking medical education and simulation to reduced downstream healthcare costs.26–28 Our results add to what is known about simulation- based education by documenting cost savings from improved quality of care that far exceed the cost of the educational intervention. Our findings demonstrate convincingly that simulation-based education in CVC insertion was a cost-effective intervention that directly benefited patients. Specific patient benefits included fewer CRBSIs and decreased length of MICU and total hospital stays.
This study confirms prior work linking the presence of a CRBSI to prolonged LOS and increased hospital costs.4,23 Both methods produced a CRBSI-associated cost of approximately $82,000. This is higher than past estimates of CRBSI-associated total hospital costs ranging from $34,50822 to $63,572.23 However, these estimates were based on data from 1998 to 2000, and ours reflect 2006–2007 data adjusted to 2008 dollars.
When interpreting any study of cost savings, it is critical to be sure that the intervention was truly responsible for lower costs. This is due to the fact that system, cultural, and process changes can also result in improved quality of care.29 However, we believe that the simulation-based intervention was the primary reason for the substantial cost savings we report. This is because no other process or quality improvement changes occurred in the MICU during the study period, and CRBSI rates did not decline in other ICU settings at NMH. Additionally, MICU staff members (fellows, faculty, and nurses) were not involved in our study and were likely unaware that simulation training was ongoing. For these reasons, we do not believe that cuing, reinforcement of CVC insertion algorithms, or other changes in unit procedures and protocols were responsible for the reduction in hospital costs shown in this study.
This study has several additional limitations. First, it reflects the experience at one hospital over a relatively short period of time. However, it is similar in scope to other reports regarding CRBSI rate reduction in critical care settings.2,5,21 Second, our regression models were simple. Inclusion of more detailed clinical data related to illness severity and susceptibility to infection might improve predictions of cost and LOS and change model coefficients. However, our cost estimates may be conservative because they do not include significant costs often incurred after hospital discharge such as home intravenous antibiotics and home health nursing services. Third, our facility rental fee of $45/h may be modest compared with fees at other simulation centers. This is primarily due to the fact that we only used space and did not incur charges for equipment, simulator staff, or supplies. Despite this, our analysis shows our intervention would remain cost-effective with higher facility rental fees. Specifically, a rate of $1000/h still provides nearly 2 to 1 return.
In summary, our results show that a simulation-based education program in the MICU of an academic teaching was very cost-effective. There are ample reasons for clinicians, hospitals, and patients to seek to decrease the incidence of CRBSI. With the trend toward reduced Medicare reimbursement to hospitals for patient care involving hospital-acquired conditions including CRBSI,1 it is imperative for providers to consider interventions, such as the use of simulation, to improve quality, and reduce reimbursement risk. These findings highlight the promise of simulation-based education to reduce costs and mitigate a significant source of harm in the critical care environment.
1. O'Grady NP, Alexander M, Dellinger EP, et al. Guidelines for the prevention of intravascular catheter-related infections. Centers for Disease Control and Prevention. MMWR Recomm Rep
2. Sherertz RJ, Ely EW, Westbrook DM, et al. Education of physicians-in-training can decrease the risk for vascular catheter infection. Ann Intern Med
3. Pronovost P, Needham D, Berenholtz S, et al. An intervention to decrease catheter-related bloodstream infections in the ICU. N Engl J Med
4. Warren DK, Zack JE, Mayfield JL, et al. The effect of an education program on the incidence of central venous catheter-associated bloodstream infection in a medical ICU. Chest
5. Coopersmith CM, Rebmann TL, Zack JE, et al. Effect of an education program on decreasing catheter-related bloodstream infections in the surgical intensive care unit. Crit Care Med
6. Boulet JR, Murray D, Kras J, Woodhouse J, McAllister J, Ziv A. Reliability and validity of a simulation-based acute care skills assessment for medical students and residents. Anesthesiology
7. Issenberg SB, McGaghie WC, Hart IR, et al. Simulation technology for health care professional skills training and assessment. JAMA
8. Mayo PH, Hackney JE, Mueck JT, Ribaudo V, Schneider RF. Achieving house staff competence in emergency airway management: results of a teaching program using a computerized patient simulator. Crit Care Med
9. Andreatta PB, Woodrum DT, Birkmeyer JD, et al. Laparoscopic skills are improved with LapMentor training: results of a randomized, double-blinded study. Ann Surg
2006;243:854–860; discussion 860–853.
10. Seymour NE, Gallagher AG, Roman SA, et al. Virtual reality training improves operating room performance: results of a randomized, double-blinded study. Ann Surg
2002;236:458–463; discussion 463–454.
11. Cohen J, Cohen SA, Vora KC, et al. Multicenter, randomized, controlled trial of virtual-reality simulator training in acquisition of competency in colonoscopy. Gastrointest Endosc
12. Sedlack RE, Kolars JC. Computer simulator training enhances the competency of gastroenterology fellows at colonoscopy: results of a pilot study. Am J Gastroenterol
13. Blum MG, Powers TW, Sundaresan S. Bronchoscopy simulator effectively prepares junior residents to competently perform basic clinical bronchoscopy. Ann Thorac Surg
2004;78:287–291; discussion 287–291.
14. Wayne DB, Didwania A, Feinglass J, Fudala MJ, Barsuk JH, McGaghie WC. Simulation-based education improves quality of care during cardiac arrest team responses at an academic teaching hospital: a case-control study. Chest
15. Barsuk JH, McGaghie WC, Cohen ER, Balachandran JS, Wayne DB. Use of simulation-based master learning to improve the quality of central venous catheter placement in a medical intensive care unit. J Hosp Med
16. Barsuk JH, McGaghie WC, Cohen ER, O'Leary KJ, Wayne DB. Simulation-based mastery learning reduces complications during central venous catheter insertion in a medical intensive care unit. Crit Care Med
17. Barsuk JH, Cohen ER, Feinglass J, McGaghie WC, Wayne DB. Use of simulation-based education to reduce catheter related bloodstream infections. Arch Intern Med
18. Wayne DB, Barsuk JH, Cohen E, McGaghie WC. Do baseline data influence standard setting for a clinical skills examination? Acad Med
19. Horan TC, Andrus M, Dudeck MA. CDC/NHSN surveillance definition of health care-associated infection and criteria for specific types of infections in the acute care setting. Am J Infect Control
20. United States Department of Labor, Bureau of Labor Statistics Consumer Price Index for Medical Care. Available at: http://www.bls.gov/data/
. Accessed April 16, 2009.
21. Kaushal R, Bates DW, Franz C, Soukup JR, Rothschild JM. Costs of adverse events in intensive care units. Crit Care Med
22. Digiovine B, Chenoweth C, Watts C, Higgins M. The attributable mortality and costs of primary nosocomial bloodstream infections in the intensive care unit. Am J Respir Crit Care Med
23. Warren DK, Quadir WW, Hollenbeak CS, Elward AM, Cox MJ, Fraser VJ. Attributable cost of catheter-associated bloodstream infections among intensive care patients in a nonteaching hospital. Crit Care Med
24. Romano PS, Roos LL, Jollis JG. Adapting a clinical comorbidity index for use with ICD-9-CM administrative data: differing perspectives. J Clin Epidemiol
1993;46:1075–1079; discussion 1081–1090.
25. Deyo RA, Cherkin DC, Ciol MA. Adapting a clinical comorbidity index for use with ICD-9-CM administrative databases. J Clin Epidemiol
26. Gaba DM. The future vision of simulation in health care. Qual Saf Health Care
27. Dunkin B, Adrales GL, Apelgren K, Mellinger JD. Surgical simulation: a current review. Surg Endosc
28. Prystowsky JB, Bordage G. An outcomes research perspective on medical education: the predominance of trainee assessment and satisfaction. Med Educ
29. Gaba DM, Dunn WF. Procedural risks in thoracentesis: process, progress, and proficiency. Chest
© 2010 Lippincott Williams & Wilkins, Inc.