As nursing professional development (NPD) practitioners, we are challenged by the question “What is the return on investment (ROI) for professional development activities?” As described in Part 1 of this series, NPD practitioners are often the first to be called when a problem exists and the first to have funding restricted when budgets are tight. In Part 1, we discussed the Kirkpatrick, Phillips, and Paramoure program evaluation models, followed by a summary of the literature reporting on ROI for professional development activities.
The synthesis of the studies on educational interventions providing a calculation of financial aspects shows no consistent method to describe financial and clinical impact of professional development activities (Opperman, Liebig, Bowling, Johnson, & Harper, 2016). The trend of reporting outcomes associated with learning activities has given rise to the next level of expectation: demonstration of financial impact of educational interventions.
This article defines the concepts of an economic assessment including simple cost analysis, benefit–cost ratios, cost-effectiveness analysis (CEA), and ROI, as well as providing formulas to calculate each. Three fictional examples of various-sized educational programs are used to demonstrate how to make these calculations and use them for decision-making.
COST ANALYSIS, BENEFIT–COST RATIOS, AND COST-EFFECTIVENESS ANALYSIS
Prior to discussing ROI, an understanding of the concepts of cost analysis, benefit–cost ratios, and CEA is essential when calculating the actual financial impact of professional development activities.
From a financial perspective, cost analysis is the initial consideration when developing an educational program. Cost analysis simply determines the least expensive option. The formula for cost analysis is to add all the costs for the program and divide it by the number of participants to obtain the cost per participant. See Figure 1 for the formulas for cost analysis. When considering multiple learning modalities, this simple cost per participant can be compared.
Although cost analysis provides information on the efficiency or least expensive modality, it does not consider program outcomes. The benefit–cost analysis compares program benefits to program costs as a ratio using dollars. The first step is clearly identifying desired program outcomes that can be observed and measured.
The next step is calculating all program costs. The benefit–cost ratio formula uses all benefits (i.e., increased productivity, quality, safety improvements, reduced turnover, increased patient volumes) and all costs (i.e., program development time, faculty costs, training supplies, equipment costs, facility fees, salary cost for employee attendance/replacement cost) to determine the financial return from the program (Warren, 2013). See Figure 1 for the formulas for cost analysis, benefit-cost ratio and return on investment.
CEA goes one step further in economic assessment, because it compares two or more different educational interventions and their outcomes. The NPD practitioner may have an option of a self-study that takes the learner an average of 2 hours to complete or a 90-minute live workshop with the same content. Both modalities are intended to accomplish the same outcome. The costs must be monetary values and calculated as cost analysis for each possible intervention. The outcomes, however, do not need to be monetary values; consider them the benefits gained from the educational intervention. For example, nonmonetary outcomes might be “increased patient engagement” or “fewer staff reporting incivility.” The combination of the cost per participant (cost analysis) and the benefits, whether monetary (calculated as a benefit–cost ratio) or nonmonetary, are used to determine the CEA.
Benefit–cost ratio and CEA collectively impact decisions about program changes and resources. Although cost analysis alone may demonstrate efficiency through lower costs, the benefit–cost analysis may demonstrate that the same program is not as effective in achieving desired outcomes. Clearly, comprehensive program evaluation requires consideration of both components—efficiency and effectiveness (Kettner, Moroney, & Martin, 2013).
In the economic assessment of a program, calculation of ROI provides further data for administrative decision-making. Calculating ROI (a) provides information for justification of programs for budgetary planning, (b) contributes to clinical decision-making and resource allocation, and (c) demonstrates the value of education.
Because of the complexity of determining ROI for programs, pragmatically, it is used in only about 5%–10% of program planning processes for priority decision-making like regulatory, higher-risk, or more expensive programs (DeSilets, 2010.)
Steps in calculating ROI:
- Identify program desired outcomes.
- Describe educational interventions proposed to meet these outcomes.
- Plan the logistics of the educational intervention with sufficient detail to identify expenses.
- Calculate program costs (planning time, supplies, setup time, faculty and staff time, etc.).
- Calculate potential savings (cost of turnover, pressure ulcer, litigation, inefficiency of program changes).
- Compare costs to savings (efficiency).
- Determine specific outcomes using observable and measurable terms (effectiveness).
In order to calculate the benefit of educational interventions, an outcome must be quantified. For example, changes in orientation should lead to greater new employee competence, confidence, and satisfaction, therefore reducing turnover. Another example is that an education activity on the catheter-associated urinary tract infections (CAUTI) bundle should lead to reduction of CAUTIs. When calculating the benefit of the educational intervention for either of these examples, the cost (of a new RN leaving or the average cost of a CAUTI) should be used to counter the cost of the program. For examples of published average costs per case of poor outcomes, see Table 1. The formula for calculating the ROI is found in Figure 1.
EXAMPLES OF FINANCIAL IMPACT ANALYSIS
Consider these examples of fictitious educational activities and how financial impact can be calculated through cost analysis, benefit–cost ratios, CEA, and ROI analysis.
Example 1: One-hour self-study compared to live class.
Situation. The organization considers requiring a 1-hour Web-based self-study module for 650 learners on changes in the procedure for pressure ulcer prevention bundle.
Background. The hospital incidence of hospital-acquired pressure ulcers (HAPU) is 25%, which is above the national average of 17% (Roe & Williams, 2014). The Centers for Medicare and Medicaid Services no longer reimburses facilities for patients with newly acquired Stage 3 and Stage 4 pressure ulcers.
According to the Agency for Healthcare Research and Quality (2014b), a full-thickness pressure ulcer costs an average of $17,286 per incident to treat. This does not include the additional emotional and physical burden for the patient.
According to the American Faculty Association (2012), the time needed to develop educational programs is 4 hours of preparation for each hour of class presented. This varies widely from Kapp and Defelice (2009) that estimates 40 (self-instructional print), 43 (stand-up classroom training), and 49 (instructor-led, Web-based training) hours per hour of training are needed for development. For purposes of these fictitious scenarios, it is assumed that the NPD practitioner is well informed of changes in pressure ulcer care and is an experienced NPD practitioner; consequently, the number of development hours is less.
Assessment. The program costs of a Web-based self-study module for supplies, salaries, and equipment are calculated as follows. The computers and software are in place; thus, no further initial expense for equipment is needed.
Cost analysis. The cost for a Web-based module is $32.78 per participant, and the cost of the live classes is 35.03. The Web-based class costs $2.25 less per participant. Additional considerations for cost analysis include the cost of educating new hires. If the organization hires an additional 50 nurses over the course of a year, the only expense for the Web-based course is the hourly salary of the new nurses. No additional program costs are incurred. If presented in a live format, in addition to the newly hired nurses’ salary, additional costs would include the NPD practitioner salary for each class and the additional administrative support salary.
Benefit–cost ratio. To calculate benefit–cost ratio, benefits are divided by total costs. Using $17,286 as the per case cost to treat a full thickness pressure ulcer, prevention of two pressure ulcers results in cost savings of $34,572. This results in a positive benefit–cost ratio for both Web-based and live class formats.
Using Web-based self-study module:
Using live class format:
(<1 = negative impact, > = positive impact)
Cost-effectiveness analysis. The difference in cost between the Web-based modules at $21,310 and the live classes at $22,775 is $1,465. The savings for the Web-based format is only positive if the Web-based course and the live classes are comparable in outcomes. CEA requires an evaluation of effectiveness of each modality in achieving the same outcomes. In this scenario, the Web-based course was determined to be equal in effectiveness resulting in similar outcomes to the live presentation. As a result, the Web-based course is more cost-effective.
ROI. To evaluate the ROI in this example, the cost of the pressure ulcer treatment must be compared to the cost of the education. As previously stated, if this educational intervention prevents two pressure ulcers, $34,572 is saved. The ROI for the Web-based self-study is 62.2%, and the live class format is 51.7%.
Using Web-based self-study module:
Using live class format:
Recommendation. In both the live and Web-based courses, the ROI is positive and easily justifies the education. The Web-based course, however, shows a higher ROI. The cost analysis, benefit–cost ratio, CEA, and ROI all demonstrate a more positive financial impact with the Web-based course. As a result, the NPD practitioner recommends development of a Web-based educational program on prevention of pressure ulcers.
Example 2: Eight-hour live class on workplace violence.
Situation. On the basis of an identified professional practice gap, a continuing education program on workplace violence is planned for 25 hourly employees from the Emergency Department.
Background: At a large, central city acute care facility, gun violence is a concern. The United States has the greatest number of gun-related injuries per capita compared to all other industrialized nations at 10.3 per 100,000 (fatal and nonfatal) occurring in 2011 (Jena, Sun, & Prasad, 2014). The average cost per incident is $18,722 (Chong et al., 2015). Escalation of violent behaviors resulted in 11 reported incidents in the Emergency Department last year. Because domestic violence frequently involves gun injuries, the community is at high risk for gun-related injuries, and escalation of violent behaviors has occurred at increasing frequency in the Emergency Department, an educational program is proposed to help increase employee safety.
Assessment. In calculating the costs for this program, the NPD department purchased predeveloped content for this course, so development time was reduced. A content expert was used to review potential programs for purchase, select one, and prepare to facilitate the course with the identified development time. Instead of 43 hours per hour of content (43 × 7 = 301 development hours) required to develop this course, 70 hours were needed (7 content hours × 10 hours = 70 hours to review, select, and prepare to facilitate this program).
Cost analysis. Simple cost analysis is the total cost of the educational intervention divided by the number of staff members participating in the education. Cost of the class per person: $15,350/25 participants = $ 614.00 per participant.
Benefit–cost ratio. When calculating the benefit–cost ratio, the total benefit is divided by the total cost. In this scenario, if one incident of violence is prevented in 1 year at $18,722 average cost per incident (Chong et al., 2015), the net benefit is $18,722. See Table 1 for published costs of outcomes. A positive benefit to the organization is noted with the calculation:
Cost-effectiveness analysis. In this example, the cost-effectiveness compares the cost of providing the 8-hour program with the current practice of no educational program. The cost comparison is $614.00 per participant versus no educational expense. The outcome is the number of incidents reported. If the educational intervention demonstrates a reduced incidence of workplace injury from violence, that outcome is better that the current data of 11 incidents last year.
Return on investment. If the proposed program produces a modest result of one less reported incident of workplace violence, the ROI is 22%.
During the year following education, two fewer incidents are reported:
Recommendation. With the increased incidents of workplace violence, employers must demonstrate due diligence to protect employees, patients, and visitors by preventing these incidents. From the combination of CEA, benefit–cost ratios, and ROI calculations, this program is clearly recommended.
Example 3: Frequency of a multi-day orientation.
Situation. An organization conducts a 7-day interprofessional orientation 10 times per year. It is considering increasing to 12 times per year to accommodate more timely incorporation of newly hired employees.
Background. The number of participants per cohort has ranged from 30 to 70. The significant range of cohort sizes makes it difficult to plan for room size, number of stations for skills lab, computer training rooms, faculty schedules, and handout preparation. In addition, participant satisfaction drops with decreased learner engagement in large classes. If decreased engagement leads to poor socialization and increased turnover, Robert Wood Johnson Foundation (2010) places the average cost for replacing an RN at $36,567. An increase to offering orientation 12 times a year eliminates cohorts with more than 45 participants and saves last minute planning time related to human resources communication, room scheduling, class coordinating, and faculty availability. Because the classes are already developed and current, additional development time is not needed. The comparison is between the current state of 10 offerings a year and the proposed change to 12 offerings per year.
Assessment. Program costs are more extensive for a 7-day program. Salaries, supplies, equipment, and even rental of space are considered.
Additional costs incurred when a cohort is over 45 participants includes two additional Informatics Nurse instructor-days for the electronic medical record class due to lack of computers and doubling the skills stations on the skills day, requiring more faculty, equipment, and space. For illustration, when a cohort is 65 instead of 45, the following costs are added:
The cost of orienting a large cohort (65 participants) is calculated by starting with the costs of the 45 participant cohort and adding expenses incurred for the larger group.
Large class per orientation cost (65 in cohort):
Cost analysis. The cost per participant is calculated by adding the expenses from a year’s worth of orientation classes and dividing it by the total number of people oriented.
Total cost $44,750/510 = $87.74/participant
Benefit–cost ratio. If offering orientation 12 times per year improves the socialization, confidence, and competence of new nurses resulting in two fewer nurses leaving before their first anniversary, a savings of $73,134 ($36,567 × 2) is realized. On the basis of 12 offerings, the benefit–cost ratio is positive and reflects positive organizational impact.
12 offerings per year
Cost-effectiveness analysis. This proposal indicates a cost of $79.17/participant for the planned 12 offerings, which is less than $87.74 /participant for 10 offerings with three large groups. If the pattern seen from large orientation classes is a higher turnover rate by first year anniversary of employment, improving socialization to the institution through more personal contact in the first weeks of orientation should improve retention. With CEA, the outcomes of both interventions (10 offerings and 12 offerings) must be compared. A conservative cost of turnover is $36,567 per employee (Robert Wood Johnson Foundation, 2010). See Table 1 for further published costs of outcomes including new RN orientation cost.
Return on investment. The ROI for offering 12 orientations per year is calculated by using the average cost for replacing two RNs of $73,134 and the cost of 12 months of offering the orientation at $40,380 in the ROI formula. The result is an ROI of 81.11%.
Return on investment:
Recommendation. By combining the calculations for cost analysis, benefit–cost ratio, CEA, and ROI, strong support for increasing the frequency of the offerings of orientation is noted. The decrease in cost per participant from $87.74 to $79.17 is a financial argument, yet when the ROI of reducing turnover is considered, it becomes a strong recommendation. Smaller cohorts allow more small group exercises to be incorporated and require fewer skills stations. Smoother centralized orientation, offered at closer intervals, should improve the new employee experience and contribute to satisfaction and retention.
This example was conservative on the benefit calculation both in averaging the cost of replacement and in estimating the number of retained staff after 1 year as a result of this educational intervention. The recommendation to reduce the cohort size and increase the frequency of offerings from 10 to 12 times per year is based on the financial and clinical impact as manifested in the better outcome of higher retention and less cost per participant.
IMPLICATIONS FOR FUTURE RESEARCH
No consistent method is routinely reported in the literature to describe the financial and clinical impact of professional development activities. Researchers and NPD practitioners reporting on educational program evaluations must regularly calculate financial impact when disseminating and publishing results. This evidence can be used to guide decisions for limited resources and to better position NPD as integral in the decision-making process in healthcare organizations.
NPD practitioners must measure the impact of education interventions to demonstrate the success of professional development activities. One seldom addressed aspect is the financial impact measurement. The two articles in this series show how routine approaches have been used (e.g., Kirkpatrick’s Levels of Evaluation, Phillips’ Five-Level ROI Framework, and Paramoure’s Measurable Instructional Design) to measure ROI in professional development. Critical appraisal of the literature, both quantitative and qualitative, revealed the importance of reporting more than participant satisfaction.
Four methods for evaluating the financial impact of educational activities were reviewed, including cost analysis, benefit–cost ratio, CEA, and ROI; plus examples were given using these methods. More consistent measuring and reporting of the financial and clinical impact of NPD activities is warranted.
The NPD practitioners must proactively demonstrate the value of educational programs. During lean economic times, participant attendance and satisfaction are not adequate metrics to convince leaders of the organizational value of educational activities.
Agency for Healthcare Research and Quality. (2014a). Interim update on 2013 annual hospital-acquired condition rate and estimates of cost savings and deaths averted from 2010 to 2013
. Retrieved from http://www.ahrq.gov/sites/default/files/wysiwyg/professionals/quality-patient-safety/pfp/interimhacrate2013.pdf
Agency for Healthcare Research and Quality. (2014b). The concentration of health care expenditures and related expenses for costly medical conditions, 2012
(Agency for Healthcare Research & Quality Medical Expenditure Panel Survey Statistical Brief #455). Retrieved from http://meps.ahrq.gov/mepsweb/data_files/publications/st455/stat455.pdf
American Faculty Association. (2012, February 8). Hours for teaching and preparation rule of thumb: 2–4 hours of prep for 1 hour of class
. Retrieved from http://americanfacultyassociation.blogspot.com/2012/02/hours-for-teaching-and-preparation-rule.html
ANA’s Handle With Care Program (2016). Safe patient handling and mobility
. Retrieved from http://www.nursingworld.org/handlewithcare
Bardach N. S., Coker T. R., Zima B. T., Murphy J. M., Knapp P., Richardson L. P., … Mangione-Smith R., (2014). Common and costly hospitalizations for pediatric mental health disorders. Pediatrics
, 133(4), 602–609.
Chong V. E., Smith R., Garcia A., Lee W. S., Ashley L., Marks A., … Victorino G. P. (2015). Hospital-centered hospital violence intervention programs: A cost-effectiveness analysis. American Journal of Surgery
, 209, 597–603.
DeSilets L. D. (2010). Calculating the financial return on educational programs. Journal of Continuing Education in Nursing
, 41(4), 149–150.
Duff J., Walker K., Omari A., & Stratton C. (2013). Prevention of venous thromboembolism in hospitalized patients: Analysis of reduced cost and improved clinical outcomes. Journal of Vascular Nursing
, 31(1), 9–14.
Dychter S., Gold D., & Haller M. (2012). Subcutaneous drug delivery. The Art and Science of Infusion Nursing
, 35(3), 154–160.
Fletcher P. C., Markoulakis R., & Bryden P. J. (2012). The cost of caring for a child with autism spectrum disorder. Issues in Comprehensive Pediatric Nursing
, 35, 45–69.
Hollenbeak C. (2011). The cost of catheter-related bloodstream infections. The Art and Science of Infusion Nursing
, 1(5), 309–313.
Jegler B. J., Johnson T. J., Engstrom J. L., Patel A. L., Loera F., & Meier P. (2013). The institutional cost of acquiring 100 mL of human milk for very low birth weight infants in the neonatal intensive care unit. Journal of Human Lactation
, 29(3), 390–399.
Jena A. B., Sun E. C., & Prasad V. (2014). Does the declining lethality of gunshot injuries mask a rising epidemic of gun violence in the United States? Journal of General Internal Medicine
, 29(7), 1065–1069.
Juillard C., Smith R., Anaya N., Garcia A., Kahn J. G., & Dicker R. A. (2015). Saving lives and saving money: Hospital-based violence intervention is cost-effective. The Journal Trauma Acute Care Surgery
, 78(2), 252–257.
Kang J., Mandsager P., Biddle A. K., & Weber D. J. (2012). Cost-effectiveness analysis of active surveillance screening for methicillin-resistant staphylococcus aureus in an academic hospital setting. Infection Control and Hospital Epidemiology
, 33(5), 477–486.
Kettner P., Moroney R., & Martin L. (2013). Designing and managing programs: An effectiveness-based approach
. Washington, DC: Sage Publications, Inc.
Kuntz J. L., Johnson E. S., Raebel M. A., Petrik A. F., Yang X., Thorp M. L., … Smith D. H. (2012). Epidemiology and healthcare costs of incident clostridium difficile infections identified in the outpatient healthcare setting. Infection Control and Hospital Epidemiology
, 33(10), 1031–1038.
Opperman C., Liebig D., Bowling J., Johnson C., & Harper M. (2016). Measuring return on investment for professional development activities: A review of the literature. Journal for Nurses in Professional Development
, 32(3), 122–129.
OSHA Safety Pays Program Estimator. (2016). Estimated costs of occupational injuries and illnesses and estimated impact on a company’s profitability worksheet
. Retrieved from https://www.osha.gov/dcsp/smallbusiness/safetypays/estimator.html
Robert Wood Johnson Foundation. (2010). Wisdom at work: Retaining experienced nurses
. Retrieved from http://www.rwjf.org/en/library/research/2010/07/wisdom-at-work–retaining-experienced-nurses.html
Roe E., & Williams D. L. (2014). Using evidence-based practice to prevent hospital acquired pressure ulcers and promote wound healing. The American Journal of Nursing
, 114(8), 61–65.
Schifalacqua M. M., Mamula J., & Mason A. R. (2011). Return on investment imperative. Nursing Administration Quarterly
, 35(1), 15–20.
Trepanier S., Early S., Ulrich B., & Cherry B. (2012). New graduate nurse residency program: A cost–benefit analysis based on turnover and contract labor usage. Nursing Economic$
, 30(4), 207–214.
Trepanier S., & Hilsenbeck J. (2014). A hospital system approach at decreasing falls with injuries and cost. Nursing Economic$
, 32(3), 135–141.
Tsimicalis A., Stevens B., Ungar W. J., Greenberg M., McKeever P., Agha M., … Moineddin R. (2013). Determining the costs of families’ support networks following a child’s cancer diagnosis. Cancer Nursing
, 36(2), E8–E19.
Warren J. I. (2013). Program evaluation and return on investment. In Bruce S. L.(Ed.), Core curriculum for nursing professional development
(4th ed. pp. 547–68). Chicago, IL: Association for Nursing Professional Development.