Lessons Learned From Measuring Return on Investment in Public Health Quality Improvement Initiatives

Abstract

Context: 

The Center for Public Health Quality and its partner, North Carolina State University Industrial Extension Service, used 2 existing, yet similar quality improvement (QI) programs to provide technical assistance to conduct return on investment (ROI) and economic impact (EI) analyses so that they could estimate their QI projects' financial impacts.

Objectives: 

The objectives of this article are to describe the approach and ongoing learning from applying ROI and EI analyses to public health QI projects and analyze the results in order to illustrate ROI potential in public health.

Design: 

We used a before-after study design for all ROI and EI analyses, spanning a 3-year time period.

Setting: 

The study was conducted as part of 2 existing public health QI training programs that included webinars, face-to-face workshops, on-site facilitation, and longitudinal coaching and mentoring.

Participants: 

The QI training programs included multidisciplinary teams from local and state public health programs in North Carolina.

Main Outcome Measure: 

Return on investment and EI calculations.

Results: 

Numerous adaptations were made over the 3 years of the ROI program to enhance participant's understanding and application. Results show an average EI of $149 000, and a total EI in excess of $5 million for the 35 projects studied. The average ROI per QI project was $8.56 for every $1 invested in the project.

Conclusions: 

Adapting the ROI approach was important in helping teams successfully conduct their ROI analyses. This study suggests that ROI analyses can be effectively applied in public health settings, and the potential for financial return is substantial.

In Brief

This study aims to describe the approach and ongoing learning from applying ROI and EI analyses to public health QI projects and analyze the results in order to illustrate ROI potential in public health.

Public health practice is undergoing rapid change, facing the need to deliver more with fewer resources. The adoption of quality improvement (QI), which has been used in many industries to improve the quality of services and products, is one of the major changes in the field.¹ In addition, governmental fiscal austerity is placing intense pressure on public health agencies, and indeed on all aspects of government, to demonstrate the value and impact of the public funds they receive.²

Quality improvement is a definitive process that responds to community and population health needs using continuous and measurable approaches.³ It requires significant upfront investment (such as staff time, new hiring, training, etc), yet promises many potential benefits, including financial benefits. Similarly, population health improvements, a common focus of public health organizations, not only require upfront investments but also promise health benefits and potential health care savings. To justify QI efforts, and especially those involving population health improvements, leaders have to understand the potential costs and benefits of the proposed work.

Return on investment (ROI) analysis is a method that is widely used in business settings to examine costs and benefits related to strategies and investments.⁴ Since ROI evolved as a business tool, it is not widely understood, nor frequently utilized by public health leaders and practitioners. However, largely due to the aforementioned changes and a pressing urgency to optimize program efficiency and outcomes in today's economic climate, ROI assessment is increasingly receiving attention among public health leaders. A notable example is that ROI was the theme of the 2013 National Public Health Week.⁵

The objectives of this article are twofold. First, we describe the approach and ongoing learning from a cross-sector partnership between the Center for Public Health Quality (CPHQ) and the North Carolina State University Industrial Extension Service (IES). This partnership has been applying ROI analysis to QI projects at local and state governmental public health organizations for 3 years. We summarize IES and CPHQ's unique experiences in performing ROI analysis on the QI projects that they support, including how they adapted ROI from business settings to public health settings. The second objective is to analyze the ROI results of QI projects completed by IES and CPHQ in order to illustrate the potential ROI that public health QI projects can yield.

Methods

Context, setting and participants

The CPHQ and the IES designed and administered 2 QI training programs. The Public Health Quality Improvement 101 (PH QI 101) program provides QI training and project support for local health departments (LHDs) and has been described in detail in previous studies.^6,7 The Division of Public Health Quality Improvement 101 (DPH QI 101) is a similar program that was adapted to meet the needs of state public health programs. Both are experiential learning programs designed to build capacity to implement QI in local and state public health entities. Both programs are taught by QI experts from CPHQ and IES, hereby referred to as CPHQ/IES faculty. The QI programs are conducted in cohorts that include 5 to 10 volunteer teams that arise from local and state entities and their partners.

Throughout the program, teams learn by applying QI tools and methods to a specific project within their workplace. Over the 8-month program, the teams attend educational sessions and receive coaching and technical assistance via on-site facilitation, webinars, and two 2-day workshops. During these sessions, they learn methods and tools to test, implement, spread, and sustain QI initiatives and conduct ROI analysis. Teams also participate in a 4-day rapid improvement event, or kaizen event (a formalized, focused event performed by the team over several days to achieve dramatic improvements), that includes on-site facilitation by a Lean expert from IES, hereby referred to as IES specialist. Lean is a continuous process improvement methodology that focuses on identifying and eliminating wasteful activities, “defining value from the customer's perspective....”⁸ The program culminates with teams developing ROI and economic impact (EI) calculations for their QI projects.

ROI calculation method

We used a before-after study design⁹ for all ROI and EI analyses, spanning a 3-year time period. This design enabled us to compare pre- to postchange intervention results.

Return on investment is a performance measure used to evaluate the efficiency of an investment. Economic impact refers to the analysis of costs and benefits of an activity. Financial benefits are tangible benefits that can be converted to monetary values. For the QI 101 programs, financial benefits have subcategories: internal benefits are related to LHD or state programmatic improvements (such as time savings due to improving patient visit time or reducing the number of process steps) and external benefits (such as reducing health care costs by preventing patients from going to the emergency department or identifying child developmental issues early that impact lifelong education costs) that reflect population or community-related benefits. Costs are distinguished as one-time costs that occur during the QI 101 program and represent the cost to implement the project (such as team time) and recurring costs, which are those that will be routinely incurred to continue the changed process and sustain the QI initiative.

During trainings, each team is expected to think through these EI/ROI components and apply the formulae that are listed below:

Table 1 shows an example analysis. In the example, the QI team incorporated into a small clinic a tobacco cessation approach—the 5As: Ask, Advise, Assess, Assist, and Arrange. The project's internal benefits resulted from embedding a template in the clinic's electronic medical record that prompted providers to acquire and document the appropriate 5A information. Submitting the required documentation on approximately 1300 patients per year resulted in additional Medicaid revenue. The team also achieved financial benefits related to time savings by implementing checklists and protocols that further streamlined and supported efficiencies in the new process. When the clinic began referring tobacco-using patients to the Quitline (QuitlineNC provides free cessation services to any North Carolina resident who needs help quitting tobacco use¹²), they were able to apply evidence, per the cited resources, to support the financial benefits outlined in the external/community benefits section of Table 1.

TABLE 1: • Example ROI Calculation—Tobacco Prevention: Implementing 5As in a Small Clinic

To calculate one-time project costs to implement improvements, the team applied an average salary rate to their time invested in the project. They also included costs of educational materials and supplies as one-time and recurring costs.

These 3 elements—Internal Benefits, External Benefits, and Costs—are then applied to the ROI formulae and results interpreted as follows:

• Internal ROI: For every $1.00 invested in this QI project, $3.31 was returned to the participating clinic (via time savings and additional Medicaid revenue). These financial returns can then be reinvested into ongoing QI opportunities at the clinic.
• Combined internal and external ROI: For every $1.00 invested in this QI project, $18.36 was returned to the clinic and the community (via the internal savings mentioned previously and the reduced medical costs and increased productivity of the non–tobacco-using workforce).

In addition to financial benefits, each team was required to include intangible, or nonmonetary, benefits. Although many methods exist for converting intangibles into monetary values, such as the Intangible Benefit Multiplier Pyramid Approach,¹³ we chose not to quantify these benefits. First, monetizing intangibles can be complex and time-consuming,¹⁴ and we needed to keep our model simple since public health agencies have limited resources. In addition, the process for converting intangibles is subjective,¹⁴ and we did not want to decrease the ROI's credibility with stakeholders. In our Table 1 case example, the tobacco team's intangible benefits included the following:

• Increased clinical staff/physician knowledge through 5A and tobacco cessation training that enabled them to better serve patients;
• Identified smokers more readily through the new tobacco questionnaire/template that was embedded in the electronic medical record for provider use; included steps and reminders for all 5As, billing, medication, and counseling; and
• Improved patient knowledge of secondhand smoke effects and health care risks due to 5A implementation.

Teams are taught that costs used in the ROI calculation relate to expenses incurred while attending trainings, the kaizen event and team meetings; testing and implementing improvements; and/or purchasing supplies or materials needed for improvements. During ROI training, teams are instructed to track all expenses related to the QI project. The key expense incurred is team member time. The total team member time is easily converted into an hourly salary amount. Any other expenses, whether one-time or recurring, need to be retained by the team for the ROI analysis.

Data collection and analysis

Early in the QI training, teams develop an aim statement, which defines the overall purpose and importance of the project and includes measurable goals that ultimately help teams demonstrate the effectiveness of their improvements. In addition, they develop a measurement plan that includes the following components¹⁵:

• Outcome measures—the ultimate result the team is trying to achieve
• Process measures—what is done to achieve the desired outcome
• Balancing measures—something that inadvertently could be harmed while trying to improve the process and outcome

Teams are guided to define at least 1 goal for each measure type and document their measures in a detailed project measurement table (see measurement table, Supplemental Digital Content 0, available at https://links.lww.com/JPHMP/A129). After developing the measurement table, the baseline data were gathered, change interventions were tested and implemented, and remeasurement of the goals occurred. When these postimplementation data had been collected, they were compared with the baseline data and analyzed for actual improvements derived from the QI project. From this information and the successful process changes that were adopted, teams determined the benefits that could be converted into monetary values and annualized.

As the measurements related directly to the QI project's aim, they most often became part of the data utilized in the ROI calculation. For example, in the Table 1 tobacco prevention case, they increased the number of clinic patient referrals to the Quitline. Since the clinic had not been referring patients previously, the QI team implemented a referral process that generated more than 100 patient referrals per year. Evidence¹⁰ shows that such referrals lead to an increase of patients who actually stop tobacco use, which is linked to savings in direct medical costs and lost productivity for employers.¹¹ Thus, the referral goal derived from the aim statement became the basis for a financial benefit that can be utilized in the ROI calculation.

Although teams were introduced to ROI calculation methods early in the program, they were also prompted by their IES specialist to consider the benefits of their change ideas during the kaizen event. Draft ROI reports, which were due 60 days after the final workshop, were submitted by QI teams to the CPHQ ROI team (several staff who were trained how to review submissions). This ROI team checked the benefits that were converted into monetary values and all mathematical calculations, probed for potential opportunities to monetize benefits that were overlooked, verified that sufficient evidence was given to support the findings, and ensured that all estimates of financial impact were conservative. The QI teams also submitted an ROI draft report to their senior leader for initial review and feedback. Within the next 30 days, the final report was submitted to their senior leader for approval before submitting it to CPHQ.

Upon receiving the completed ROI analyses, CPHQ staff entered ROI results into a Microsoft Excel spreadsheet. The spreadsheet was designed to filter data and provide mean and cumulative results, which could be customized for CPHQ stakeholders as well as analysis for the present study.

Reporting ROI/EI results

Guidance about communicating ROI results to stakeholders is provided to QI teams during the final workshop, when teams' projects are wrapping up and they are ready to begin the calculations. Communication tips include the following:

• Refer to the stakeholder analysis that was included in the QI project's charter. Develop a plan for when, how, and what to communicate to specific stakeholder audiences.
• Schedule face-to-face presentations since ROI is relatively new for public health audiences. In-person presentation allows a more thorough explanation of the methodology and a chance to emphasize meaningful aspects of the ROI analysis. It also provides opportunity to address questions immediately and dispel misinterpretations.
• Illustrate with examples that the ROI calculation is based on evidence, thereby strengthening its credibility. When converting benefits and costs into monetary values, describe the conservative approach utilized.
• Focus on how the ROI contributes to achievement of the organization's mission versus solely on dollars saved.
• Negate assumptions that positive results should be used to reduce budgets. This kind of thinking could starve their most innovative and effective performers.
• Emphasize the value of the intangible benefits. Sometimes overlooked, but when presented effectively often carry substantial influence.
• Consider stakeholders with whom ROI results should not be communicated. Some projects are important to improve although labor-intensive team time causes costs to exceed financial benefits. Misunderstood negative ROIs can lead to devaluation of intangible benefits.

Interventions

Although IES had been teaching and utilizing ROI in QI efforts in manufacturing and health care settings, the adaptation and application of ROI to the QI 101 public health projects evolved through substantial iterative learning. These adaptations over the 3-year period are listed below.

Adaptation 1: Early introduction of ROI in the QI 101 program

Because ROI was not included in the QI 101 training programs until 2011 (the PH QI 101 program originated in 2010 and DPH QI 101 in 2011), ROI had to be introduced to early cohorts retroactively. We learned quickly that this retroactive approach was neither effective nor sustainable and also saw inconsistencies in ROI calculation approaches, accuracy, and the number of completed submissions. In these early cohorts, processes and expectations for teams to complete and submit ROI work were less defined, thus we had limited success in obtaining accurate and timely ROI results. The newness of ROI terms and critical thinking, coupled with learning how to apply the knowledge to calculate financial benefits accurately and credibly, was commonly reported as issues by teams.

In 2011, ROI training was incorporated into QI 101 near the end of the program. After observing many teams struggle to complete ROI analyses, receiving participant feedback about needing ROI knowledge at the beginning of the program, and researching ROI best practices, we began to introduce ROI thinking early for subsequent QI 101 cohorts.

Adaptation 2: Ongoing technical assistance and coaching

To align with the early introduction of ROI in the training programs, we also asked each QI team's faculty coach to address ROI throughout the coaching experience to continue to sensitize team thinking toward this work. In addition, IES specialists helped the team identify intangible and tangible benefits and costs during the kaizen event, while process improvements are fresh in their minds.

Adaptation 3: Improved structure and guidance for QI teams

To help QI 101 participants understand the entire ROI process, CPHQ/IES faculty created a model to guide them through planning, executing, and communicating ROI results. This “big picture” approach proved important to provide them with understanding early in the QI project. The “ROI Process Model for Public Health” is shown in the Figure.

FIGURE: •. ROI Process Model for Public Health^a

Adaptation 4: New tools and templates to facilitate ROI process

In 2012, we created new tools to aid QI teams in developing ROI analyses, while promoting a standardized method. The ROI Memory Jogger Questionnaire (see questionnaire, Supplemental Digital Content 1, available at https://links.lww.com/JPHMP/A130) and the ROI Memory Jogger Worksheet (see worksheet, Supplemental Digital Content 2, available at https://links.lww.com/JPHMP/A131) prompt QI teams to identify benefits and costs to include in their calculation. Participants confirmed that the tools were useful in directing their thinking and approach.

Adaptation 5: Changing ROI calculations to be more comprehensive

Based on feedback from participants and stakeholders, we altered the ROI calculation method to better match the needs of QI projects in public health settings and the QI 101 program. For example, we changed the time period from 1 to 2 years and began utilizing a one-time versus recurring costs approach to offset year 1 costs. These costs were associated with team time to learn and apply QI concepts and tools at workshops, webinars, kaizen events, and so forth, and would not occur in the project's year 2. We also added the external/community benefit section to distinguish between the benefits to the program/organization and benefits to the community (external). This change not only provided clarification as to internal and external benefits but also accentuated the far-reaching effects that some improvements had on the community.

Related to these changes was the adaptation of our ROI Calculation Form (see form, Supplemental Digital Content 3, available at https://links.lww.com/JPHMP/A132) to better provide step-by-step guidance for listing the 2-year benefits and costs as well as formulae to automatically generate the ROI and the EI.

Adaptation 6: Ensuring teams complete the ROI assignment

In early cohorts, we had difficulty obtaining ROI analyses from PH QI 101 teams. This was due to several problems that are presented, along with improvements, as shown in Table 2. By implementing changes, a more recent PH QI 101 cohort achieved 100% ROI analysis completion. Since DPH QI 101 participants had access to an on-site IES specialist for coaching and follow-up, high completion rates were achieved from this program's inception.

TABLE 2: • Improvements to Increase ROI Analysis Completion Rate

Results

Between June 2011 and March 2014, 83 teams completed QI 101. Nineteen of these teams were not required to submit ROI results (early cohorts) or are currently in the process of developing the analyses. It is the 64 remaining teams that are included in the present study of ROI results. Approximately 55% of these 64 teams completed an ROI analysis.

Table 3 illustrates the internal and external financial benefits, costs, EI, and ROI over 2 years for 35 state and local QI projects. Remarkably, all but 2 QI projects produced a positive EI. The negative ROIs result when the project costs exceed the financial benefits. Negative ROIs in QI 101 projects were due to labor-intensive projects that raised costs (such as Web site or survey-related improvements) and inability to make the changes necessary to generate financial benefits (such as QI teams that encountered cultural barriers in interdepartment initiatives).

TABLE 3: • ROI Results

The QI projects overall yielded greater internal ($3 178 859) than external ($2 654 504) benefits, although both were substantial. Overall EI averaged $149 233 and the average ROI was $8.56. Comparing state to local results, the average internal benefits among LHD projects exceeded those from state projects. However, average external benefits for the state projects exceeded local projects.

Among the LHD teams, 46% completed ROI analysis. These LHD teams achieved more than $2.4 million in internal benefits as compared with $1.2 million in external/community benefits. These combined internal and external benefits minus costs of $421 458 resulted in more than $3.2 million in EI for the 24 projects or an average EI of $134 753. Their average ROI was $7.67.

The DPH teams had a 92% ROI completion rate. The 11 DPH teams completing the analysis achieved more than $700 000 in internal benefits and nearly $1.5 million in external benefits. Considering the combined benefits and total costs of $188 738, their EI was more than $1.9 million, with an average EI of more than $180 000. Their ROI averaged $10.54.

Discussion

Investing in QI projects via the QI 101 programs generated substantial financial benefits at both the state and local levels. The greatest benefit from these 6- to 9-month QI projects was internal, yet the communities also reaped significant external benefits. Not surprising, given the often greater scope of state programs, state QI projects showed the greatest community benefits.

The ROI process utilized by the CPHQ/IES partnership underwent a gradual transformation over the 3 years covered by this study, requiring a number of iterative adaptations derived from QI team and CPHQ/IES faculty feedback, unexpected results or problems, and best practices from other industries. Nevertheless, this study demonstrates that ROI analyses can be adapted and applied to QI projects in public health settings.

Calculating ROI for public health initiatives has proven to be difficult—lag times exist between implementation and resulting benefits, calculating community benefit and the value of a life is difficult, and proving causation between a project and the benefit can be cumbersome.¹⁶ Despite these challenges, ROI is beginning to be used more frequently in public health to communicate the value of public health and prevention. Return on investment analyses are currently used in 3 areas: program effectiveness, aggregate public health spending, and QI projects.¹⁷

Although the resources for calculating ROI for public health improvement projects are limited, there are notable efforts in addition to the present study that are attempting to develop the tools that public health organizations need to calculate ROI for their projects. One of these efforts is the Association of State and Territorial Health Officials' (ASTHO) ROI tool for improvement projects undertaken through the National Public Health Improvement Initiative. Like the CPHQ/IES tool, the calculations are done for the user once the data are collected and entered into the Excel spreadsheet. Although this tool is still in a pilot phase and is being tested by a user network, ASTHO has incorporated many of the same fundamental concepts as the CPHQ/IES effort. For example, the ASTHO model stresses the importance of building ROI into the project inception, measuring baseline data, categorization of costs and benefits, and involving both the QI team and the end users (ie, senior leaders) in the creation of the ROI report. Notable differences include ASTHO's inclusion of more complex concepts and calculations such as amortization and present value calculations.¹⁷

Another effort is the Center for Health Care Strategies ROI Forecasting Calculator and ROI Template (template was developed by researchers at the University of North Carolina at Chapel Hill). Like both the CPHQ/IES tool and the ASTHO tool, Center for Health Care Strategies' tool also calculates the ROI for the user once the data are collected and entered. However, this tool is specific to clinical projects focused on patients served by Medicaid and requires detailed information such as risk stratification.¹⁸ Return on investment calculators also exist for specific QI projects, such as the Public Health Institute's ROI tool for remote patient-monitoring programs¹⁹; however, these tools may be too project-specific to be generalized to QI projects.

Although ROI is just now emerging as a priority in public health, it is more common in other industries, such as health care. However, existing health care models have industry-specific components that are not easily adapted for public health. In addition, these models are resource-intensive to complete and recommend using complex calculations to adjust for inflation, discount future money, and depreciate equipment. These models also advocate for and suggest the involvement of multiple groups within the organization such as QI program staff, financial officers, clinical staff, statisticians, data analysts, programmers, and consultants.²⁰ These resources are unlikely to be available in many public health settings.

Ultimately, the approach to using ROI in public health settings must balance rigor with being simple enough to execute with the limited resources and overworked staff often seen in public health organizations.¹⁶ In the present study, we attempted to balance adequate rigor with enough simplicity and support to allow public health organizations that have never used ROI to successfully assess the EI of their QI work. In addition, we hope that the training and experience they gained will allow them to hardwire ROI calculations into other continuous improvement projects beyond the QI 101 program.

As stated earlier, the QI/Lean approach utilized in the QI 101 program is a continuous improvement methodology. As program participants learn and adopt this continuous improvement mindset and apply it to future process improvements, this may perpetuate additional EI/ROI. Thus, ROI could continue to evolve depending on the impact of the changes on goals and measurements.

Limitations

Quality improvement project team members were generally not financial officers, health economists, or accountants. Thus, their ROI results likely underestimate the actual financial benefits because of difficulty in identifying benefits and converting these to monetary values. We also stressed in all calculations that in order for ROI to be credible for stakeholders, financial estimates should be conservative. In addition, we were unable to estimate the financial benefits of the large number of public health professionals that learned how to apply QI to their work through the QI 101 program, such as the value of future QI projects they could perform and presumably would yield additional financial benefits. Thus, again, our estimates of EI and ROI are likely underestimates.

Teams participating in QI 101 had a large amount of expert support, such as assistance in performing their assignments and engagement with their leadership to encourage supporting the time needed for the team to work on their QI project as well as addressing barriers. Thus, these teams' EI/ROI results may not be generalizable to other QI projects supported with less expertise and/or time and resources that result in lesser project results.

Reporting negative ROIs could be disappointing to teams that worked so diligently with hopes of showing a positive return on their time and efforts. Thus, it is possible that teams with a negative EI/ROI may not have completed/shared their results, causing us to overstate overall results. However, we believe that this potential bias had little to no impact on our results because we promoted an atmosphere of accomplishment for team learning and project completion, honesty, and objectivity in reporting, and emphasized the importance of intangible benefits.

The ROI Process Model for Public Health includes the communication of results to key stakeholders as the final step. We were not able to monitor how teams communicated their ROI results—assisting teams with generating their final ROI analyses and continuously improving our assistance required all of the resources available in the present study. Going forward, we intend to focus more attention to communication of ROI results. This topic may be a good one for future studies.

A final potential limitation is that the ROI analyses in the present study were from the perspective of the QI teams and their organizations, thus we did not include the costs of the grant-funded QI 101 training program. These costs (a one-time cost of approximately $18 000 per team) were not included since they were not paid by the team's organization. Thus, a societal perspective would have demonstrated a lower EI (though still substantial) than presented in the present study.

Conclusions

Return on investment and EI analysis is increasingly being applied in public health. There is hope that it will be a viable and effective methodology to evaluate performance of public health QI projects and to justify program continuance during an era of decreasing budgets.

In this study, QI teams employed ROI/EI methodology to determine whether the QI project's benefits outweighed the investment. We have described our ROI approach and how we applied many lessons along the way to improve the method. Overall results to date reflect that ROI analyses can be effectively utilized in public health settings. Our article suggests that with ongoing practice and experience, ROI can become a meaningful, recognized application to assist public health in a new economic era.