In the first column of this two-part series, we discussed the measurement process to establish evidence for performance improvement and the need to ensure that the selected measures to assess performance are valid and reliable. We reviewed steps one and two of the measurement process: defining the concept of interest and creating an operational definition that describes how the concept will be measured. In this column, we'll take a look at the last three measurement process steps—developing or selecting measurement tools, planning data collection strategies, and testing and refining the measurement process—including issues that you may face when you actually attempt to apply metrics in a real-world setting. This involves data collection in the clinical environment, with actual clinical staff and patient care situations. We'll continue to illustrate our exploration using the two examples of patient satisfaction and nursing staff turnover.
Selecting measurement tools
In many cases, nurse leaders may find that the decision regarding which tool will be used to measure, for example, patient perceptions of care received, has already been made at the organizational level for reasons that may be slightly different than the program-specific goals you're addressing. The measure may only address a few dimensions of nursing care, such as responsiveness to call lights or nursing communication. The areas covered by the measure may not reflect the areas identified as a focus for improvement, creating a problem between the “fit” of the measure and the goal of nursing leadership. Feasibility may dictate the use of existing organizational data even if there isn't an exact match to your goals. In such cases, it's important to review the actual items being used to better interpret the results and generate improvement ideas.
Frequently, a measure uses individual respondent-level data that are then combined to provide a unit- or organizational-level result. For example, individual patients may be surveyed regarding perceptions of care after discharge and these data aggregated to provide a hospital average and, in some cases, unit-level averages. Another example is to count the number of staff exits from a unit and then calculate a turnover rate for that unit, or combine all exits to create a hospital-level result. Hospital-level data may then be used in reimbursement calculations or for assessing manager performance.
Because of the importance of performance data at the hospital level to institutional business objectives, it's important for a nurse leader to assess several things to ensure correct application of the data to a particular situation. This requires assessment of mathematic reliability and validity (Do the results truly represent the group being considered?), as well as conceptual validity (Is the measure providing information that relates to the question being asked?). For example, patient perceptions are often reported as a performance metric relative to the unit of discharge. A manager might ask several questions as unit-level findings are assessed: “How long were most patients on my unit?” “Was it most of the hospital stay or a few days at the end after a long and rocky stay in the ICU?” Another question might be, “Are there subgroups of patients discharged from my unit who are known to have had stressful hospitalizations and whose needs we may have trouble meeting?” These known variations in patient experience should be considered when communicating and using unit-level results so that clinical nurses understand that leaders recognize the patient care issues nurses face on a daily basis.
It's also important to assess the sample size for aggregated measures. This relates to reliability as a measure of the true range of possible data points. For instance, if a unit has a response rate of 50 patients and there were 1,000 discharges in a month, is 5% an adequate sample size? If there's a very small sample size, then the weight of one response may be exaggerated. For example, if one patient out of five respondents is “very dissatisfied” that drops the “average” for the unit substantially.1 Therefore, minimum response rates and sample sizes are often set by commercial systems as a guide for providing unit-level data.
In addition to assessing the validity of the actual measure, it's important for nurse leaders to assess the relevance and appropriateness of comparison groups if benchmarking is an important part of the metric. Benchmarks may be set up using one reference group when your hospital collects data differently from others in the reference group.
For instance, if nursing turnover is being measured, it's usually considered the number of individual staff exiting the organization divided by the number of individual staff on board at the beginning and end of the period, typically measured as an annual metric. If there are clinical nurses working in various departments that aren't part of the nursing department (such as OR or ambulatory care nurses) and those nurses are included in the hospital average, then this obviously needs to match what's included for benchmarking organizations.
For turnover assessment, it's also important to consider the purpose of the analysis and adjust the measurement accordingly. Are you investigating the unit-level concept of leadership effectiveness in staff retention or the organizational package of compensation offered by the organization at large? For unit-level performance improvement in turnover rates, it's important to count all exists, including transfers to other units, as a part of the metric. At the hospital level, however, the area of interest is often nurses who leave the hospital for another employer.
Planning data collection
After a nurse leader has settled on a way to measure the concept of interest, the data collection process should flow from the operational definition. In both the examples we've been using, an organization often has existing data available that can be captured and displayed to reflect performance over time. The development of a simple data recording tool in a spreadsheet program can facilitate construction of data tables and graphics to display performance.
For instance, a table for the turnover rate might include elements such as period of performance (calendar or fiscal year), exits meeting the required definition during that period, staff on board at the beginning of the period, staff on board at the end of the period, and a field for the calculated turnover rate for that period (exits divided by the average of staff on board at the beginning and end of the period). If available, an appropriate benchmark can also be added as a field. The manager can use the table to discuss the data or convert it to a graph to show trends over time visually. (See Table 1.)
Refining the measurement process
Between data collection and refining the process, there's the huge world of performance assessment. Often, it isn't until a nursing leadership group first sits down to look at performance data and understand what they mean and how areas compare that issues with measurement and data collection emerge. This creates strong motivation to improve the clarity and consistency of the data collection process so that performance assessment is valid (reflects actual performance).
In our turnover case, a nurse leader was attempting to show how a planned reduction in staffing had the additional effect of destabilizing the workforce and increasing turnover rates. The first iteration of results showed only turnover information (based on a head count of staff) and didn't address the staffing budget for the organization, which was expressed in full-time equivalents (FTEs). Some reviewers thought that the department was overstaffed because the onboard count exceeded the budgeted FTE.
The nurse leader first had to address confusion about the two ways to measure staffing levels and then revise the display of findings to show that the budgeted staffing for the organization had been reduced from fiscal year 2012 to 2013. This helped tell the story that the budgeted reduction had an impact on staff turnover as the department then dropped below its new budgeted level and experienced a jump in turnover rates. Benchmark comparisons also changed and nursing in this organization went from comparing favorably to its peer group of hospitals to lagging behind.
Data that speak volumes
As your organization accumulates valid and reliable data regarding unit- and organizational-level performance, the conversation begins to shift from “Why isn't my area reflected in the data?” to “What can we do to improve this process or share these best practices?” This is when you know that you have measures that matter.