A hospital is a complex organization that utilizes a multitude of professional personnel and resources in the undertaking of various medical therapeutic services. The establishment of a performance management system in the hospital; therefore, it is more challenging when compared with other industries. Therefore, various indicators should be in place and are necessary for the evaluation of a hospital's ongoing performance.1 For appropriate use in an organization such as a hospital, these indicators must also be categorically or purposely systemized.
After its publication in the Harvard Business Review in 1992 by Kaplan and Norton,2 the concept of the balanced scorecard (BSC) was adopted by several business enterprises around 1996. Considering its purposes and business environment flexibility, BSC was considered to be a useful management tool.3 Many articles have further highlighted the balanced scorecard, and authors have described the establishment and incorporation of the balanced scorecard into different management environments4–7 or analyzed factors for successful BSC integration.8–10
The BSC is a tool for performance management and performance evaluation. Although some chose to demonstrate BSC with numbers and spider diagrams, its function is to deliver reminders for various situations in a continuing manner.11 A well-designed scorecard should include a warning system for the success or failure of a target goal.12 Some research has examined the use of color-coding as a warning reminder. This would help to clearly define deviations of the performance evaluation indictor values from the set range or target.13 Meaningful warning levels must be designed into the information system of the BSC. Formulae are calculated based on predefined targets. Monthly comparisons are performed between the actual data and the set target value, and performance is anticipated to improve through consolidation of accurate data.
The purpose of setting warning levels is for the immediate demonstration of either good performance, which exceeds the target value, or inadequate performance, which is lower than the target value after data comparison. Therefore, in addition to a warning system, a well-designed scorecard should also have a method for visualization of the warning. Some research has described using a “signal light” method in the BSC (such as green, red, and yellow lights) as reminders for improvement in different situations. Management personnel could then detect areas for improvement more rapidly and counter with a responsive program of action.14 Previous researchers have also considered using black or red on the instrument panel of the BSC to represent numbers.15 Others have suggested that during demanding time periods, the BCS could have an “alarm clock” function to remind departments of the most important problems that require timely resolution.16
2.1. Data sources
The case hospital is a public medical center containing approximately 3000 beds with more than 6000 employees. In 2001, this institution began to conduct research and simulations for the integration of performance management systems into the medical department using the BSC information system. The target value and warning value for individual goals in each unit was stored in the system. For units where performance levels generated a “red light” or values exceeded warning values, an automated email reminder system notified the department and requested a response containing specific approaches for improvement.
Different types of indicators were recorded by the BSC. Each indicator represented a different factor and was visualized using different formats. The standardization process was essential to enable the comparison of these indicators on the same scale. The case hospital used the yearly average for each indicator for the previous year as the BSC indicator target values. For management consolidation, the system converted the score to a light signal display. When scores dropped to below 60, performance was lower than even the acceptable lowest value. This was an indicator for urgent improvement and reviews, which was accompanied by a red light signal. The user feedback mechanism established by this system distributed e-mail notifications to each department head on a monthly basis for red light indicators. A written reply with proposed improvement approaches was also requested.
In the past, hospital management professionals were mainly concerned with recommendations on the design and successful integration of the balanced scorecard. However, the effect of BSC on the hospital organization after implementation was rarely described. The limited research that did exist that investigated implementation effects were chiefly in the form of subjective questionnaires, and this research did not provide verification of actual captured data from real cases.
This research thesis hopes to utilize the experimental data produced by the incorporation and implementation of the BSC in hospitals over the last decade to investigate the effects of the BSC red light tracking warning system on performance improvement. Through this investigation, the factors that influence improvement of hospital performance, and the use of triggering indicators to enhance that performance, can be better understood.
The data used for analysis re secondary data derived from the hospital's implementation of the BSC between 2004 and 2010. It is repeated measurements data that could be treated as two-level data. Each medical department's monthly index value was set as the first-level, and the 67 first-line medical departments of the hospital were set as the second level. Considering that the number of physicians in different medical departments varied considerably, the number of physicians in each medical department was used as a covariate in our regression.
This study defined four dimensions of a total of nine key performance indicators, including controllable cost (CC), medical records completion rate (MRCR), rate of prompt consultation (RPC), average length of stay (ALS), occupancy rate (OR), bed turnover rate (BTR), infection rate (IR), unscheduled readmission rates (URR), and hospitalized accident rate (HAR). The operational definitions of these indicators are in Table 1.
3. Study design
This research was designed to be a retrospective follow-up study. The dependent variables were the various indicators, and they were continuing in nature. The data set was made up of two-level data because of the necessity for repeated measurements, which could show significant intracorrelation in each department. The linear mixed model (LMM) was applied for correcting the correlated errors. The unit's size is Level 2 variable, where other independent variables are time-varying and were set at Level 1 variable, and the departmental unit was considered as a grouping variable. Because we were interesting in the between-group differences of effect of red light warning, the red light warning was a random effect, and the other variables were fixed effect. The detailed definitions can also be seen in Raudenbush and Bryk.17
The red light warning was a binary variable. To most effectively observe the red light warning response time for each indicator, three different kinds of red light warnings were included: 1 month prior, 3 months prior, and 6 months prior.
From April 2005 to December 2008, the red light warning system at the case hospital was followed-up and maintained every month. But after 2009, management policies of the hospital changed. The red light warning system was intact, but no follow-up actions were performed. For this reason, the time period with follow-up management was assigned 1 and the rest of the time assigned 0. In addition, the correlation between the red light warning and its follow-up period was included for testing as to whether the follow-up could enhance the effect of the red light warning.
The department unit size was determined by a factor score of the number of attending physicians, the number of outpatients, and the number of days patients were hospitalized. Factor analysis was used to extract the score: the greater the factor score, the larger the departmental unit. Because the effect of red light warning could vary in part depending on the size of the department, the interaction term was built into the analysis.
For variable control, each month was used as the time variable to represent the effect of a long-term trend. The basis month was January 2004, which was assigned 0. Considering the effects of the Chinese Lunar New Year and closing accounts, at the end of a year, we included two dummy variables to denote February and December. Lastly, indicator values were influenced by prior periods; if the prior values were high, then the value for the subsequent period would also increase. Therefore, the indicator values for the period before the prior period were also included as a control variable. SPSS 19.0 was the statistical package used for analysis, especially the Mixed syntax in IBM SPSS advanced statistics v.19.0 was used to estimate LMM coefficients.
3.1. Data processing and statistical analysis
Table 2 shows a summary of each indicator. Only CC was a counting index; the rest are ratio indices. Therefore, CC results were directly influenced by the number of days as well as unit size. The average and units of measurement were significantly different between the indicators. Therefore, it was necessary to divide the standard deviation by the average to obtain a coefficient of variation (CV) to compare differences on the same scale. The administration indicator showed the least variation and was the most stable. In particular, there was no distinctive change in the RPC. The second lowest variation was with the admission performance indicator. The OR was particularly stable. By comparison, the highest variation was observed in the quality of health care indicator, which was also the least stable. It is likely that this was as result of the uniqueness of the admission quality indicator. It monitored rare events with a low rate of occurrence, where a small number of cases would cause significant differences between the samples. Thus, the relative variation became elevated.
The red light warning rate represents the ratio of red light warnings out of all the samples. The upper threshold is a value of weight, which was set in the beginning when this information system was set up. The weight of positive indicators must be smaller than 1, and for negative indicators greater than 1. Adjustments could then be made in accordance with the characteristics of each indicator. The average indicator values for the departmental unit in the previous year were multiplied by the weight value to obtain the red light threshold. If values were lower than this threshold (higher than the threshold for negative indicators), then the red light warning was issued.
It was observed that the rate of red light warning for the admission performance indicator was low. This meant that admission performance was largely above standard and it also meant that the threshold value for this indicator was appropriately defined. More than one-third of the samples in URR were flagged with a red light, indicating that the threshold criterion was too strict.
The LMM of each indication is shown in Table 3. Intraclass correlation (ICC) was the percent of between-group variance to the total variance under the null hypothesis, which can assess the appropriateness of a hierarchical linear model. The between-group variance of the nine indicators was significantly greater than 0. This demonstrated that there were definite differences between medical departments in the indicator values, and that LMM would be required. Contrasted with ICC, entering an independent variable decreases the percentage of variance between medical department considerably for all indicators except RPC. In other words, the final model reduced the between-department effect by 5%∼68%.
The month of the year that was being examined was also considered to be a variable. February had reduced CC, MRCR, and OR, and elevated BTR and URR. For the month of December, the most significantly influencing indicator was CC. Furthermore, ALS was shorter and the BTR was also higher in December.
After controlling for other variables, it was found that the indicator affected most by prior values was OR, then IR. The ALS and URR indicators scored lower in their continuous indicator values, with RPC having no significance.
After red light warning occurred, improvements were observed for 5 indicators, namely CC, MRCR, OR, BTR, and HAR within the subsequent first month. The first two indicators, CC and MRCR, immediately improved in the month after the warning, but they regressed after 3 months. The latter three indicators, OR, BTR, and HAR, were better maintained, and improvements were retained over a longer period. However, the indicator RPC presented unique circumstances; it actually worsened after a red light warning, a seemingly negative effect. The ALS, IR, and URR indicators were not significantly different from both before and after the red light warning occurred.
After controlling for the influence of the red light warning and other variables, it was observed that the CC and URR were elevated, and the MRCR was reduced during the follow-up management period. Other indicators were not affected significantly. However, during follow-up, the red light system was more effective in the improvement of CC, IR, and MRCR. This suggests that follow-up management created a supportive effect to the red light warning, and ongoing efforts to remedy noted problems. CC and IR showed this supportive effect the month after the red light warning. The last, MRCR, only showed this effect in the third month.
Different medical unit sizes only showed significant variation in CC. Larger medical units demonstrated higher CC. But the department size can also weaken the effect of the red light warning for CC and MRCR, and enhance the effect for ALS. Regarding MRCR, the weak/enhanced effect was apparent in the month following a red light warning, a phenomenon that was also observed for CC and ALS after 3 months.
To address the random effect, there are five indicators that were associated with the significant differences observed in between-department effects for the 1-month past warning timetable. Among those five indicators, IR and MRCR also displayed a difference of effect at the 3- and 6-months post-warning timetable, respectively. We also observed the random effect of intercept, and the average values between departments were significantly different on all indicators. However, this did explain the applicability of LMM.
The month of February had fewer days in total; therefore, a reduction in CC was noted. Furthermore, February is generally the month that the Chinese New Year is celebrated. Thus, patients are typically less willing to be admitted during this period, which leads to lower OR, and increased BTR and URR. Additionally, the atmosphere over holiday also led to a lower MRCR rate.
As a result of the year-end settling of accounts, CC was significantly higher in December than in any other month. Moreover, patients were unwilling to be admitted over New Years Eve, which meant that the ALS was shorter and the BTR was higher.
Prior indicator values could be implied sustainability indicators. If the regression coefficient is between 0 and 1, it is defined as a convergent sequence. As the value approaches 1, it was synchronous with prior indicator values, which means the value was sustained. The OR and IR are more stable factors, while the ALS, URR, and HAR indicators are more susceptible to the patient's condition and the impact of unexpected events. This demonstrated that other influential independent variables have not been included in the model of ALS, URR, and HAR, and resulted in the indicator's present value deviating from the previous value. Theoretically, the variation of RPC was low, and should have a elevated indicator of sustainability. However, the variation of RPC was so marginal that intercept claimed all variability, resulting in insignificant prior indicator values.
The red light warning system demonstrated manifest effectiveness in the areas of finance, administration, and admission performance. However, its efficacy as a warning device for quality of health care type indicators was less noteworthy. Red light warnings did have short-term effects that were apparent in the month following red light warning. However, for indicators such as CC and MRCR, these improvements were mere productivity bursts, and could not be maintained. Regression in these indicators was observed in less than three months time. Interestingly, the improvements in admission performance were able to be maintained for a longer duration of time. This observation was consistent with the positive influence exerted by management intervention on quality of health care that was described in previous research.18 The red light signal, supplemented by a consistent follow-up management method, was effective in the improvement of hospital performance.
Follow-up management can enhance the effect of the red light warning on CC, MRCR, and IR. The feature of these three indicators that made them more susceptible to management intervention was the fact that they are essentially manager-dependent, while other indicators more rely on patient cooperation and compliance.
RPC showed a negative effect of after red light warning occurred. The average RPC was 96.04%, and the variation in this figure was small. It would appear that RPC was customarily maintained at a high level, and the addition of a red light warning could not capture the variability.
A portion of the variation seen for CC can be attributed to the size of the medical unit. Other indicators were not strongly influenced by size. However, the red light warning resulted in greater improvement by smaller units in their CC and MRCR. With larger units, measurable benefits were observed relating to ALS. This demonstrated that smaller medical units were faster to react to administrative indicators and more flexible with financial indicators compared with larger medical units. But, in terms of admission performance, larger medical units were better at making necessary adjustments. This is logical in that more patients and more bed mean that there are more resources to rearrange. The red light warning effect for quality of health care was similar, regardless of the size of the medical unit.
In conclusion, our observations and analysis suggest that red light warning follow-up management is an immediate and important regulating tool that is appropriate in the control of finance or administration in a hospital environment, where improvement depends upon consistent attention. The study results also showed that red light warnings have a direct benefit for admission performance. Prior research has shown that the overall effectiveness of a hospital is demonstrated by its quality of health care,19 and quality of health care requires continued improvement over a longer period of time. The monthly red light warning system provides a reminder for situations where improvement is needed, and also enables continued surveillance to watch for and monitor improvements that have already occurred.
The hospital's information system could actually be utilized to provide multifaceted assistance to the hospital manager. Currently, the use of decision support systems (DSS) and decoding related groups (DRGs) proved that DRGs-DSS could assist the coding personnel in increasing coding quality and decreasing training time.20 The balanced scorecard is an effective tool in the management of performance, and could also be regarded as an early warning system.21 The red light warning provides additional warning for indicators that suggest underperformance is taking place. The case hospital believed initially that the design would be beneficial and improve its varied hospital systems. The current situation has also proven that such warning systems do indeed have uses, and could improve hospital-related indicators.
1. Wan TTH., 1995. Analysis and evaluation of health care systems: an integrated approach to managerial decision making, Health Professions Press, Baltimore.
2. Kaplan RS, Norton D. The balanced scorecard: measures that drive performance. Harv Bus Rev
3. Bible L, Kerr S, Zanini M. The balanced scorecard: here and back. Management Accounting Quarterly
4. Wachtel TL, Hartford CE, Hughes JA. Building a balanced scorecard for a burn center. Bur
5. Marr TJ, Mullen DM. Balanced scorecards for specialists: a tool for quality improvement. Minnesota Med
6. Wells R, Weiner B. Using the balanced scorecard to characterize benefits of integration in the safety net. Health Serv Manage Res
7. Goalen S, Soparlo L, Teare G. Whole system measures of quality. Paper presented at: 2007 Saskatchewan Association of Health Organizations (SAHO) Conference, 22 April 2007, Saskatoon, SK.
8. Macdonald M. Using the balanced scorecard to align strategy and performance in long-term care. Healthc Manage Forum
9. Pieper SK. Reading the right signals: how to strategically manage with scorecards. Healthc Exec
10. Chen XY, Yamauchi K, Kato K, Nishimura A. Using the balanced scorecard to measure Chinese and Japanese hospital performance. Int J Health Care Qual Assur
11. Vos MF, Schoemaker H., 2005. Accountability communication management: a balanced scorecard for communication quality, Purdue University Press, Indiana.
12. Mountain States Group, and Department of Health & Human Services Program Support Center (U.S.)., 2006. Balanced Scorecards for Small Rural Hospitals: Concept Overview & Implementation Guidance, HRSA Information Center, Merrifield, VA.
13. Bible L, Kerr S, Zanini M. The balanced scorecard: here and back: from its beginnings as a performance measurement tool. Management Accounting Quarterly 2006 June 22.
14. Stewart WE. Balanced scorecard for projects. Project Manage J
15. Ritter M. The use of a balanced scorecard in the strategic management of corporate communications. Corporate Communications
16. Rivenbark WC, Peterson EJ. A balanced approach to implementing the balanced scorecard. Pop Gov. 2008;31-37.
17. Raudenbush SW, Bryk AS. 2002. Hierarchical linear models: applications and data analysis methods, 2nd ed. Sage, Thousand Oaks, California; London.
18. Shortell SM, Becker S, Neuhauser D., 1976. The effect of management practices on hospital efficiency. In: Shortell SM, Brown M, editors., Organizational research in hospitals. Blue Cross Association, Chicago, IL, pp. 90-107.
19. Ruchlin HS. Problems in measuring hospital institutional productivity. Top Health Care Financ
20. Agudo LF, Sanjuán IM, Magallón MV. Mutual fund management in real time, by the use of the balanced scorecard. II JORNADAS DE INNOVACIÓN DOCENTE, TECNOLOGÍAS DE LA INFORMACIÓN Y DE LA COMUNICACIÓN E INVESTIGACIÓN EDUCATIVA EN LA UNIVERSIDAD DE ZARAGOZA 2008. Zaragoza, Spain, 2008.
21. Lee MT, Su ZY, Hou YH, Liao HC, Lian JD. A decision support system for diagnosis related groups coding. Expert Syst Appl