Among the 2265 combinations of 755 dates and three end points, there were 12 with externally studentized residuals19 larger than three. We repeated all calculations excluding these 12 outliers as one of our sensitivity analyses.
As another sensitivity analysis, we repeated the regression analyses using a different type of regression, namely regression trees with a least absolute values loss function (Systat 12). Regression trees are automatically created for predicting relationships including interactions. We used least absolute values as the loss function so that the mean absolute error could be compared directly with that of the preceding classical multivariable linear regression. To compute a regression tree for each of the three dependent variables, and to maximize the potential predictive ability of the model, we used all independent variables in Table 1 plus all possible sums and differences of those variables. Jackknife estimation was used to calculate the standard error of the reduction in the absolute error.
As a final sensitivity analysis, we repeated the linear regression analyses using a different dependent variable, specifically, the time after which there were 16 or fewer ORs in use. We studied 16 ORs because, each day, there are eight anesthesiologists who are assigned to work after 4 pm, if necessary, and each typically medically directs two ORs. We used this secondary end point of the earliest times when there were ≤16 ORs to explore our hypothesis that if the independent variables were poor predictors of the three primary dependent variables, the cause of poor prediction would be that they better predict when many anesthesiologists are working. If most of the add-on cases are performed by the three on-call anesthesiologists and most of the elective (scheduled) cases are performed by the other anesthesiologists, then the numbers and hours of elective cases would poorly predict the hours worked by the three anesthesiologists who are on-call.
Survey of Anesthesiologists
Although we obtained data on three dependent variables, for the survey of anesthesiologists we considered just the smallest (time when ≤2 ORs) and largest (time when ≤6 ORs), thereby reducing the number of questions (Table 4). The survey was performed at the study hospital during a 2-wk period. One of the anesthesia clinical directors asked 32 of 42 anesthesiologists to participate, with 10 missed due to their being on vacation, at a meeting, or working outside the main ORs. All 32 anesthesiologists invited completed the survey but with a few missing values. The two anesthesiologists who were aware of the survey design were excluded. The survey was performed by those two anesthesiologists at the study hospital as a quality improvement project to help them decide whether to post the historical data and to assist in their explaining to the department how the information can be used. The results in this article were implemented at the hospital the week after the survey was completed. The statistical analysis was comparison of observed proportions to ½ (StatXact-8, Cytel Software, Cambridge, MA).
The study hospital has call team members leaving when the numbers of ORs in use are ≤6 ORs, ≤4 ORs, and ≤2 ORs. The three thresholds studied were the times when these numbers of ORs were always in use. For the three thresholds and two end points of mean and 95% upper confidence limit of the 80th percentile, differences among days of the week were as large as 45 min (Table 1). Differences between end points for the same weekdays were as large as 245 min. Comparatively little additional knowledge was available in the late afternoon on the working day before surgery, as the mean absolute error was reduced by only 4.1–6.0 min (Table 3, Fig. 3).† Information available more days before the day of surgery (e.g., 1 wk) would have had even less incremental predictive value.
The mean absolute error of anesthesiologists’ estimates for 80th percentiles was 60 min, principally because of underestimation of the 80th percentiles (Table 5). As hypothesized, more than half (69%, P = 0.0003) of anesthesiologists’ estimates for 80th percentiles had error >30 min, whereas errors of this magnitude were less for the mean (44%, P = 0.0004).
Months in advance, anesthesia providers at hospitals frequently choose and trade shifts and make other arrangements for their call days. Anesthesiologists’ intuitions were inaccurate as to the hours that they work (Table 5). Any anesthesia group that bills for its time or tracks its cases has all of the data required for the forecasts, just as they do for specialty-specific staffing.1,2,16 The numbers of simultaneous anesthetics (i.e., ORs in use) at each time can be calculated from the anesthesia start and end times. The results show essentially negligible value to the use of data within 1 day of surgery (e.g., number or hours of cases actually scheduled) other than historical data for the particular day of the week from an anesthesia billing,1 OR information, or anesthesia information management system. The results match previous findings that if months ahead the durations of OR workdays are forecasted appropriately for each specialty, there is financially negligible incremental value for reducing anesthesia group costs by using additional information to even perfectly predict case durations.16,20–23
Although the incremental value of additional data was strongly statistically significant (i.e., P < 10−6) (Table 2), the absolute reductions in forecasting errors were small relative to the choice of the statistic to report (Table 3, Fig. 3). Furthermore, these absolute reductions likely overestimate the actual value of using such information in decisions. The issue is one of trust in technology. When a prediction for how late people work is given before staff schedules are made, by definition the estimate is retrospective and provides little information about what the individual will experience on one specific future evening. Providing the estimate is similar to a web site that gives the average January low and high temperature in Orlando, FL. For a traveler contemplating a vacation there in January, this information may be useful. In contrast, making a prediction on January 16 for the low temperature the next day runs the risk of people perceiving psychologically that “weather forecasts are unreliable.” This is because of a human bias to convert probabilities of events (40% chance of temperature <70°C) into binary perceptions (e.g., “the actual temperature was 65°C, so the meteorologist was wrong”).
Our independent variables captured information about the variables relevant to the working day before surgery. By studying data at this point in time, we likely overestimated the potential value of the information practically useful for regression analysis, because some decisions may need to be made sooner. For example, when staffing (OR allocation) and case scheduling decisions are made based on maximizing the efficiency of use of OR time, most services fill their allocated time no sooner than 3 days before the day of surgery.16,24 At that time, 81% of cases have been scheduled (Fig. 2). By studying data from the day before surgery, we likely underestimated the predictive value relative to using information from the day of surgery. However, by the day of surgery, an anesthesiologist’s desire to work or not to work late for pay is generally irrelevant as to whether the anesthesiologist works late (i.e., they are on-call and have to stay until the cases are done). We showed previously that, under such circumstances, compensation is no longer salient.25 Thus, we cannot identify an argument for why providing updated information on the day of surgery related to the predicted end of shift times would be useful.
Although different anesthesia directors running the control desk make different decisions that affect how late ORs are in use, we did not include individuals as independent variables.26 Who will be scheduled to manage the desk would not be known when an anesthesiologist is choosing his or her schedule. Furthermore, we could not envision informing a person on call that tomorrow she will likely work 30 min later because the person at the control desk is Dr. Smith not Dr. Lawrence. We think that behavioral differences of individuals running a control desk show the value of managerial decision-support systems making recommendations to reduce such variability.23,27 We think that they also show why anesthesia providers will need and/or want to continue to be paid hourly when working late, as compared with occasionally working long scheduled shifts.25
Our data analysis applies only to hospitals with add-on cases and cases being moved among ORs. If every case were known at least 1 wk in advance, there were no add-on cases, each surgeon has a list of cases for the day, and the cases are not moved among ORs, then results would be different. The survey was performed by one anesthesia group immediately before implementation. Although we have no reason to suspect that results would differ among groups, we have no supportive data.
In conclusion, a department’s survey of its anesthesiologists identified a 1 h mean absolute error in estimates for the latest (80th percentile) time that they work when on-call, with more than half (69%) of estimates incorrect by >30 min. Because the two end points of mean and 95% upper confidence limit of the 80th percentile differed substantively, different decisions should rely on different statistics. Because the two end points differed among days of the week, statistics should be reported by weekday. This information can be calculated from anesthesia billing data (or equivalent sources1) and given to anesthesia providers months ahead before staff scheduling. Additional knowledge available on the working day before surgery (e.g., numbers of scheduled cases) had a statistically significant benefit but reduced the mean absolute error by only 4.1–6.0 min extra. The results are especially useful when interpreted along with recent behavioral studies of anesthesiologists’ managerial decisions late in afternoons.25,26
1. Dexter F, Epstein RH. Optimizing second shift OR staffing. AORN J 2003;77:825–30
2. Dexter F, Epstein RH. Calculating institutional support that benefits both the anesthesia group and hospital. Anesth Analg 2008;106:544–53
3. Ernst EA, Lasdon LS, Ostrander LE, Divell SS. Anesthesiologist scheduling using a set partitioning algorithm. Comput Biomed Res 1973;6:561–9
4. Ernst EA, Matlak EW. On-line computer scheduling of anesthesiologists. Anesth Analg 1974;53:854–8
5. Holley HS, Heller F. Computerized anesthesia personnel system. Int J Clin Monit Comput 1988;5:103–10
6. Dexter F, O’Neill L. Weekend operating room on-call staffing requirements. AORN J 2001;74:666–71
7. Miller RD, Cohen NH. The impact of productivity-based incentives on faculty-based compensation. Anesth Analg 2005;101: 195–9
8. De Grano ML, Medeiros DJ, Eitel D. Accommodating individual preferences in nurse scheduling via auctions and optimization. Health Care Manag Sci. In press. doi: 10.1007/510729-008-9087-2
9. Dexter F, Macario A, Lubarsky DA, Burns DD. Statistical method to evaluate management strategies to decrease variability in operating room utilization. Application of linear statistical modeling and Monte-Carlo simulation to operating room management. Anesthesiology 1999;91:262–74
10. Moore IC, Strum DP, Vargas LG, Thomson DJ. Observations on surgical demand time series: detection and resolution of holiday variance. Anesthesiology 2008;109:408–16
11. Masursky D, Dexter F, O’Leary CE, Applegeet C, Nussmeier NA. Long-term forecasting of anesthesia workload in operating rooms from changes in a hospital’s local population can be inaccurate. Anesth Analg 2008;106:1223–31
12. Wachtel RE, Dexter F. Influence of the operating room schedule on tardiness from scheduled start times. Anesth Analg 2009; 108:1889–901
13. Staudte RG, Sheather SJ. Robust estimation and testing. New York: Wiley, 1990:104–6
14. Clopper CJ, Pearson ES. The use of confidence or fiducial limits illustrated in the case of the binomial. Biometrika 1934; 26:404–13
15. Newcombe RG. Two-sided confidence intervals for the single proportion: comparison of seven methods. Stat Med 1998;17: 857–72
16. McIntosh C, Dexter F, Epstein RH. Impact of service-specific staffing, case scheduling, turnovers, and first-case starts on anesthesia group and operating room productivity: tutorial using data from an Australian hospital. Anesth Analg 2006; 103:1499–516
17. Strum DP, Vargas LG, May JH, Bashein G. Surgical suite utilization and capacity planning: a minimal cost analysis model. J Med Syst 1997;21:209–22
18. Strum DP, Vargas LG, May JH. Surgical subspecialty block utilization and capacity planning. A minimal cost analysis model. Anesthesiology 1999;90:1176–85
19. Neter J, Wasserman W, Kutner MH. Applied linear statistical models. 3rd ed. Homewood, IL: Irwin, 1990:398–400, 450–1
20. Dexter F, Traub RD. How to schedule elective surgical cases into specific operating rooms to maximize the efficiency of use of operating room time. Anesth Analg 2002;94:933–42
21. Dexter F, Epstein RD, Traub RD, Xiao Y. Making management decisions on the day of surgery based on operating room efficiency and patient waiting times. Anesthesiology 2004; 101:1444–53
22. Dexter F, Macario A, Ledolter J. Identification of systematic under-estimation (bias) of case durations during case scheduling would not markedly reduce over-utilized operating room time. J Clin Anesth 2007;19:198–203
23. Dexter F, Epstein RH, Lee JD, Ledolter J. Automatic updating of times remaining in surgical cases using Bayesian analysis of historical case duration data and instant messaging updates from anesthesia providers. Anesth Analg 2009;108:929–40
24. Dexter F, Traub RD, Macario A. How to release allocated operating room time to increase efficiency. Predicting which surgical service will have the most under-utilized operating room time. Anesth Analg 2003;96:507–12
25. Masursky D, Dexter F, Garver MP, Nussmeier NA. Incentive payments to academic anesthesiologists for late afternoon work did not influence turnover times. Anesth Analg 2009;1622–6
26. Stepaniak PS, Mannaerts GH, de Quelerij M, de Vries G. The effect of the operating room coordinator’s risk appreciation on operating room efficiency. Anesth Analg 2009;108:1249–56
27. Dexter F, Willemsen-Dunlap A, Lee JD. Operating room managerial decision-making on the day of surgery with and without computer recommendations and status displays. Anesth Analg 2007;105:419–29
*For readers who are used to using stepwise and all variables regression, the deleted residuals are those used to calculate the prediction sum of squares.
†Using regression trees, the mean ± standard error of the reductions in the absolute error were 0.00 ± 0.00, 0.00 ± 0.00, and 0.08 ± 0.01 min for ≤2 ORs, ≤4 ORs, and ≤6 ORs, respectively.© 2009 International Anesthesia Research Society