The number of operating rooms (ORs) for which block time should be planned for each surgical subspecialty is appropriately a long-term organizational process balancing cost and strategic objectives.1 However, choosing the hours for each workday that each OR should be staffed is a decision that should be based on actual usage.1–3 It should not be a discretionary or subjective procedure but should arise from quantitative analyses that balance the direct costs of unnecessary underutilized OR time with the direct and indirect/intangible expenses of overutilized OR time (see Background section for definitions).2–5
The statistical process of choosing the number of hours of staffing to plan for each OR each day is a classic newsvendor problem (explained below in Background section).4,5 Consequently, understanding the newsvendor problem is critical to OR management at hospitals.3–5 In this review article, we give examples of poor decisions made by OR managers, show the relationship between these OR management decisions and the newsvendor problem, and describe experimental studies of the newsvendor problem whose results may explain the behavior of OR managers. We conclude by demonstrating the necessity of computerized decision-support systems (DSSs). The questions that we answer are listed in Table 1.
Figure 1 is the cover art used by Anesthesia & Analgesia for a review article on how to calculate appropriate OR staffing.3 The picture on the right nicely summarizes the appropriate use of the newsvendor model to plan staffing of ORs on a given day. The picture on the left shows the observed behavior of OR managers while they unknowingly misapply the newsvendor principles. Managers consistently underestimate the duration of the workday for individual specialties and try to force the hours of cases into time slots that are too brief.2,3,6–11 For example, suppose that on Mondays the 67th percentile of workload for ophthalmology has been 11 hours over the past 3 months. Workload is expected to be the same for the next 3 months. Still, staffing is planned on Mondays for 10 hours, because the “block” is 10 hours long. Epidemiological studies show that many hospitals would not plan staffing for 11 hours, even though the staff will be working an 11-hour day.3,6–11 Even if hospital contracts limit shifts to a maximum of 10 hours, staff could sign up in advance to work late (i.e., to be on call). On most Mondays, there is a rush to find volunteers to stay late. If staffing were planned correctly, this problem should arise only on 33% of Mondays. Nevertheless, it occurs on most Mondays. This situation is irrational, yet managers consistently create conditions that allow it to occur. We hear concerns from nursing directors that their nursing staff does not want to work the 11-hour shifts of Figure 1, but these preferences are irrelevant because the nurses end up working 11 hours anyway. Nursing directors respond to incentives arising from interpersonal relationships and are highly sensitive to social aspects associated with adjusting staffing.12 However, the extent to which those social aspects influence their staffing decisions is unknown. Abouleish et al.7 estimated that the average cost to an anesthesia group resulting from ORs not being allocated to maximize OR efficiency was $1.6 million annually (in 2003 dollars).
Staffing for weekends is another example of suboptimal probabilistic staffing decisions by OR managers. Methods have been developed to ensure that nurse anesthetists on call on weekends are scheduled to be on call for as few hours as possible, while maintaining a specified level of risk that the number of staff will be insufficient to provide prompt patient care.6,13–15 Surgical suites at 6 hospitals were analyzed.13 Four were understaffed and the other 2 had more on-call hours than necessary based on the risk that the managers themselves considered appropriate. The reasons that managers persistently make poor staffing decisions are unknown.
In another example, a gynecology team in the United Kingdom was allocated 8-hour sessions for their list of cases.2 During 54 consecutive lists of cases, the team averaged 64 ± 8 minutes of overutilized time. Even though 10 hours of staffing should be planned for 8-hour sessions if the average hours of cases exceed 8 hours 40 minutes,2 only 8 hours of staffing were planned for each session.
Although robust statistical methods for determining how OR staffing decisions should be made have been extensively developed,1–8,16–21 less is known22–24 about the psychological processes that influence managers to do such a poor job of matching staffing to actual workload. Their actions clearly demonstrate that they are not applying the mathematics of the newsvendor problem to the task of deciding how much OR time to allocate (analogous to how much product to order) to match expected demand (analogous to the expected amount of product that will be sold).
Over the past decade, researchers in behavioral operations have been exploring the psychology and mathematics of the newsvendor problem in laboratory experiments.25–36 They use volunteers, usually paid economics or business school students who typically have studied newsvendor problems in class, to decide how much of a hypothetical product to order under circumstances of uncertain demand. Information is provided on cost and selling price, and participants are either told the demand distribution or given examples of historical demands. After a participant makes an ordering decision, the actual demand is revealed. The ordering decision is then compared with the correct choice based on the actual demand. After seeing the results of their decisions, participants then proceed through another round of ordering to determine whether they were influenced by prior results and if learning occurred through numerous iterations.
In this article, we review these psychological studies and critically evaluate their potential relevance to OR management. These experiments last 15 minutes to 1 hour, not the 15 months typical of multiple staffing decisions made for the OR. Nonetheless, the basic science psychological studies involve >1000 participants, each making up to 100 decisions (Table 2), and provide results uncannily similar to the behavior observed in OR management situations. These studies therefore provide mechanistic insights into OR decision-making behavior. Furthermore, the students face none of the organizational pressures encountered by OR managers. Consequently, the experimental results isolate the psychological factors from other more sociological and cultural issues that influence decisions. The studies generate recommendations that can be used to mitigate human biases that influence decision making (Table 1).
In the classic newsvendor problem,37 managers must choose how much product to order in the face of uncertain demand. The product is purchased at a cost c and sold at a price p. Profit for each item is p − c. However, the product has a limited lifespan (e.g., live Christmas trees, newspapers, or OR time). Unsold product has little or no salvage value (e.g., unused OR time is worthless at the end of the day, just like today's newspaper being sold at a newsstand tomorrow). The underage cost per item resulting from choosing to order too little product is p − c, the profit that was not realized. The overage cost per item of choosing too much product is c, the cost of the excess product purchased but unsold.
The optimal amount to order equals the 100 × r/(1 + r) percentile of the probability distribution of demand, where the critical ratio r = (p − c)/c, the ratio of the profit for each item sold to the cost of each item purchased. For example, if c = $3 and p = $9, then r = 2 and the 67th percentile of forecasted demand is the optimal order quantity. If the ratio is 2:1, you order r/(1 + r) = 2/(1 + 2) or 2/3 of the probability distribution of demand for the product. The percentile also equals 100 × (underage cost/[underage cost + overage cost]) = 100 × (p − c)/p.
If the price p far exceeds the cost c (high-profit product), then a rational manager would order far more than the mean expected demand of the product. If the price p is barely more than the cost c (low-profit product), then a rational manager would order far less than the mean expected demand of the product to avoid having a large quantity of excess inventory. Table 3, which is divided into 3 sections, provides sample newsvendor problems in which a manager engages in 10 rounds of ordering both a low-profit and a high-profit item.
The newsvendor problem can also be framed in a slightly different fashion. Suppose that you have a product to ship. The product is packaged into boxes that come in only 3 sizes (analogous to the scheduled length of the OR day). You contract to ship a certain number of boxes in 3 months (analogous to OR time to be staffed). The contracted boxes are shipped via ground at the lowest cost, but you pay to ship all your boxes even if you have fewer boxes than planned. If you underestimate the demand for your product, you must send the difference via air and pay a premium (analogous to overutilized time). The overage cost is the price of shipping by ground. The underage cost is the price of shipping by air, twice the price of ground shipping. The optimal amount of capacity to reserve in advance for ground shipping is the 67th percentile of demand, where 67% = 100 × (underage cost/[underage cost + overage cost]).
Strum et al.4,5 showed the analogy of the newsvendor problem to the planning of OR staffing. The mathematics is similar to that of shipping boxes, and was subsequently expanded by Olivares et al.22 An OR manager must decide how much OR time q to book for a series of surgical cases before the cases have even been scheduled. Typically, the decision is made 2 to 3 months before the day of surgery, before staff scheduling. The durations of the procedures are unknown, as are the number of cases and the procedures themselves. Nevertheless, the probability distribution of times to complete the workload in the OR (i.e., the demand) is known from historical OR information system data, anesthesia billing data, or anesthesia information management system data.3,38 Demand tends to follow either a normal or Weibull distribution.2,4,5 A Weibull distribution is similar to a normal distribution but can be skewed to the left or right, so that the mean differs from the median.2,4,5,39 The overage cost for labor is proportional to the underutilized OR time. If the cases run longer than q, there is a penalty for the costs of overtime wages paid to staff and the intangible costs incurred when anesthesia providers have to stay later than planned. The underage cost is r per unit of overutilized time. The optimal amount of OR time to book is the r/(1 + r) quantile of the probability distribution of the OR workload (i.e., demand). Typically, r is considered to equal 2. The recognition that underage relates to the more expensive overutilized OR time, despite the fact that there is not an analogy based on p and c, was one of the major contributions of Strum et al.4,5
Routine application of the newsvendor problem relies on the following definitions adapted from a 2006 review article by McIntosh et al.3:
OR workload for a service on a weekday refers to its total hours of cases including turnover times. OR workload is the equivalent of demand, which is a random variable that typically follows a normal or Weibull distribution.2,4,5
Underutilized OR time = (allocated [staffed] OR time) − (OR workload), or zero if this value is negative.3–5 Underutilized OR time equals the allocated OR time minus the OR workload, provided the allocated OR time is larger than the OR workload. Otherwise, the underutilized OR time is 0 hours. Underutilized time is the hours for which staffing had been planned but the OR sat idle. Underutilized time is equivalent to an overage in which too much product has been ordered.4,5
For example, staffing for the ophthalmology OR is planned for 10 hours. Last Monday, ophthalmology had 9 hours of cases including turnovers. There was 1 hour of underutilized OR time.
Overutilized OR time = (OR workload) − (allocated [staffed] OR time), or zero if this value is negative.3–5 Overutilized time is the hours that staff worked beyond the hours for which staffing had been planned. Overutilized time is equivalent to an underage in which demand exceeds the amount of product ordered.4,5
For example, staffing for the ophthalmology OR is planned for 10 hours. Two Mondays ago, the ophthalmologists performed 12.5 hours of cases because of a late-running vitrectomy. They had 2.5 hours of overutilized OR time.
OR efficiency is maximized when the inefficiency of use of OR time is minimized. The inefficiency of use of OR time = (cost per hour of underutilized time × hours of underutilized time) + (cost per hour of overutilized time × hours of overutilized time). OR efficiency is essentially profitability. If demand (OR workload) cannot be controlled, the goal on the day of surgery should be to reduce the hours of overutilized time.3,9,11,16–18,23
Suppose that each 1 hour of overutilized OR time costs 2 times as much as each 1 hour of underutilized OR time (i.e., r = 2). Then, staffing for an OR should be planned for the 67th percentile of expected workload so that the day finishes early 2/3 of the time. Multiple extensions to this formulation have been made for purposes of OR management decision making, such as scheduling cases, releasing allocated OR time, scheduling add-on cases, and reducing turnover times.3,16–21
We consider results from several behavioral operations studies of the newsvendor problem, explain common psychological errors, and then comment on the potential significance of these studies to OR management.
Schweitzer and Cachon25 published a landmark article that established the protocol frequently used for conducting experiments that examine participant decisions in the newsvendor problem. MBA students in an operations management course were provided complete information on the cost c and selling price p of an imaginary product and the number of rounds that would be played.25 Demand was uniformly distributed, meaning that all demands within the specified range were equally probable and that demand was constant from one day to the next. This uniform distribution resulted in decision making that was far simpler than OR managers would face in practice. Both high-profit and low-profit conditions were created. The corresponding optimal ordering choices were the 75th and 25th percentiles of demand, respectively. Before each round and before making a decision, participants could view tables and graphs with essentially all combinations of the data.
On average, the first inventory order of participants who faced the high-profit condition was significantly higher than the first inventory order of participants who faced the low-profit condition, indicating some insight into the proper quantity to order. Nevertheless, a consistent too low/too high pattern of ordering persisted. Orders for high-profit products were consistently lower than the expected profit-maximizing quantities. Orders for low-profit products were consistently too high. These results match the tendency of OR managers to choose staffing that is biased in the direction of the mean demand for OR time. The important finding is that the students did not choose the proper quantity of goods to order even when demand followed a uniform distribution. Decisions would be even more complicated for OR managers, because they do not face simple uniform distributions but more complex normal or Weibull distributions.2,4,5
In another set of experiments,27 both undergraduate and executive MBA students had the option of ordering a high-profit stock or a low-profit stock, with demands adjusted so that the optimal order was the same for both conditions. Students ordered too little of the high-profit stock and too much of the low-profit stock. Furthermore, the behavior of many students was consistent with the fallacy that demand in the next round would correlate either positively (“I'm on a lucky streak”) or negatively (“My number is due”) with demand in the previous round.40 An analogy in OR management to a positive correlation would be to give more OR time to a surgeon whose underutilized OR time has averaged 3 hours per workday over the past 3 months,2,41 but who has been busy for the most recent 2 weeks. We see this happen routinely in OR committees and hear it attributed to politics and pleasing the surgeons. In the experimental studies, however, nearly the same phenomenon is observed even though no political factors are present. Providing participants with feedback about the payoffs associated with options not taken as well as those taken did not significantly improve profitability.27 Thus, we should expect OR managers to have difficulty making good decisions, when MBA students cannot make good decisions in the absence of interruptions, organizational lobbying, or personnel to appease.
Undergraduate economics students participated in a similar experiment in which demand was uniformly distributed.29 Again, this probability distribution is far simpler than the normal or Weibull distributions observed in practice. Optimal order quantities were the 25th, 50th, or 75th percentile of demand. In OR management, the optimal quantity would be about the 67th percentile.2–8 The initial order quantity of the economics students averaged the 50th percentile of demand, regardless of the optimal order quantity. As above, the students ordered too little when they should have ordered more than the 50th percentile and too much when they should have ordered less than the 50th percentile.
Other groups of undergraduate business students ordered a quantity close to the mean demand.28,34,35 Orders for high-profit products were too low and orders for low-profit products were too high. Of 250 MBA students, 44% ordered a quantity equal to the mean value of expected demand.33
Ordering was compared among undergraduates, graduate business students, and managers experienced in newsvendor-type ordering decisions. The undergraduates initially ordered product at an average of the 50th percentile of demand instead of the 75th percentile.30 Although managers were significantly higher at the 64th percentile, with graduate students in between, all groups performed on average far below the optimal value of the 75th percentile.
In a single-round newsvendor experiment with MBA students interested in operations management,31 each student had to ask for pertinent information, such as cost, price, and demand distribution. Although almost half the participants correctly identified the overage and underage costs, the students did not act on this information to generate the optimal order quantity.
Undergraduate management students who had taken a basic course in statistics26,32 were divided into groups, one with a uniform distribution and the other with a normal distribution of demand. Participants tended to increase their order if prior demand was higher than the past order. Participants tended to reduce their order if prior demand was lower than the past order. Results were similar for both uniform and normal distributions. This behavior is analogous to allocating OR time based on the workload over the past 2 weeks rather than the past 9 months.3,42
In summary, these studies show that student participants trying to solve the newsvendor problem universally make poor decisions. They tend to anchor on mean demand and give too much attention to the most recent value of demand. Both behaviors occur even when conditions are far simpler than those occurring naturally in OR management.
Several reasons may explain why decision makers order an inventory quantity that differs from the expected profit-maximizing quantity and, equivalently, why OR managers choose staffing plans that do not maximize OR efficiency. The following represent some of the many possible biases that can explain the behavior of students in the experiments and managers planning staffing for the OR.
Many behaviors arise from the theory of bounded rationality.43–45 The quality of decisions made by individuals is limited by the amount of information available to them, the cognitive limitations of the human mind in evaluating and processing the information that is available, and the limited amount of time they have to make decisions. Instead of being rational and reaching the optimal solution, people tend to simplify the choices available to them.
- Anchoring with insufficient adjustment: Decision makers anchor on mean demand and adjust suboptimally toward the optimal demand. This results in the low/high pattern of ordering and the pull-to-center effect described above.
- Demand-chasing and recency bias40: Decision makers anchor on a prior order quantity determined by previous demand and adjust suboptimally toward the optimal order quantity. If the quantity ordered was less than the optimal quantity, a stockout or underage likely occurred. Recency bias would then cause order quantities to increase. In other words, if the OR ran late one day causing overutilized time, recency effects would tend to cause the OR manager planning staffing for the next 3 months to plan additional staffing, even if unnecessary. Likewise, if the OR ended early on the day staffing was planned for the next 3 months, staffing might be planned below optimal levels.
- Reinforcement and inertia: Decision makers respond only to the effects of the actual decision made and ignore all other alternatives. With high reinforcement bias, a decision that yields a positive payoff is more likely to be used again regardless of the profitability of other decisions, a characteristic that can generate inertia.
Many biases reflect the risk preferences of the individual:
- Waste aversion: Too little product is ordered because the decision maker does not want any left over, which would result in an overage. Waste aversion is essentially an intentional overestimate of the cost of an overage. For example, even though ORs should end early on approximately 67% of days, a biased OR manager would not want to plan staffing for longer than the scheduled length of the workday. The manager therefore plans too little staffing, resulting in unnecessary overutilized OR time.
- Stockout aversion: Too much product is ordered because the decision maker does not want to run out, which would result in an underage. Thus, the OR manager plans too long a day for fear of overutilized OR time (e.g., overtime). Stockout aversion is essentially an intentional overestimate of the cost of an underage.
- Myopic loss aversion: The decision maker may be more averse to losses than attracted to the same-sized gains.46 Losses have a greater psychological impact than gains.47–49
Many people are guided by a common belief that is incorrect:
- Gambler's fallacy: A bettor who has not won during previous rounds may believe that his or her number is due and that it is their turn to win. He or she may expect the odds of winning to increase if there has not been a recent win. The results of the next round are negatively correlated with those of the previous round.
The preceding lists show that there are many potential causes of poor decision making. Causes of poor decision making that apply in a specific situation can be identified using experimental studies in which participants are given different conditions that are insightful mechanistically. Those conditions may not occur naturally outside of the laboratory. Just as for a normal distribution and all other symmetric distributions, the 50th percentile of demand for a uniform distribution would be optimal if overutilized time costs the same as underutilized time. Because overutilized OR time costs twice as much as underutilized OR time, then the OR should finish early 67% of the time. In OR management, correct answers are typically around the 67th percentile. However, the selling price of hypothetical goods can be chosen experimentally so that the optimal percentile is the 25th.25,29 Student participants choose quantities that significantly exceed the 25th percentile but are less than the 50th percentile. This result shows that participants anchor their responses to the mean.
The use of experiments to understand behavior is important because observational studies are inadequate. We do not know of a situation in OR management in which staffing should or would be planned for the 25th percentile of demand. This situation might occur if anesthesiologists received bonus pay for hours worked after 4 PM and then deliberately created long turnovers and delays in the middle of the day to push the cases into bonus hours. The total hours of cases would then be longer than the planned hours of staffing. The anesthesiologists would have deliberately manipulated the demand for the OR to make it appear greater than it actually was. However, we showed that such behavior does not occur in practice,50 and so experiments are needed to create the situation artificially.
The experimental studies cited above show that 2 of the listed biases are common: anchoring on mean demand and recency effects. Recency effects refer to the tendency of participants to chase demand, believing that the next round will correlate with the previous one, even when each round is independent. As above, this is analogous to the OR committee that ignores a statistical analysis with 9 months of data in lieu of a surgeon's usage during the past 2 weeks.3,42 These findings are important because decisions made by OR managers that result in insufficient staffing are often attributed to politics, personalities, organizational inertia, etc. However, study participants isolated from such issues by ordering hypothetical products in laboratory experiments committed the same errors as OR managers. Because the study participants did not obtain the optimal percentile under controlled experimental conditions, it is not surprising that OR managers at hospitals lacking (or ignoring) DSSs for staffing analyses plan shorter durations of the workday than optimal. The experimental studies show that the behavior reflects innate psychological biases. Furthermore, the biases occur under conditions of demand that follow a uniform distribution, far simpler than the normal distribution or Weibull distribution of demand seen in ORs.2,4,5
Additional experiments help us understand the extent to which education can influence these biases, or if instead computerized DSSs must be used to make good decisions (Table 1).
Frequency of Feedback
We previously studied how much data to use when reports of appropriate staffing are calculated for OR managers.3,42 In our study of OR staff planning using different amounts of data, each increase in the number of months of data up to 9 months resulted in a statistically significant reduction in expected labor costs.3,42 There were large incremental benefits in using at least 7 months of data.3,42 However, we have not been able to design a practical experimental study to evaluate how often those reports should be provided.
Our experience is that some hospitals choose to have analyses performed every 3 to 4 months, before staff scheduling is performed. However, an important question is whether receiving reports more often, such as once a month, would improve decision making by learning, even when staffing is changed every 3 to 4 months. On the one hand, monthly reports might be good because they would prevent managers from forgetting important principles in between report periods. On the other hand, more frequent feedback may degrade performance if it leads decision makers to focus on the most recent data and to overreact to random noise and variability in workload.28
In 1 experiment, participants were restricted to ordering a fixed quantity that would be applied to the next 10 demand rounds.27 Participants made a total of 100 decisions. Before each decision, they were given the expected profit and range of profits associated with each of 3 order options. Performance was enhanced when decisions were made less frequently, especially for a low-profit stock, with achievement of approximately 90% of the profit potential. Negative correlations between rounds and anchoring on mean demand were reduced when decisions were made every 10 rounds instead of every round. These results suggest that staffing decisions can be made too frequently with adverse consequences. Limiting OR management decisions to every 3 to 4 months may be best.
Undergraduate business students participated in an experiment of 30 rounds in which the optimal order quantity was the 75th percentile of demand.28 The students received feedback after every round, every 3 rounds, or every 6 rounds, and thus made 30 decisions, 10 decisions, or 5 decisions, respectively.28 When the variance in market demand was small, feedback frequency was of little importance. However, when the variability was high, frequent feedback actually resulted in poorer decisions.
In another experiment with undergraduates in which the variability in demand was equivalent to the small variability of the previous study, feedback was provided and decisions were made every 5 rounds to reduce recency effects.29 The pull-to-center effect was stronger (i.e., decisions were worse) than when decisions were made every round.
Investigators have examined why feedback is not beneficial in experiments involving undergraduate business students. Data labels were viewable on a computer screen, but the data were revealed only when the participant positioned the computer mouse cursor over a box.28 The information was visible until the cursor was moved out of the box. The students who received feedback every 5 rounds (1) looked at more of the available information after each decision than those who received feedback every 1 round, (2) acquired information from more rounds (average of the most recent 5.7 rounds vs most recent 2 rounds), (3) spent less time accessing information about their previous decisions (8% vs 14% of the time), and (4) spent more time accessing information about the cumulative effects of their decisions (40% vs 25%).28 The students receiving feedback every 1 round consistently purchased too little product (i.e., planned too little OR staffing) compared with that which would maximize their profit (i.e., the efficiency of use of OR time).28 Thus, even when students could rely on all previous rounds of data, they focused instead on their most recent decisions. More frequent feedback worsens or does not improve decisions because of recency effects and chasing of demand. We know of no reason to think that OR managers would be less sensitive to these psychological biases than undergraduate business students.
Taken together, these studies show that more frequent feedback results in no improvement in performance (small coefficient of variation) or worse performance (large coefficient of variation). Consequently, providing OR managers with reports that are more frequent than those needed for decisions is unlikely to be of value (Table 1). The evidence is especially convincing if one considers the expense of analyst preparation of a report and the time needed for a manager to read and discuss it. Limiting analyses to those occasions when decisions should be made is likely adequate.
While doing our work in creating decision-support tools and reports (e.g., as shown in Ref. 3), we have previously considered the value of showing historical OR workload in a graphical format. For example, we thought about plotting workload for each day of the week (vertical axis) over months (horizontal axis) and including a horizontal limit line to indicate recommended staffing levels.3 We also considered providing a moveable cursor that the OR manager(s) could use to evaluate the inefficiency of use of OR time under alternative decisions. The relevant experimental study27 showed that there is no sound scientific reason to expect that such a tool would be beneficial. Instead, the tool would merely occupy the time of OR managers who would have to learn how to use it and analysts who would have to generate interactive reports rather than simple PDF files. When given a choice, analysts prefer methods that require the least effort, even if the quickest method results in poorer-quality decisions.51 Decision makers often use subjective judgments rather than statistical methods52 even though the latter are more accurate. The problem with poor decision making in newsvendor problems is not how the data are displayed but that humans are inherently unable to interpret the reports and make good staffing decisions because of psychological biases.
Value of Education
Given so many psychological biases, it is important to determine whether education can enhance the quality of staffing decisions made by OR managers (Table 1). Laboratory experiments provide some insight into whether decision making improves with appropriate feedback and learning. Overall, the results below show that, even for students who are studying the newsvendor problem in classes and have near perfect conditions for learning from their previous decisions, performance improves very slowly if at all with multiple iterations.
In 1 experiment, participants made 30 rounds of decisions. Feedback was provided after each round and student participants had the opportunity to learn from experience. Nonetheless, average order quantities did not adjust toward the expected profit-maximizing quantity across rounds.25 Performance did not improve.
Undergraduate and executive MBA students chose orders from among 3 options: the 35th, 50th, or 75th percentile.27 After each round, students were provided feedback on the profit (inefficiency of use of OR time) that they would have achieved had they made each of the other 2 decisions.27 Feedback resulted in little improvement in performance after 30 rounds, even when participants were given the moving average of the results of other decisions. Slight learning was documented after 100 rounds. Importantly, notice how trivially simple these decisions are compared with those that OR managers are expected to make, and still learning is weak.
Undergraduate business students received feedback after every round, every 3 rounds, or every 6 rounds.28 Performance did improve over time and was greatest when feedback was given every round. However, even after 100 rounds, the quantity ordered was markedly suboptimal, midway between the 50th percentile and the optimal order quantity of the 75th percentile.
Learning does sometimes occur over the course of experiments. Over 100 rounds, 77% of management students moved further away from the mean ordering quantity and closer to the optimal quantity.32 In another study, undergraduate management students showed greater profit in the last 20 rounds than the first 20 rounds and made smaller changes in order quantities with each round in the second half of the experiment.26 Undergraduate economic students participated in a similar experiment.29 They learned over 30 rounds, especially for the low-cost high-profit product for which the 75th percentile was the optimal quantity. Conditions thus resembled the OR management situation in which the 67th percentile was the optimal choice.
For freshman undergraduate business students, graduate business students, and working managers, experience did improve performance. When additional demand information was provided, students increased their profits. Managers, however, did not use the additional analytical information but continued to rely on historical data.30 In a variation of the protocol, each category of participants was divided into 2 groups. One group received basic information as in the other experimental studies. The other group received a 60-minute video lecture about the newsvendor problem and how demand influences profit (i.e., principles analogous to those about the inefficiency of use of OR time). The 1-hour video training significantly improved initial performance but did not enhance learning after repeated rounds beyond that expected from supplying information on demand and expected profit.30
Although the studies above show that the degree of learning is heterogeneous among experiments, the consistent finding is that when learning occurred, it was weak. Learning was weak even though the students received feedback every minute, not every 3 months as would OR managers. Demand distribution was uniform, not normally distributed. All necessary information to make the decisions was provided. Politics and personalities were not relevant. Students were not interrupted with other tasks. They had to perform the equivalent of planning staffing for 1 service, not the 50 or so combinations of service and day of the week that are typical of the OR environment.3–8 Given that learning seems to be of little to no benefit under ideal conditions, we perceive no reason to expect OR managers to learn from experience under realistic conditions.
Computerized DSSs are needed to generate staffing recommendations. Future research is needed to evaluate the influence of classroom teaching on trust in the automatic DSS reports. In other words, we do not know from the preceding studies whether training would enhance the likelihood that OR managers would follow the computerized recommendations even when faced with organizational pressures to make contrary decisions. We also do not know the extent to which the content and format of the display may have an important effect on trust (see Discussion section).
Limited Order Quantities
In OR management, we cannot plan 10.31 hours of staffing but must choose staffing options that are combinations of routine staff scheduling paradigms (e.g., 8-, 10-, or 12-hour shifts). A study has investigated the impact of this limitation in terms of its ability to improve decision making.27 Limited order options reduce the decision set from which experimental participants can choose, thereby reducing the cognitive processing requirements.
Undergraduate students with complete distribution and cost knowledge averaged ordering at the 61st percentile instead of the optimal 75th percentile over 100 rounds.27 Results were indistinguishable when order options were reduced to 9 choices or 3 choices (35th, 50th, and 75th percentiles). Performance did not improve. This result was a consequence of the psychological biases influencing decisions, including anchoring on mean demand and the gambler's fallacy.27
Thus, we have no reason to expect that the limited options for planning staffing result in either more or less psychological bias (Table 1).
Heterogeneity in Decision Making
When Stepaniak et al.11 studied scheduling coordinators at an OR control desk who filled gaps in the daily OR schedule, they found heterogeneity in decision making because 2 of the 4 coordinators at the hospital were psychologically risk averse and 2 were not. Similar heterogeneity may exist between OR managers who make newsvendor decisions, resulting in large differences in staffing.
Student participants were classified into nonoverlapping clusters based on their ordering decisions.27 Only 30% of participants chose the correct ordering quantity, with a mode of the 75th percentile. Another 30% had a mode of the 50th percentile (i.e., anchored on mean demand). Approximately 25% of students tended to increase orders when overages (underutilized OR time) were observed in the preceding round. Approximately 10% tended to increase orders when underages (overutilized OR time) were observed from the preceding round. The latter 2 groups chased demand, even though demand was statistically independent between rounds (e.g., focusing on the idea of either a hot hand or the gambler's fallacy that my number is due). Approximately 5% had a distribution of choices over the 100 rounds that was not different from random.
In another experiment, the correct percentile of 75% was chosen by only 19% of MBA students.31 Mean demand was selected by 23%. Undergraduate business students chose optimal ordering quantities only 13% of the time.34
In other experiments involving both students and experienced managers, up to 65% of participants chose order quantities that were >20% away from the mean order quantity for the group.27
Professional buyers tended to order a quantity that was less than the mean demand, whereas student participants tended to anchor on mean demand.33 The buyers' behavior was consistent with an objective to maximize the probability of reaching some target goal, not necessarily to maximize profit. Different OR managers may have differing goals.
In an experiment to test managers' abilities to obtain relevant information, MBA students31 who had previously learned about the newsvendor model and expressed a specific interest in operations were told that their task was to choose the appropriate ordering quantity but were not given additional information. Over 15 minutes, they could ask questions that would be answered before they had to make their ordering decisions. Each student had to ask for pertinent information, such as cost, price, and demand distribution. Only 3 of the 20 participants asked necessary questions, knew how to make the decision, and got the correct answer. There was no correlation between the number of questions asked or the time taken by participants to make decisions and the quality of their decisions. They had problems with the abstractness of the task. Most of the participants failed to identify key information that should have affected their decisions. The most common information sought was the previous decision. Most participants asked about decisions made in the past, decisions made by the competitors, decisions made by vendors, etc. Thus, although all participants had learned the science, they wanted to know what other people did, even though such information was irrelevant for determining the optimal order quantity.
The relevance for OR management is partly that large heterogeneity in decisions among individuals should be expected. However, surgical suites typically have only 1 or 2 decision makers for the hours of staffing to plan for each OR. Thus, relying on the performance of those individuals may be counterproductive because participants chosen from a random pool would have only a 20% to 30% chance of good performance. In addition, it is likely that the managers will fail to ask the correct questions and will not use the information they do have available to make the best possible decisions. They are subject to numerous psychological biases that influence their decisions.
The only way that a facility can determine whether its managers are performing well is to undertake the expense of calculating the appropriate OR staffing using computerized methods. Because computerized DSSs are necessary to check the managers, the facility might as well use the staffing plan it calculates instead of one created by the managers. Furthermore, the staffing plan should be included in the anesthesia group's contract with the facility.12,40
Types of Providers and Prior Knowledge
Perhaps 1 drawback of these experimental studies is that they involved students with little or no experience. A manager with experience in OR management might excel, make better decisions, and be more amenable to education (Table 1).
Three groups of participants were compared: freshman undergraduate business students who had not had a course in operations management, graduate business students with at least 1 undergraduate course in operations management, and working managers whose jobs revolved around newsvendor decision making.30 Demand had a uniform probability distribution. The optimal order quantity was the 75th percentile. Before participants made their first decisions, they were shown the demand for 50 earlier rounds and were told the mean. They were not shown a histogram or told that the probability distribution was uniform. After 40 rounds, the participants were told that the probability distribution was uniform. After another 40 rounds, the participants were given a graph showing that the 75th percentile was the choice that maximized profit (the inefficiency of use of OR time). Although the participants could have deciphered the optimal value from the information of the preceding phase of the study, the objective was to compare different groups of participants when they were not explicitly provided with the demand distribution.
The results reinforce the findings that participants are unable to translate even complete knowledge of the probability distribution and relative cost values into an optimal order quantity. However, when the optimal quantity was provided, most people used it.30 Ironically, the managers had significantly worse performance than the students in 1 phase of the experiment because they were the least likely to use the correct answer when it was provided.
Importance of the Decision
One difference between OR managers and the students used for the experimental studies is that the students had little at stake (e.g., $20 payment), whereas the job performance of OR managers is evaluated based on the work schedules they create. Two sets of experiments did evaluate probabilistic choices made by student participants when they had a chance of “winning” a relatively large sum of money. Decision processes were fundamentally different depending on whether choices were made over the domain of gains or losses, with participants being more risk averse over the domain of losses.53,54 In the domain of losses, decisions were not based on actual probabilities but on vague categories of “beliefs” that they assessed to the certainty of specific events. Marked heterogeneity existed among participants.
OR managers often make poor decisions when planning staffing.6–11,13 A decade ago, there was no reasonable way that they could make good decisions if they wanted to, because the science was not developed. However, that is no longer the situation. The mathematics is understood. Multiple review articles have been published (Fig. 1). The intriguing question is why statistical methods are used only some of the time. Why is it that if a surgeon uses >8 hours of OR time on nearly 100% of Mondays, many managers would still plan 8 hours of staffing every Monday? In our experience, people at hospitals routinely say “it is politics.” However, no evidence supports this conclusion.11,12,18,23,24,50 Politics does not explain this behavior because the staff works late, regardless of the decision. Similarly, poor decisions are made by managers on the day of surgery and when choosing the arrival and fasting times of patients.18,55
An important issue is the incentives of the OR manager to make good decisions. The manager may be charged with reducing variability in costs, thereby deliberately overstaffing to eliminate unexpected overtime. Alternatively, the manager may be directed to minimize budgeted costs through understaffing, with overtime seen as a factor out of his or her control. These pressures may serve to reinforce or counteract innate biases that influence decision making. The political ramifications of staffing decisions cannot be ignored. However, the addition of politics to the results simply strengthens the inescapable conclusion that computerized DSSs are needed for implementation of altered staffing levels.
Laboratory experiments show that people make poor decisions even when politics is not a factor. The problem is psychological biases, not organizational behavior. Although students with better memories and greater cognitive abilities made risky decisions that were closer to expected values than other students, biases still affected decisions.35 Decisions were not made to maximize profits.33 The experimental studies teach us that optimal decisions are often not made even when sophisticated mathematics are not needed. Thus, changing how the mathematics are displayed or taught is very unlikely to improve decisions.
The experimental studies are not perfectly analogous to situations faced by OR managers. However, they are close and have some advantages in that the students can be given conditions that do not exist in practice for OR managers. Furthermore, the studies of undergraduates generally have sample sizes of scores25,26,28–32 or hundreds27,28,33–35 of participants and use coefficients of variations that are very large to inflate the effect size (Table 2). These large sample sizes and parameter choices are necessary because of the heterogeneity of behavior that results in large SDs in the results. Unlike managerial decisions on the day of surgery that are made by many clinicians (e.g., whoever is on-call),18,23,50,56 there simply are not enough OR managers making staffing decisions to study at organizations that are comparable with each other. Thus, most likely, OR management will need to rely on the basic science studies using students to understand the reasons for the biases and how to compensate for them. At a minimum, we must first identify conditions from experimental studies under which managers seem to make good decisions and only then test the quality of their decisions in practice. In the absence of such studies, we must assume that use of computerized DSSs is necessary. We already know that the recommendations are optimal.3–5
Other differences exist between the experimental studies and OR management. Rounds are shorter, lasting minutes rather than months. The number of rounds is much greater than would occur in years of OR decision making. Opportunities for learning are thus much greater in the experimental studies, although little learning occurs. Other factors that may influence staffing decisions in real life are absent from the experimental studies. For example, OR managers are faced with many other pressures, such as staff not wanting to have their work hours changed, OR committees being inactive because the committee head is on vacation for 2 weeks, lack of availability of analyst time because of end of the fiscal year budget work, or concerns of how decisions might be perceived by surgeons. All would seem to suggest that if students cannot take advantage of learning and education, and make mistakes when offered free food and cash in return for good decisions, then OR managers are even far less likely to make good decisions. What we learn is that inherent psychological biases and anchoring on mean demand, as observed in the experimental studies, are sufficient to explain the poor decisions made by OR managers. Organizational or sociological pressures need not be invoked. They may just make poor decisions even more likely, especially when the penalty for poor decisions is minimal.12,40
A weakness of the experimental studies for product purchases is the assumption of complete knowledge of demand. Although untrue for product purchasing, this assumption is realistic for OR management. Most overutilized OR time is caused by cases that start in what would otherwise be underutilized OR time.16,57 In addition, surgery is nonpreemptive (i.e., once started, it cannot be abandoned to complete another task and finished later). Therefore, we have a pure newsvendor problem and (essentially) perfect knowledge of demand. Our review is thus highly relevant and the newsvendor problem is particularly appropriate to the OR. The finding that managers make poor newsvendor decisions can be extrapolated to the OR.
An important unknown is factors that influence OR managers' trust in computerized DSSs and their willingness to follow the recommendations of DSSs.58–61 Users who have access to DSSs often fail to utilize them if the recommendations are sometimes incorrect.62–64 Subjective impressions of the expertise and capabilities of human alternatives have a large effect on the trust placed in DSSs.65 Trust is enhanced when the quality of the automated decisions is high and when the user understands the capabilities of the DSS and the algorithms used to generate the output.59,66 For computerized DSSs that plan staffing for the OR, the accuracy and validity of the computer recommendations are not in doubt. A DSS can make better decisions than a human. Because trust was enhanced through greater understanding of the DSS,66 education may be an important component in convincing OR managers to place reliance on DSSs. Although research has also shown that a positive mood increases reliance on computerized DSSs,67 the relevance of this finding to OR managers seems unclear. Trust in DSSs needs to be a focus of educational research.
Computerized reports cannot completely replace OR managers. We must remember the importance of differentiating between 2 distinct roles played by OR managers in planning staffing. As a first step, managers must review previous OR workloads and use their expert judgment to improve forecasts of OR workload for the future (e.g., based on their knowledge of changes in local competition).68 Likely, they perform this step very well. In our experience, managers often know the trends in OR workload. Committees can be particularly insightful.34 Committees tend to have qualitative information about changes in surgical groups and that information is important. Computerized forecasts will not make this role obsolete. The step that should be automated is the conversion from the expected OR workload to the appropriate staffing plan. Over the past decade, the science has been developed to make this decision well. Managers are poor at solving the newsvendor problem. They need to rely on DSSs.
REW and FD helped to design the study, write the manuscript, and perform a literature review.
Franklin Dexter is the Associate Editor for Statistics and section Editor for Economics, Education, and Policy for the Journal. The manuscript was handled by Steve Shafer, Editor-in-Chief, and Dr. Dexter was not involved in any way with the editorial process or decision.
The University of Iowa, Department of Anesthesia, performs statistical analyses for anesthesia groups and hospitals. REW and FD receive no funds personally other than their salaries from the State of Iowa. They have tenure with no incentive program tied to contracting entities and declare no financial conflicts of interest.
The authors thank Polly Biyu He for helpful comments on the manuscript.
1. Wachtel RE, Dexter F. Tactical increases in operating room block time for capacity planning should not be based on utilization. Anesth Analg 2008;106:215–26
2. Pandit JJ, Dexter F. Lack of sensitivity of staffing for 8 hour sessions to standard deviation in daily actual hours of operating room time used for surgeons with long queues. Anesth Analg 2009;108:1910–5
3. McIntosh C, Dexter F, Epstein RH. Impact of service-specific staffing, case scheduling, turnovers, and first-case starts on anesthesia group and operating room productivity: tutorial using data from an Australian hospital. Anesth Analg 2006;103:1499–516
4. Strum DP, Vargas LG, May JH, Bashein G. Surgical suite utilization and capacity planning: a minimal cost analysis model. J Med Syst 1997;21:209–22
5. Strum DP, Vargas LG, May JH. Surgical subspecialty block utilization and capacity planning. A minimal cost analysis model. Anesthesiology 1999;90:1176–85
6. Dexter F, Epstein RH, Marsh HM. Statistical analysis of weekday operating room anesthesia group staffing at nine independently managed surgical suites. Anesth Analg 2001;92:1493–8
7. Abouleish AE, Dexter F, Epstein RH, Lubarsky DA, Whitten CW, Prough DS. Labor costs incurred by anesthesiology groups because of operating rooms not being allocated and cases not being scheduled to maximize operating room efficiency. Anesth Analg 2003;96:1109–13
8. Freytag S, Dexter F, Epstein RH, Kugler C, Schnettler R. Allocating and scheduling operating room time based on maximizing operating room efficiency at a German university hospital. Chirurg 2005;76:71–9
9. Dexter F, Weih LS, Gustafson RK, Stegura LF, Oldenkamp MJ, Wachtel RE. Observational study of operating room times for knee and hip replacement surgery at nine US community hospitals. Health Care Manage Sci 2006;9:325–39
10. Van Houdenhoven M, Hans EW, Klein J, Wullink G, Kazemier G. A norm utilisation for scarce hospital resources: evidence from operating rooms in a Dutch university hospital. J Med Syst 2007;31:231–6
11. Stepaniak PS, Mannaerts GHH, de Quelerij M, de Vries G. The effect of the operating room coordinator's risk appreciation on operating room efficiency. Anesth Analg 2009;108:1249–56
12. Masursky D, Dexter F, Nussmeier NA. Operating room nursing directors' influence on anesthesia group operating room productivity. Anesth Analg 2008;107:1989–96
13. Dexter F, Epstein RH, Marsh HM. Costs and risks of weekend anesthesia staffing at six independently managed surgical suites. AANA J 2002;70:377–81
14. Dexter F, O'Neill L. Weekend operating room on-call staffing requirements. AORN J 2001;74:666–71
15. Dexter F, Epstein RH. Holiday and weekend operating room on-call staffing requirements. Anesth Analg 2006;103:1494–8
16. Dexter F, Traub RD. How to schedule elective surgical cases into specific operating rooms to maximize the efficiency of use of operating room time. Anesth Analg 2002;94:933–42
17. Dexter F, Epstein RD, Traub RD, Xiao Y. Making management decisions on the day of surgery based on operating room efficiency and patient waiting times. Anesthesiology 2004;101: 1444–53
18. Dexter F, Willemsen-Dunlap A, Lee JD. Operating room managerial decision-making on the day of surgery with and without computer recommendations and status displays. Anesth Analg 2007;105:419–29
19. Dexter F, Traub RD, Macario A. How to release allocated operating room time to increase efficiency. Predicting which surgical service will have the most under-utilized operating room time. Anesth Analg 2003;96:507–12
20. Dexter F, Macario A. When to release allocated operating room time to increase operating room efficiency. Anesth Analg 2004;98:758–62
21. Dexter F, Abouleish AE, Epstein RH, Whitten CW, Lubarsky DA. Use of operating room information system data to predict the impact of reducing turnover times on staffing costs. Anesth Analg 2003;97:1119–26
22. Olivares M, Terwiesch C, Cassorla L. Structural estimation of the newsvendor model: an application to reserving operating room time. Manage Sci 2008;54:41–55
23. Dexter F, Lee JD, Dow AJ, Lubarsky DA. A psychological basis for anesthesiologists' operating room managerial decision-making on the day of surgery. Anesth Analg 2007;105:430–4
24. Dexter EU, Dexter F, Masursky D, Garver MP, Nussmeier NA. Both bias and lack of knowledge influence organizational focus on first case of the day starts. Anesth Analg 2009;108:1257–61
25. Schweitzer ME, Cachon GP. Decision bias in the newsvendor problem with a known demand distribution: experimental evidence. Manage Sci 2000;46:404–20
26. Benzion U, Cohen Y, Peled R, Shavit T. Decision-making and the newsvendor problem: an experimental study. J Oper Res Soc 2008;59:1281–7
27. Bolton GE, Katok E. Learning by doing in the newsvendor problem: a laboratory investigation of the role of experience and feedback. Manuf Serv Oper Manag 2008;10:519–38
28. Lurie NH, Swaminathan JM. Is timely information always better? The effect of feedback frequency on decision making. Organ Behav Hum Decis Process 2009;108:315–29
29. Bostian AJA, Holt CA, Smith AM. Newsvendor “pull-to-center” effect: adaptive learning in a laboratory experiment. Manuf Serv Oper Manag 2008;10:590–608
30. Bolton GE, Ockenfels A, Thonemann U. Managers and students as newsvendors—how out-of-task experience matters. University of Cologne Working Paper Series in Economics 2009;39:1–23. Available at: http://ockenfels.uni-koeln.de/RePEc/download/wp0039.pdf
. Accessed January 15, 2010.
32. Benzion U, Cohen Y, Peled R, Shavit T. Decision-making and the newsvendor problem—an experimental study. J Oper Res Soc 2008;59:1281–7
33. Brown AO, Tang CS. The impact of alternative performance measures on single-period inventory policy. J Ind Manag Optim 2006;2:297–318
34. Gavirneni S, Xia Y. Anchor selection and group dynamics in newsvendor decisions—a note. Decis Anal 2009;6:87–97
36. Cokely ET, Kelley CM. Cognitive abilities and superior decision making under risk: a protocol analysis and process model evaluation. Judgm Decis Mak 2009;1:20–33
37. Whitin TM. Inventory control and price theory. Manage Sci 1955;2:61–80
38. Dexter F, Epstein RH. Optimizing second shift OR staffing. AORN J 2003;77:825–30
39. Dexter F, Bayman EO, Epstein RH. Statistical modeling of average and variability of time to extubation for meta-analysis comparing desflurane to sevoflurane. Anesth Analg 2010;110:570–80
40. Tversky A, Kahneman D. Judgment under uncertainty: heuristics and biases. Science 1974;185:1124–31
41. Dexter F, Epstein RH. Calculating institutional support that benefits both the anesthesia group and hospital. Anesth Analg 2008;106:544–53
42. Epstein RH, Dexter F. Statistical power analysis to estimate how many months of data are required to identify operating room staffing solutions to reduce labor costs and increase productivity. Anesth Analg 2002;94:640–3
43. Simon HA. A behavioral model of rational choice. Q J Econ 1955;69:99–118
44. Kahneman D. Maps of bounded rationality: psychology for behavioral economics. Am Econ Rev 2003;93:1449–75
45. Su X. Bounded rationality in newsvendor models. Manuf Serv Oper Manag 2008;10:566–89
46. Wang CX, Webster S. The loss-averse newsvendor problem. Omega 2009;37:93–105
47. Gneezy U, Kapteyn A, Potters J. Evaluation periods and asset prices in a market experiment. J Finance 2003;58:821–37
48. Haigh MS, List JA. Do professional traders exhibit myopic loss aversion? An experimental analysis. J Finance 2005;60:523–34
49. Thaler RH, Tversky A, Kahneman D, Schwartz A. The effect of myopia and loss aversion on risk taking: an experimental test. Q J Econ 1997;112:647–61
50. Masursky D, Dexter F, Garver MP, Nussmeier NA. Incentive payments to academic anesthesiologists for late afternoon work did not influence turnover times. Anesth Analg 2009;108:1622–6
51. Chenowith T, Dowling KL, St. Louis RD. Convincing DSS users that complex models are worth the effort. Decis Support Syst 2004;37:71–82
52. Remus W, O'Connor M, Griggs K. Does feedback improve the accuracy of recurrent judgmental forecasts? Organ Behav Hum Decis Process 1996;66:22–30
53. Cohen M, Jaffray JY, Said T. Individual behavior under risk and under uncertainty: an experimental study. Theory Decis 1985;18:203–28
54. Cohen M, Jaffray JY, Said T. Experimental comparison of individual behavior under risk and under uncertainty for gains and for losses. Organ Behav Hum Decis Process 1987;39:1–22
55. Dexter F, Xiao Y, Dow AJ, Strader MM, Ho D, Wachtel RE. Coordination of appointments for anesthesia care outside of operating rooms using an enterprise-wide scheduling system. Anesth Analg 2007;105:1701–10
56. Ledolter J, Dexter F, Wachtel RE. Control chart monitoring of the numbers of cases waiting when anesthesiologists do not bring in members of call team. Anesth Analg 2010 [Epub ahead of print]
57. Dexter F, Macario A, Traub RD. Which algorithm for scheduling add-on elective cases maximizes operating room utilization? Use of bin packing algorithms and fuzzy constraints in operating room management. Anesthesiology 1999;91:1491–500
58. Parasuraman R, Riley V. Humans and automation: use, misuse, disuse, abuse. Hum Factors 1997;39:230–53
59. Seong Y, Bisantz AM. The impact of cognitive feedback on judgment performance and trust with decision aids. Int J Ind Ergon 2008;38:608–25
60. Vashitz G, Meyer J, Parmet Y, Peleg R, Goldfarb D, Porath A, Gilutz H. Defining and measuring physicians' responses to clinical reminders. J Biomed Inform 2009;42:317–26
61. Lee JD, See KA. Trust in automation: designing for appropriate reliance. Hum Factors 2004;46:50–80
62. Fransoo JC, Wiers VCS. An empirical investigation of the neglect of MRP information by production planners. Prod Plann Control 2008;19:781–7
63. Kelton K, Fleischmann KR, Wallace WA. Trust in digital information. J Am Soc Inf Sci Tech 2008;59:363–74
64. Xin L, Hess TJ, Valacich JS. Why do we trust new technology? A study of initial trust formation with organizational information systems. J Strat Inform Syst 2008;17:39–71
65. Madhavan P, Wiegmann DA. Effects of information source, pedigree, and reliability on operator interaction with decision support systems. Hum Factors 2007;49:773–85
66. Cramer H, Evers V, Ramlal S, van Someren S, Rutledge L, Stash N, Aroyo L, Wielinga B. The effects of transparency on trust in and acceptance of a content-based art recommender. User Model User-Adap Interact 2008;18:455–96
67. Djamasbi S. Does positive affect influence the effective usage of a Decision Support System? Decis Support Syst 2007;43:1707–17
68. Masursky D, Dexter F, O'Leary CE, Applegeet C, Nussmeier NA. Long-term forecasting of anesthesia workload in operating rooms from changes in a hospital's local population can be inaccurate. Anesth Analg 2008;106:1223–31