Journal Logo

Individual QI Projects from Single Institutions

Simulation to Predict Effect of Citywide Events on Emergency Department Operations

Ferraro, Nicole M. MSc*;; Day, Theodore Eugene DSc

Author Information
doi: 10.1097/pq9.0000000000000008
  • Open



Discrete event simulation (DES) has increasingly become a useful tool in quality improvement initiatives to analyze the effects of potential changes in hospital facility planning as healthcare costs continue to rise. The emergency department (ED) is a particular aspect of healthcare that has recently been subject to rising costs and increased visits, leading to a complex environment that requires resource and capacity planning to serve all patients demanding care. DES has been used in modeling the ED in numerous scenarios to analyze many components in that environment. In particular, DES has been used to manage and forecast overcrowding occurrences in the ED.1,2 DES has also been useful in identifying bottlenecks and resource requirements in the system that could be improved to decrease the overall patient length of stay (LOS).3 The authors based their simulation on the typical patient levels seen in the ED and, along with the other studies mentioned above, looked at the effects of a routine surge on ED resources. Still, other studies have examined the effects of catastrophic, unforeseen surges, such as those seen during natural disasters,4–6 or other large-scale casualty events, such as disease outbreaks,7 car accidents,8 or terrorist attacks.9 In particular, the simulation developed by Hirshberg et al9 was used to identify potential bottlenecks in the flow of casualties stemming from an urban terrorist attack scenario and analyze the use of essential surgical resources to aid in future hospital emergency plans for such situations.

The challenge of predicting demand for ED services has been previously described,10 as have the consequences of overcrowding on patient care and mortality.11,12 Thus, there is a need for predictive models that can allow EDs to adequately prepare for potential crowding situations, such as those that are known in advance, and will allow time for EDs to make necessary staffing and resource changes. Events that could cause anticipated crowding would include those such as large-scale sporting events, like the World Cup or the Olympics, and high-profile visits, such as the 2015 World Meeting of Families (WMF) in Philadelphia that culminated with papal appearances. This latter event brought a large influx of visitors to the city and a massive egress of residents. We present here a DES model that was developed using data from the ED at the Children’s Hospital of Philadelphia (CHOP) from July 2014 to June 2015. We then used these data to predict the effect of changes in the patient arrival stream in anticipation of the papal visit, the subsequent consequence of the number of admitted patients, and the average patient LOS.


Model Development

We developed the DES model of the ED at the CHOP according to a standardized 4-step process previously described by our group.13–15 Briefly, model development includes (1) decomposition of the system into its constituent elements (eg, locations, entities, resources, paths); (2) development of a system flow (ie, how do patients move from one location to another and consume system resources at each location); (3) implementation of these elements in a computer model using MedModel 2011 (Promodel Corp, Allentown, PA); and (4) validation of the model using face, internal, and external validation techniques. The model was developed and validated according to best practices promulgated by the Society for Medical Decision Making.16,17 The development and analysis of this model were determined not to be human subjects research by the CHOP’s Institutional Review Board, and thus exempt from oversight.

Figure 1 shows the flow of patients through the system. Patients arrive and receive an initial triage, at which point they are brought to resuscitation if necessary. If they do not require resuscitation, they are brought to an examination room if one is available, and wait in the waiting room if one is not. Once an examination room is available, a nurse and resident physician see the patient. The resident then reports on the patient to the attending physician, who then sees the patient. Any necessary laboratory tests or imaging studies are obtained and procedures performed. The process is repeated as needed. Finally, patients may be discharged home, admitted to the hospital, or admitted to the Emergency Department Extended Care Unit. Patients in the Emergency Department Extended Care Unit are observed for 23 hours and then either discharged or admitted to the hospital. Patients who do undergo resuscitation may be seen after that in an examination room, admitted to the hospital, or discharged directly.

Fig. 1.:
Simulated patient flow.

Model Validation

Face validation: the model was developed with continuous input, advice, and review from system stakeholders and demonstrated to those same individuals to verify that the model accurately captured patient flow, workflow, and the teaching model of the CHOP’s ED operations. This process entailed meetings with physician, nursing, and administrative staff to describe, run, and query the model in detail. The clinical staff reviewed the system flowcharts and verified the accuracy of the model of patient flow. Model output was determined by expert opinion to be reflective of real-world performance. Internal validation: Author Day conducted a complete code review of the model to ensure there were no discrepancies between the intent of the design and the function of the model. System stress tests were used to validate that the dynamic queues performed as designed, and pathway analysis was used to demonstrate that patient flow operated as intended. System stress tests consist of overwhelming the model with simulated arrivals that exceed capacity and observing the results. Conceptually, this stress testing is equivalent to forcing the ED to operate at 300% capacity for weeks at a time. Do simulated patients become permanently blocked? Does the simulation freeze due to competing resource demands? The simulation should perform as designed during a stress test, even if the queues increase without bound. Any complex queuing system can be overwhelmed, but for internal validity, those patients that successfully reach the front of the queue should be processed as normal—as was observed in this case. External validation: simulated patient volume, the overall LOS, and admitted LOS were compared with real-world values and verified to be within acceptable parameters to accurately emulate the real-world system (see Model Validation under Results for numerical details of external validation).

Scenario Analysis

Before the WMF, the model was used to simulate full ED operations for the period July 1, 2015–December 31, 2015. The WMF, September 21–27, was included in this interval. This simulation included the entire patient flow through the system (Fig. 1), all human resource workflows, and interactions with other hospital services (imaging, laboratory, and admission). The lead-up time and postevent time in the model functioned as “warm up” and “cool down” intervals for the event simulation. This timing ensured that the model began and ended the event period in normal operation, rather than starting from an empty ED. Thus, we were able to determine how the ED would likely respond, starting from normal operations to eleven separate scenarios. In these scenarios, the patient arrival volume varied from a 50% decrease to a 50% increase, in increments of 5% for the week of the WMF. The 0% scenario—1,575 patients per week—represented the control group. Ten statistically independent runs were generated per experiment and used as a model for changes in the LOS and number of admitted patients based on the change in patient arrivals. For each run, the number of patients admitted during the event week and the average LOS for all patients during that time were recorded. We compared simulated arrivals, admissions, and Emergency Severity Index distributions with real-world data recorded during the papal visit. The Emergency Severity Index is a classification of emergency patients from 1 (in need of resuscitation) to 5 (minor complaint), which is designed to identify the number of services a patient is likely to require, and how urgently they need attention from emergency medical services. Values between the various simulation scenarios were calculated using linear interpolation.


Model Validation

Simulated data from ten 1-year independent but statistically identical simulation runs (representing approximately 83,000 patients per year) were examined for LOS of all patients and all admitted patients. Simulated mean LOS was 226.1 ± 173.3 minutes (mean ± SD) for all patients and 352.1 ± 170.3 minutes for admitted patients. Real-world mean LOSs for the fiscal year 2014 were 230.6 ± 134.8 for all patients and 345.0 ± 147.7 for admitted patients. Thus, simulated mean LOS values deviated less than 2% from real-world measured data, indicating that no clinically significant differences between mean LOS estimations for all or admitted patients exist between the model and the real-world system as verified by medical personnel. Hourly and weekly volumes and demand were reviewed with clinical staff during face validation to ensure appropriate verisimilitude. Admission times and LOS were based on dynamic inpatient censuses to ensure accurate representation of ED to inpatient admission times. Thus, simulated and empirical data coincide sufficiently for model validity.

Scenario Analysis

The number of patients seen during the papal visit weekend totaled 425 patients, with 84 of those as admissions. This number of patients represents a 39.65% reduction compared with the usual number of patients admitted over that same time. Based on this decrease, the linear equation fit to the simulation data was used to predict the average number of patients admitted and their average LOS. We also calculated the average LOS of patients that would have been admitted except for the decreased volume. Table 1 shows the comparison between the simulated values and the real-world observations.

Table 1.:
Real-world and Simulated Data for the WMF Week

Figure 2 shows the range of estimations of LOS for all and admitted patients for each of the eleven simulated scenarios and the real-world observations. From this, it is clear that as volume increases, LOS increases approximately linearly until incoming volume exceeds 125% of normal, at which point we see a sharp increase in LOS. This observation indicates that the system is saturated and unable to respond normally beyond this point. In terms of the queuing systems, this describes a systemic instability, where queues for service increase without bound as time goes on. Knowledge of this breakpoint allows hospital emergency personnel to communicate accurately with other interested parties (eg, municipal emergency services) to triage patients to facilities most able to provide prompt care. Figure 3 shows the range of estimations of patient volume.

Fig. 2.:
LOS for simulated and real-world patients during the WMF.
Fig. 3.:
Volume for simulated and real-world patients during the WMF.


Emergency planning for large-scale events is a necessary part of hospital operations. However, estimates of ED volume during these events are notably absent in the literature. This deficiency results in a wide range of possibilities that require planning for very low to very high demand for services. Additionally, one must consider the risk of events such as communicable disease, accident, and terrorism. A DES provides an appropriate test bed for such scenario analysis. As shown above, the DES model we built provided a spectrum of estimations of incoming patient volumes. We used the event to demonstrate the predictive validity of the model we built: the LOS and number of admissions seen during the real-world event lay almost precisely on the curve estimated given the volume encountered. Predictive validity (ie, the ability of a simulation’s estimations to be verified with prospective real-world observations) is rare—but not unknown18—in the literature.

It is worth noting that the observed decrease in volume during the WMF was not a scenario of serious concern when planning. Certainly, emergency planners are concerned with sharp increases in patient volume, surges, and disasters. The observed decrease and our accurate estimation of its effects, however, provide confidence that the model is capable of accurately estimating the effects of various conditions. Thus, event planners can use the model’s estimations of volume increases with the assurance that those estimations are likely to be both accurate and precise. EDs are unlikely to reduce staffing in response to possible decreases; however, this type of analysis allows us to estimate the staff needed for an appropriate service threshold, validated against real-world performance under extraordinary conditions.

Having the estimations of admissions and LOSs based on various potential volume levels allowed the ED staff to prepare for a variety of volume scenarios and improved situational awareness regarding the likely impact on emergency services during the WMF event. This sort of predictive guidance helps institutions plan for responding to rapidly changing conditions and events that lead to changes in ED crowding. With events such as the WMF, where they are known long in advance and have uncertain effects on demand for services, predictive modeling provides both estimations of the effects of different volume levels and maximum capacities.

DES models are frequently developed in partnerships between academic and clinical personnel, using commercially available software. In this case, the development was an example of a mentored internship,19 in combination with clinical staff and faculty, and embedded DES developers using a development framework detailed in a prior publication by our team.20 These models are sometimes proprietary and modified, requiring a knowledgeable developer to tailor a simulation to each particular system to be analyzed. DES has limitations in that it analyzes a somewhat simplified system and cannot take into account every individual process or diagnosis that may occur within a real-world department. They cannot predict the real-world future. However, they are capable of providing the likely effects of targeted interventions or systemic responses to changing environments. DES models are agnostic to scenarios not modeled; thus, to provide useful information, event planners must query the model with specific scenarios of likely events.


Thoroughly designed and validated DES models are capable of providing crucial insights into systemic issues. Demonstrations of predictive validity are rare but important to encourage adoption of this useful planning and analysis tool. Specifically, the use of DES to support emergency planning and operations can provide valuable and timely insights into expected systems dynamics under potential stressful demand for services. In our case, the CHOP was able to plan for ED operations under the unknown but potentially challenging WMF.


The authors would like to thank James Callahan, MD, and Joseph Zorc, MD, of the Children’s Hospital of Philadelphia Emergency Department for their assistance in developing and validating the simulation and providing prospective scenario insight before the WMF.


The authors declare that no conflicts of interest exist with regard to the production or publication of this manuscript. This article was supported by a 2015 Implementation Science Working Group Pilot Grant from the Department of Medical Ethics and Health Policy in the Perelman School of Medicine at the University of Pennsylvania.


1. Hoot NR, LeBlanc LJ, Jones I, et al. Forecasting emergency department crowding: a discrete event simulation.. Ann Emerg Med. 2008;52(2):116125.
2. Duguay C, Chetouane F. Modeling and improving emergency department systems using discrete event simulation.. Simulation. 2007;83(4):311320.
3. Brenner S, Zeng Z, Liu Y, et al. Modeling and analysis of the emergency department at University of Kentucky Chandler Hospital using simulations.. J Emerg Nurs. 2010;36(4):303310.
4. Yi P, George SK, Paul JA, et al. Hospital capacity planning for disaster emergency management.. Socio-Econ Plan Sci. 2010;44(3):151160.
5. Paul JA, George SK, Yi P, et al. Transient modeling in simulation of hospital operations for emergency response.. Prehosp Disaster Med. 2006;21(4):223236.
6. Wu S, Shuman L, Bidanda B, et al. Agent-based discrete event simulation modeling for disaster responses.. Proceedings of the 2008 Industrial Engineering Research Conference. 200819081913.
7. Brillman JC, Burr T, Forslund D, et al. Modeling emergency department visit patterns for infectious disease complaints: results and application to disease surveillance.. BMC Med Inform Decis Mak. 2005;5(1):4.
8. Hirshberg A, Holcomb JB, Mattox KL. Hospital trauma care in multiple-casualty incidents: a critical view.. Ann Emerg Med. 2001;37(6):647652.
9. Hirshberg A, Stein M, Walden R. Surgical resource utilization in urban terrorist bombing: a computer simulation.. J Trauma. 1999;47(3):545550.
10. McCarthy ML, Zeger SL, Ding R, et al. The challenge of predicting demand for emergency department services.. Acad Emerg Med. 2008;15(4):337346.
11. Hoot NR, Aronsky D. Systematic review of emergency department crowding: causes, effects, and solutions.. Ann Emerg Med. 2008;52(2):126136.
12. Richardson DB. Increase in patient mortality at 10 days associated with emergency department overcrowding.. Med J Aust. 2006;184(5):213–216.
13. Ferraro NM, Reamer CB, Reynolds TA, et al. Capacity planning for maternal-fetal medicine using discrete event simulation.. Am J Perinatol. 2015;32(8):761770.
14. Day TE, Chi A, Rutberg M, et al. Addressing the variation of post-surgical inpatient census with computer simulation.. Pediatr Surg Int. 2014;30(4):449456.
15. Day TE, Sarawgi S, Perri A, et al. Reducing postponements of elective pediatric cardiac procedures: analysis and implementation of a discrete event simulation model.. Ann Thorac Surg. 2015;99(4):13861391.
16. Karnon J, Stahl J, Brennan A, et al. Modeling using discrete event simulation: a report of the ISPOR-SMDM Modeling Good Research Practices Task Force-4.. Med Decis Making. 2012;32(5):701711.
17. Eddy DM, Hollingworth W, Caro JJ, et al. Model transparency and validation: a report of the ISPOR-SMDM Modeling Good Research Practices Task Force–7.. Med Decis Making. 2012;32(5):733743.
18. Day TE, Al Roubaie AR, Goldlust EJ. Decreased length of stay after addition of healthcare provider in emergency department triage: a comparison between computer-simulated and real-world intervention.. Emerg Med J. 2013;30(2):134138.
19. Day TE, Goldlust EJ, True WR. The role of mentored internships for systems engineering in improving health care delivery.. Acad Med. 2010;85(9):14051407.
20. Rutberg M, Wenczel SL, Devaney J, et al. Incorporating discrete event simulation into quality improvement efforts in health care systems.. Am J Med Qual. 2015;30(1):3135.
Copyright © 2017 the Author(s). Published by Wolters Kluwer Health, Inc. All rights reserved.