Rapid and reliable methods of text communication to mobile anesthesia care providers are important to patient care and to efficient operating room (OR) management. Whether the message is an ad hoc urgent request for assistance (e.g., “Come STAT OR 3”), an automated clinical reminder generated from an anesthesia decision support system (DSS) (e.g., “Your patient in OR 2 is due for a repeat dose of cefazolin”), a notification from the OR director (e.g., “Please relieve in OR 4”), or a compliance message (e.g., “Please sign your attestation for Mary Smith on 1/14/2012”), the expectation is that the transmitted message will be received expeditiously by the recipient. Although several-minute delays are reasonable for some of these examples, the most timesensitive communications determine the maximum acceptable latency of a message delivery system.
With the maturation of anesthesia information management systems (AIMS),1 automated messaging including delivery of vital sign alerts,2,3 clinical recommendations,4,5 quality of care,6 staff assignment,7 and compliance or billing issues8–10 have been described. Understanding delays in the delivery of such messages is thus of importance to evaluate the effectiveness of these technologies.
Although some anesthesia groups are using cell phone text messaging as a means of communication, we are concerned about using this technology for critical message delivery. Cellular coverage is poor in some hospital locations for some carriers, and we do not have a locally installed cell network for in-house cell phone use. Furthermore, current cellular networks are neither designed to have sufficient bandwidth to handle unusually high peak volume, nor intended for emergency communications.a Consequently, text message failures or prolonged latencies are expected during periods of extreme network activity (e.g., natural disasters and other local or regional emergenciesb,c or nearby large gatherings of people in concentrated placesd,e). As an example, Figure 1 demonstrates 5 hours of prolonged text message latencies during a major Winter storm in Philadelphia in January 2011. In other words, had cell phone texting been relied upon, “Stat C-section OB 1” would not have been delivered in a timely fashion because it was snowing. Hospital communication needs are critical during “all-hazards” events, f so reliance on devices expected to be unreliable during inclement weather is imprudent.
Potential hazards of reliance on personal cell phones for intrahospital communication were also highlighted in the recent communication by Mukhtar et al.11 describing what happened in an Egyptian hospital when the government shut down Internet and cell phone services during the 2011 Arab Spring uprising.
Our previous work related to communication latency focused on the delivery of messages from our DSS to the AIMS workstations in each OR12 and methods of analyzing latency data with highly skewed probability distributions (e.g., 50th percentile of latency 2 seconds vs 99th percentile 2 minutes).13 In the current report, we quantify the additional latency incurred after initiation of the text message until its delivery on mobile devices. We describe methodology for evaluation of the performance of text messaging systems for implementing DSS capabilities within an AIMS. Our minimum criteria to qualify a device for critical notification were a mean latency of <30 seconds and no more than 1 in 200 pages (0.5%) having latencies exceeding 100 seconds.g
In Appendix B, we give an example of Vanderbilt University Medical Center’s (VUMC) subsequent use of the methodology to evaluate their existing ad hoc text paging processes.
The latencies of the department’s current numeric-only radiofrequency pagers (Bravo Plus; Motorola Solutions, Schaumburg, IL) were measured manually to establish baseline expectations by anesthesia staff of page transmission times.
We sequentially investigated the performance of 3 alphanumeric paging devices at Thomas Jefferson University Hospitals (TJUH). Each device (1) could be interfaced with computer-generated messages from our DSS, and (2) could receive numeric pages through the hospital’s internal phone system (Table 1). Tests were conducted in hospital locations with high signal quality reported by the device, and involved very short messages (e.g., “Test 1234”). Computer clocks were synchronized to a network time server and were verified to be within 1 second of each other. Measured latency did not include the time to open and read the message, studied previously.13,14
The remainder of the Methods provides technical details for analysts and software engineers wishing to replicate or apply our work. Readers unlikely to be involved personally in programming efforts can skip to the Results section, if desired.
Text messages sent to the first 2 alphanumeric devices, M90 Messenger (Unication, Taiwan, Republic of China) and Titan III (Sun Telecom, Louisville, KY), were transmitted from the DSS to an external server via the Internet using a common gateway interface maintained by the hospital’s paging vendor (SkyTel, a wholly owned subsidiary of American Messaging, Lewisville, TX). The messages were then returned to the hospital for delivery to the pagers via a locally installed, dedicated transceiver.
The final alphanumeric device, Advisor II (Motorola Solutions, Schaumburg, IL), used pathways within the hospital’s local Gigabit Ethernet network, including a main and backup transmitter, with no external dependencies.
All 3 devices were widely used within the hospital at the time of testing. The Advisor II pagers functioned as the primary alphanumeric device for the page operators to notify code and rapid response teams about the location of emergencies. However, text paging to these devices was not available to hospital staff. In response to a request from the anesthesia department, additional software was purchased by the hospital (Zetron, a wholly owned subsidiary of JVC Kenwood, Redmond, WA) to allow dedicated interfacing by anesthesia DSS with the paging system and to permit ad hoc text messaging.
For messages sent manually, latency was measured using an electronic stopwatch activated simultaneously when the test message was initiated (e.g., # key on the phone or send button in a software application). The watch was stopped when the pager indicated receipt of the test message through an audible beep. Manual tests were performed for all 3 alphanumeric devices.
Automated determination of latency varied according to the device tested. For test messages sent automatically to the 2-way M90 Messenger pager via SkyTel (approximately every 15 minutes for 21 weeks), the send time was retrieved from a log file created by the SQL Server (Microsoft, Redmond, WA) stored procedure creating the message. An ASPX.NET (Microsoft) web page was programmed to send these messages and to retrieve message delivery timestamps using the HTTP (Hypertext Transport Protocol) and SkyTel’s public socket server connection program.h Messages and timestamps were matched and stored in a database table on one of our AIMS servers for subsequent analysis.
Similar HTTP queries were constructed to send manual messages to the 1-way Titan III pagers via the SkyTel pathway. Automated testing was not possible for this device because timestamps of message delivery were not accessible (i.e., there is no transmission from this device back to the server).
Messages sent to the 1-way Advisor II devices via the hospital’s internal network were generated from an SQL stored procedure that wrote a small text file for each message (containing the page number and the message) to a server monitored by the Zetron interface software.i Test messages were sent every 2 minutes, 24 hours a day. Each file’s contents were passed to the paging console, which then transmitted the message. A text file was created on the Zetron server at the time of successful transmission of each message. Timestamps from the files were retrieved into an Excel workbook using Visual Basic for Applications (Microsoft). Latency was measured as the time the message was initiated from the SQL server until the time of transmission. Transmission timestamps and the corresponding clock times when pagers beeped were within 1.5 seconds.
The effect of mass paging on system latencyj was also assessed by sending 100 consecutive pages from an SQL stored procedure. The latency of each message was measured from initiation of the series of pages.
Latencies for test messages sent automatically from the SQL server were combined in 1-week batches to provide a sufficient number of messages to calculate small percentages of latencies longer than 100 seconds.13,15 The mean ± SE of the batched means are reported. To assure that this use of batched means resulted in statistically independent samples, we relied on Short Message Service text messages sent to cell phones, which have known unreliable service (see Introduction and footnotee) (Fig. 2). We also calculated the mean (among batches) of the percentage latencies exceeding 100 seconds ± SE of the mean percentage. The confidence intervals were calculated after Freeman-Tukey transformation, and then the inverse was computed.16,17
Latencies for comparative devices and those found to be unsuitable after a short period of testing (days) are reported as mean and 99th percentile.
The latency for our traditional phone-initiated numeric-only pagers (Bravo Plus, Motorola Solutions) was mean 7 seconds and 99th percentile 10 seconds (n = 200 pages). Thus, baseline expectations of our staff were for rapid page transmission.
M90 Messenger Pager (2-Way via SkyTel)
Manual testing using the in-house phone system to send n = 100 numeric pages to this device demonstrated a mean latency of 16 seonds and 99th percentile of 30 seconds. Over the subsequent 21 weeks, 13,697 automated test messages were analyzed (Fig. 3). The mean latency was 36 ± 7 seconds, with 1.5% ± 0.5% of latencies exceeding 100 seconds (n = 21 batches). The test period was extensive because SkyTel engineers and the hospital communications group incrementally tried to improve performance, adding antennae, repeaters, and enhanced network communication components. These steps were not successful.
Titan III Pager (1-Way via SkyTel)
Results for this pager via the SkyTel pathway were unsatisfactory. For n = 200 pages, the mean latency was 131 seconds, the 99th percentile was 591 seconds (9.9 minutes), and 33% of pages took longer than 100 seconds to arrive.
Advisor II Pager (1-Way via Zetron)
Initial manual testing using the in-house phone system to send numeric pages to the Advisor II pagers demonstrated a mean latency of 7 seconds, and 99th percentile of 9 seconds (n = 100 pages).
Over the subsequent 9 weeks, test messages were sent continually (Fig. 4). Initially, the mean latency was 12 seconds and the 99th percentile was 22 seconds (n = 9953 pages). During the third week of testing, a console hard drive and the 40-year-old transmitter were replaced. Over the following 6 weeks, the mean latency was 8 ± 0.2 seconds (n = 6 batches). No page had a latency longer than 100 seconds (0% ± 0%, n = 40,190 pages).
Impact of Multiple Simultaneous Pages on Overall Latency
Two tests of a simulated mass page were performed using the Advisor II and Zetron system (Fig. 5). In the first test (before the transmitter replacement), the mean time to clear each successive message was 2.1 seconds (n = 100 pages), whereas in the second test (after the upgrade), it was 1.9 seonds (n = 100 pages).
Implementation details of the Zetron paging system are described in Appendix A. Application of the preceding approach to testing of an existing system at a different hospital in another city (VUMC) is described in Appendix B.
The importance of our report is that it demonstrates that developers of anesthesia communication and DSSs need to conduct a formal latency analysis of the communication pathways and devices under consideration for urgent message delivery to mobile users. A principal result of the study is that intermittent manual testing over a period of days with hundreds of pages is insufficient to characterize long-term paging system performance, as such testing may occur fortuitously during periods of “fast” connectivity. This is what we experienced with the SkyTel system and the M90 pagers, and explains the difference in latency findings from the initial and subsequent testing. Automated testing over a period of many weeks with thousands of pages is necessary. Similar considerations apply for evaluation of pagers used on an ad hoc basis.
The latency differences between the text paging system in use at TJUH and VUMC are likely attributable to the network pathways in use (local versus Internet, respectively) (Appendix B and Fig. 6). The data suggest that if rapid, reliable text message delivery is required, internal systems are preferable. Based on the small latencies of the text paging system at TJUH and the fact that only a portion of pages sent required immediate attention, the paging pathway is not an important factor in the overall performance of the anesthesia DSS.
Other investigators have examined the use of pagers for real-time clinical event monitoring and notification for general medical or intensive care unit use.18–22 None measured latency attributable to the communication pathway. Sandberg et al.23 studied mean latency within a notification system that sent text messages to alphanumeric pagers when a simulated patient was taken to the wrong OR, based on identification of patients using radiofrequency tags. They did not formally evaluate the reliability of the communication pathway (e.g., 95th or other upper percentiles).
Our study was limited to latencies from the technical performance of the system, and did not account for human behavior.12,14 A study of response times to pages by anesthesia care providers has not been performed; however, we have reported average acknowledgment times for messages delivered as pop-up messages on anesthesia workstations of 1.3 minutes.13,14 Although we studied text messages (Fig. 1) and 2-way SkyTel pagers (Fig. 3), these approaches can result in protected health informationk being transmitted and stored on vendors’ servers in an unencrypted format. This may raise privacy concerns. Finally, DSS and ad hoc text messaging is dependent on several components that can malfunction (e.g., AIMS servers, Zetron software), resulting in failure of text message delivery. Under such circumstances, backup communication pathways need to be used (e.g., numeric pages via the phone system, overhead pages, cell phones).
Our report highlights that wireless message delivery systems involving processes outside the local hospital network (i.e., public Internet) are subject to intermittent periods of long latency. Whereas rejection of a device and pathway can be made after a few hours of evaluation (e.g., see Titan III in Results), testing to accept use of a device takes weeks (Fig. 3). The effective sample size is not the number of pages, but rather, the number of batches of pages. Acceptable latency during a brief period of testing with hundreds of pages may not be indicative of long-term performance. The appropriate interpretation of our findings is not that Zetron software and Advisor II pagers perform excellently, but rather that local testing is required of whatever text message delivery system is proposed.
APPENDIX A: IMPLEMENTATION AT THOMAS JEFFERSON UNIVERSITY HOSPITALS
The Thomas Jefferson University Hospitals anesthesia department decided to switch to the Advisor II and Zetron system for in-house text message delivery from the anesthesia decision support system and for ad hoc alphanumeric paging. The department paid a one-time fixed cost ($115) for each pager; the hospital absorbs maintenance costs of the paging system. Each staff member retained his or her 4-digit pager number, allowing a seamless cutover with no need to update pager lists throughout the hospital. When a pager breaks or is misplaced, the communications department quickly reprograms a spare device to the user’s pager number. The pagers use AA batteries that typically last several weeks and are replaced easily. Special numbers are programmed into each pager, allowing a single page to be broadcast to everyone in a group (e.g., “All Staff” and “Attendings”) instead of a large batch of individual pages (which would result in delayed messages). Individual ad hoc text messages can be sent from each AIMS workstation using a hyperlink to an ASPX.NET page (Fig. A1), and numeric pages via the phone system. An SQL job scheduled every 15 minutes alerts system administrators by e-mail and cell phone text message if more than 7 messages are present in the Zetron queue, a possible indication of system failure.
APPENDIX B: EXAMPLE OF APPLYING THE EVALUATION METHOD AT ANOTHER HOSPITAL
We tested latency of the alphanumeric pagers at Vanderbilt University Medical Center (VUMC) for potential use as a messaging component within their anesthesia decision support system.2 VUMC uses Advisor II pagers (the same model as in use at Thomas Jefferson University Hospitals [TJUH]), with service provided by a third party vendor (Aquis Communications, Yorktown, VA). Text messages are sent to external servers using the Simple Network Paging Protocol (SNPP) over a public Internet connection and then transmitted to the pagers via local radio towers (the same final pathway as at TJUH). Automated latency testing was performed every 5 to 6 minutes using a dedicated computer with custom software provided by the vendor. Messages were sent and timestamps captured for messages that return to a data receiver attached to the computer. The prolonged latencies occasionally observed at TJUH with SkyTel and at VUMC with Aquis are likely attributable to dependencies on the Internet, not the pager or local transmission process.
Manual testing at convenient times over a 2-week period demonstrated a mean latency of 31 seconds and a 99th percentile of 101 seconds (n = 99 pages). Pages from the automated testing process were batched in 6 one-week periods (n = 11,915, approximately 2000 pages per batch) (Fig. 6). The mean latency was 19.8 ± 0.5 seconds. The percentage of pages exceeding 100 seconds was 0.6% (95% confidence interval, 0.2%–1.1%).
Dr. Franklin Dexter is the Statistical Editor and Section Editor for Economics, Education, and Policy for the Journal. This manuscript was handled by Dr. Steven L. Shafer, Editor-in-Chief, and Dr. Dexter was not involved in any way with the editorial process or decision.
Name: Richard H. Epstein, MD, CPHIMS.
Contribution: This author helped design the study, conduct the study, analyze the data, and write the manuscript.
Attestation: Richard H. Epstein has seen the original study data, reviewed the analysis of the data, approved the final manuscript, and is the author responsible for archiving the study files.
Name: Franklin Dexter, MD, PhD.
Contribution: This author helped design the study, conduct the study, analyze the data, and write the manuscript.
Attestation: Franklin Dexter has seen the original study data, performed the analysis of the data, and approved the final manuscript.
Name: Brian Rothman, MD.
Contribution: This author helped design the study, conduct the study, analyze the data, and write the manuscript.
Attestation: Brian Rothman has approved the final manuscript.
We thank Warren Sandberg, MD, PhD, and Jason Lane, MD, for performing some of the manual latency testing at Vanderbilt University Medical Center. We also appreciate the assistance of Nick Forlidas, Aquis Communications, who provided log files for the test messages from Vanderbilt.
a National Communications System. SMS over SS7. Technical Report Technical Information Bulletin 03-2 (NCS TIB 03–2), December 2003. Available at: http://www.ncs.gov/library/tech_bulletins/2003/tib_03-2.pdf. Accessed August 9, 2012.
b Government of Canada Office of Critical Infrastructure Protection and Emergency Preparedness, Incident Analysis: The September 11, 2001 Terrorist Attacks—Critical Infrastructure Protection Lessons Learned, September 2002. Available at: http://www.publicsafety.gc.ca/prg/em/ccirc/_fl/ia02-001-eng.pdf. Accessed August 9, 2012.
c Joint ITU-T/OASIS Workshop and Demonstration of Advances in ICT Standards for Public Warning, October 2006. Available at: http://www.isoc.org/pubpolpillar/docs/Highlights-and-Actions-2a1.pdf. Accessed August 9, 2012.
d Elliott AM. Texters to Experience 6 Hour Delays on New Year’s Eve, December 2007. Available at: http://www.pocket-lint.com/news/news.phtml/11895/. Accessed August 9, 2012.
e Meng X, Zerfos P, Samanta V, Wong S, Lu S. Analysis of the Reliability of a Nationwide Short Message Service. Proceedings of IEEE INFOCOM, 2007. Available at: http://www.cs.ucla.edu/wing/publication/papers/Meng.INFOCOM07.pdf. Accessed August 9, 2012.
f Sauer LM, McCarthy ML, Knebel A, Brewster P. Major Influences on Hospital Emergency Management and Disaster Preparedness. Disaster Med Public Health Prep 2009;3:S68–73 Available at: http://www.dmphp.org/cgi/content/full/3/Supplement_1/S68. Accessed August 9, 2012.
g We selected 30 seconds as a tradeoff between the short latency of our numeric pages (see Results) and the additional time to call back to determine the reason for the page. We selected 100 seconds for the upper limit of pager latency based on geographic considerations related to response times at our hospital. Other facilities may choose a different threshold, but the principles described are identical. The expected number of DSS pages in the department was 200 per day.
h Details available at: http://www.skytel.com/destineer/ss_paging.cgi. Accessed January 6, 2012.
i Details of the proprietary Zetron text file format are available from the vendor on purchase of the interface software.
j We are aware of a medical institution using alphanumeric pagers where an enterprising individual managed to send a lunch meeting reminder to more than 1000 individuals at one time. This resulted in a prolonged degradation in paging system performance. The software at that hospital was subsequently changed to prevent more than 5 simultaneous messages at one time.
k Providers may inadvertently include elements of protected health information, as defined by the Health Insurance Portability and Accountability Act of 1996 (HIPAA), in their messages. These may be exposed during transmission over public Internet pathways or retrieved by third parties from vendors’ servers. Paging and cell phone vendors are not subject to HIPAA regulations and do not have Business Associate agreements covering their use of such information.
1. Egger Halbeis CB, Epstein RH, Macario A, Pearl RG, Grunwald Z. Adoption of anesthesia information management systems by academic departments in the United States. Anesth Analg. 2008;107:1323–9
2. Rothman B, Sandberg WS, St. Jacques P. Using information technology to improve quality in the OR. Anesthesiol Clin. 2011;29:29–55
3. Epstein RH, Dexter F. Implications of resolved hypoxemia on the utility of desaturation alerts sent from an anesthesia decision support system to supervising anesthesiologists. Anesth Analg. 2012;115:929–33
4. St. Jacques P, Sanders N, Patel N, Talbot TR, Deshpande JK, Higgins M. Improving timely surgical antibiotic prophylaxis redosing administration using computerized record prompts. Surg Infect. 2005;6:215–21
5. Wax DB, Beilin Y, Levin M, Chadha N, Krol M, Reich DL. The effect of an interactive visual reminder in an anesthesia information management system on timeliness of prophylactic antibiotic administration. Anesth Analg. 2007;104:1462–6
6. Ehrenfeld JM, Epstein RH, Bader S, Kheterpal S, Sandberg WS. Automatic notifications mediated by anesthesia information management systems reduce the frequency of prolonged gaps in blood pressure documentation. Anesth Analg. 2011;113:356–63
7. Epstein RH, Dexter F. Mediated interruptions of anaesthesia providers using predictions of workload from anaesthesia information management system data. Anaesth Intensive Care. 2012;40:803–12
8. Sandberg WS, Sandberg EH, Seim AR, Anupama S, Ehrenfeld JM, Spring SF, Walsh L. Real-time checking of electronic anesthesia records for documentation errors and automatically text messaging clinicians improves quality of documentation. Anesth Analg. 2008;106:192–201
9. Spring SF, Sandberg WS, Anupama S, Walsh JL, Driscoll WD, Raines DE. Automated documentation error detection and notification improves anesthesia billing performance. Anesthesiology. 2007;106:157–63
10. Kheterpal S, Gupta R, Blum JM, Tremper KK, O’Reilly M, Kazanjian PE. Electronic reminders improve procedure documentation compliance and professional fee reimbursement. Anesth Analg. 2007;104:592–7
11. Mukhtar A, Hasanin A, ElAdawy A, Osman S, Ahmed A, Nassar H, Saad D, Zaghloul A, Sarhan M, Reda M. The Friday of rage of the Egyptian revolution: a unique role for anesthesiologists. Anesth Analg. 2012;114:862–5
12. Epstein RH, Dexter F, Ehrenfeld JM, Sandberg WS. Implications of event entry latency on anesthesia information management decision support systems. Anesth Analg. 2009;108:941–7
13. Ledolter J, Dexter F, Epstein RH. Analysis of variance of communication latencies in anesthesia: comparing means of multiple lognormal distributions. Anesth Analg. 2011;113:888–96
14. Dexter F, Epstein RH, Lee JD, Ledolter J. Automatic updating of times remaining in surgical cases using Bayesian analysis of historical case duration data and instant messaging updates from anesthesia providers. Anesth Analg. 2009;108:929–40
15. Law AM, Kelton WD Simulation Modeling and Analysis. 19912nd ed New York McGraw-Hill:551–3
16. Dexter F, Marcon E, Epstein RH, Ledolter J. Validation of statistical methods to compare cancellation rates on the day of surgery. Anesth Analg. 2005;101:465–73
17. Miller JJ. The inverse of the Freeman-Tukey double arcsine transformation. Am Stat. 1978;32:138
18. Wagner MM, Pankaskie M, Hogan W, Tsui FC, Eisenstadt SA, Rodriguez E, Vries JK. Clinical event monitoring at the University of Pittsburgh. Proc AMIA Annu Fall Symp. 1997:188–92
19. Eisenstadt SA, Wagner MM, Hogan WR, Pankaskie MC, Tsui FC, Wilbright W. Mobile workers in healthcare and their information needs: are 2-way pagers the answer? Proc AMIA Symp. 1998:135–9
20. Wagner MM, Tsui FC, Pike J, Pike L. Design of a clinical notification system. Proc AMIA Symp. 1999:989–93
21. Chen HT, Ma WC, Liou DM. Design and implementation of a real-time clinical alerting system for intensive care unit. Proc AMIA Symp. 2002:131–5
22. Poon EG, Kuperman GJ, Fiskio J, Bates DW. Realtime notification of laboratory data requested by users through alphanumeric pagers. J Am Med Inform Assoc. 2002;9:217–22
23. Sandberg WS, Häkkinen M, Egan M, Curran PK, Fairbrother P, Choquette K, Daily B, Sarkka JP, Rattner D. Automatic detection and notification of “wrong patient-wrong location” errors in the operating room. Surg Innov. 2005;12:253–60