Tablet computers and smart phones have gained popularity in anesthesia departments for both instructional and patient care purposes.1–3,a,b Some United States anesthesia training programs have deployed iPads (Apple, Inc., Cupertino, CA) to their residents in lieu of providing book stipends or laptop computers.c Medically related uses of the iPads include reading textbooks and journal articles, accessing online content, completing anesthesia preoperative consults, using the hospital’s electronic medical record, and reviewing prior anesthesia records. In addition to the authors’ institutions, several other anesthesia departments have software teams developing applications for mobile devices for both clinical and educational use.d
A unique application for mobile devices running the Apple iOS operating system called VigiVU™, developed by Vanderbilt University’s anesthesia department,4 is currently in department-wide use. VigiVU features include real-time operating room video streaming, text paging functions, anesthesia record viewing, and an operating room schedule view that delivers perioperative information related to patient status.5 Users can subscribe to specific rooms to receive notifications of out-of-range vital signs6 and patient status alerts as patients progress through their perioperative care (Fig. 1). The notifications are customizable at the case level by each user and are designed to improve patient flow (e.g., patient arrival in the holding area). System messages and messages from other providers are sent through the Apple Push Notification (APN) service with the same messages delivered concurrently to hospital-provided radiofrequency alphanumeric pagers (Bravo Plus, Motorola, Schaumberg, IL) through a third party vendor (Aquis Communications, Yorktown, VA). The APN service is Apple’s propriety method of sending messages to mobile devices on which applications have been enabled to receive such messages (even if they are not active), and users have elected to receive them. A typical use by vendors is to send notifications of a new software version.
We previously studied latencies of text message delivery to a variety of devices in the context of critical messaging. Pathways involving Internet connections such as cell phones and dedicated text paging devices (including Aquis) were subject to an unacceptably high incidence of prolonged (>100 seconds) latencies.7 Staff perceptions at Vanderbilt were that APN notifications arrive sooner than pages sent via Internet connections to their alphanumeric pagers. However, our prior work demonstrated that devices need to be extensively tested over many weeks to evaluate whether performance is indeed adequate for critical use.7 For example, even if 1% of messages were prolonged (>100 seconds) beyond an acceptable threshold, this would affect several messages every day, given typical daily numbers of pages.7
Vanderbilt anesthesia staff are required to carry alphanumeric pagers as their primary message receipt device. The APN service in VigiVU is an optional feature and is not intended to serve as a pager replacement. Although we previously demonstrated that the Aquis paging system has a 0.6% (95% confidence interval, 0.2%–1.1%) incidence of prolonged (>100 seconds) message delivery,6 Apple’s explicit statement that delivery of APN messages is only “best effort” and is not guaranteede has resulted in continued use of the text pagers as the primary notification portal. Since our previous studies had excluded multiple seemingly appropriate devices,7 the reliability of the APN service needed to be established.
The objective of this study was to evaluate if the (best-case) performance of the APN service is sufficiently reliable to be used for delivery of time-critical text messages. We based our analysis of reliability on our previously published acceptance criteria of a mean latency of less than 30 seconds and not more than 0.5% of pages having latencies exceeding 100 seconds.7 The upper limit of 100 seconds was originally selected based on geographic considerations related to response times at a hospital (Ref. 7, footnote g). This choice is also useful because it facilitates graphical representation on logarithmic scales (e.g., Fig. 2). Because there are potential differences in the reliability of APN message delivery over cellular and wireless local area network (WLAN) pathways as well as in hardware performance, we studied both communication pathways using an iPhone® (4S), an iPad® (third generation), and 2 iPod Touch® (4th generation) devices, all running iOS 6.0 (Apple Inc.).
The free Java® (Oracle America, Inc., Redwood Shores, CA) application “Push Me Baby”f was modified to transmit APN messages sequentially to multiple iOS devices, using their device tokeng as the identifier, and was installed on a MacBook Pro computer (“Mac”) running Mac OS X Lion (Apple Inc.). These messages included a unique message identifier, the local clock timestamp of message transmission, the offset of the local clock from a network time server, and free text comments that could be added to note unusual events (e.g., resumption after a software crash). Custom software was written in Objective-C® (Apple Inc.), extensively tested and debugged using Apple’s interactive development environment, and installed on the mobile devices under the department’s Apple Developer provisioning certificate, with APN push notifications enabled. The iOS application was designed to log the message text and the time that the message was received to a file that could then be downloaded for analysis. We did not test latency of message delivery from within the VigiVU application, because our testing method required the use of dedicated software.
As shown in Figure 3, clocks on mobile Apple devices gradually drift from reference clocks and are only synchronized every few days to the Apple time server. The Mac clock also drifts, but to a lesser extent, because it is synchronized more frequently to the Apple time server. To adjust for these inaccuracies, the custom software on each device asynchronously queried the United States Naval Observatory (USNO) atomic clock time serverh at 1-minute intervals to determine the offset of the device’s internal clock from the reference time. The most recently received value of the offset was transmitted by the Mac application as part of the APN message text. When the iOS device received the message, it was logged to a local file along with the arrival timestamp and the most recent offset from the device relative to the USNO reference time. Local clock times were then corrected by the relative offset, averaged using the median value in a 30-minute window centered around the transmission time (for the Mac) or receipt time (for the iOS devices). The sampling window was adjusted not to overlap transitions due to resynchronization events, detected automatically by a step transition in successive offsets ≥500 milliseconds (Fig. 3). Message latency was calculated as the number of milliseconds between the corrected transmission and receipt times. We verified the accuracy of our time adjustments on the tested iOS devices (using the USNO atomic clock) through the iOS application Emerald Time (Emerald Sequoia, LLC, Los Gatos, CA) which displays the device offset using multiple Network Time Protocol servers on the Internet. We checked the MAC clock using time services provided by the National Institute of Standards and Technology.i
Devices and networks available to the investigators were used for testing. The Mac computer used to transmit messages was connected through a wireless router and cable modem to the Internet through a local service provider (Comcast, Philadelphia, PA). Mobile devices received the APN messages via several pathways. The iPhone used the AT&T 3G cellular network, the iPad used the secure WLAN at Thomas Jefferson University Hospital (TJUH, Philadelphia, PA), one of the iPods used the secure WLAN at Vanderbilt University, and the other iPod was connected via a wireless router and cable modem to the Comcast Network. For the first half of testing (September and October 2012), the Mac, iPhone, and one of the iPads were located in Atlantic City, NJ. During this phase, one iPad was located at TJUH and the other iPad was at Vanderbilt University Medical Center (Nashville, TN). For the second half of testing (November and December, 2012), one of the iPods, the iPhone, and the iPad were at Vanderbilt University Medical Center, and the Mac and other iPod were in downtown Nashville. The principal author relocated from TJUH to Vanderbilt to begin a sabbatical in early November, resulting in the gap between the 2 phases of testing.
Messages were transmitted sequentially at 15-second intervals from the Mac to the 4 iOS mobile devices, 24 hours a day, resulting in 1 message transmitted per minute to each device. Occasional lapses in transmission occurred due to technical reasons (e.g., software crashes, devices being powered off accidentally). Devices were maintained in fixed locations with high signal strength during all testing periods.
Latencies for test messages were combined in daily batches to provide a sufficient number of messages (nominally 1440 per 24 hours) to calculate the (expected) small percentages of latencies longer than 100 seconds.8,9 The method of batch means is described in Refs. 8 and 9. The means (among batches) of the percentage of latencies exceeding 100 seconds ± SEM percentage are reported. The exact 95% upper confidence limits for the percent of days with at least 1 prolonged latency (>100 seconds) were calculated using the method of Blyth-Still-Casella (StatXact version 9.0, Cytel Inc., Cambridge, MA). We also calculated the mean latency ± SE of the batched means.
All devices had mean latencies of <30 seconds and fewer than 0.5% of pages had latencies exceeding 100 seconds using the APN system and either WLAN or cellular Internet connectivity (Table 1). Among more than 173,000 test messages, there were no APN latencies longer than 100 seconds for the 2 iPods and the iPad using WLAN pathways. For the iPhone using the cellular pathway, 0.03% ± 0.01% of APN messages were received >100 seconds after transmission, an order of magnitude below our 0.5% threshold. Mean latencies were also an order of magnitude lower than our 30 seconds threshold, with the iPhone having the longest, but acceptable latency of 3.22 ± 0.07 seconds.
Rare instances of markedly prolonged latencies (>5 minutes) were noted for the iPhone device using the cellular pathway (Fig. 2), but not for the other devices (Figs. 4–6). Performances of the iPod Touch (Figs. 4 and 5) and iPad (Fig. 6) were superior to that of the iPhone, based on the distribution of nearly all latencies on the iPod and iPad devices below 1 second and absence of any latencies above 100 seconds. The upper 95% confidence limits of days with at least 1 prolonged APN message were 42% for the iPhone and between 5% and 8% for the other devices, at a message frequency of approximately 1300 per day.
Prolonged testing conducted over several months and involving several hundred thousand APN messages demonstrated that the Apple APN service appears sufficiently robust for use where messages need to be received in timely fashion. Overall, message delivery was remarkably rapid, with subsecond mean latency for the iPad and iPod Touch devices, and just over 3 seconds for the iPhone. The slowest APN pathway (iPhone, mean 3.2 ± 0.7 seconds) performed better (P < 0.0001) than the other Internet-based alphanumeric text message providers we have studied7 such as 2-way SkyTel (mean latency 36 ± 7 seconds, 1.5% ± 0.5% exceeding 100 seconds) and Aquis (mean latency 19.8 ± 0.5 seconds), or text messages delivered via the Short Message Service (SMS) to smart phones.j However, if use of the APN service is being considered for time-sensitive messaging, testing similar to what we describe in the environment where devices will be deployed should be performed before implementation. Every facility and information system implementation differ, and our testing was deliberately best case.
Because the sample size for latency testing is the number of batches (days), not the number of messages,7 one can only be 95% certain for iPod Touch and iPad devices using a WLAN connection that at least 19 of every 20 days will have no prolonged latencies, assuming a daily paging frequency of 1300.k,l Although the frequency of pages with prolonged latency was extremely low using the iPhone device on a 3G cellular network, there were still intermittent instances of messages taking longer than 10 minutes to arrive. Our results show that one can only be 95% certain than no more than 2 of every 5 days will have 1 or more pages with prolonged latency when using the iPhone on a 3G network. This is consistent with Apple’s disclaimer that APN message delivery is not guaranteed (see footnote e in the Introduction). However, no message delivery system, including text paging systems running completely within a hospital’s infrastructure, will be 100% reliable, since hardware components can fail (e.g., power supplies, transmitters, servers, access points). Backup communication systems need to be in place (e.g., an overhead paging system, walkie talkies, or another method not dependent on the APN servers), because if there is an interruption to Internet service that prevents inbound or outbound connections to the Apple APN servers, the APN message process will fail entirely. This was experienced at the end of the testing period when Internet service in Atlantic City, where the APN messages were being initiated, was lost as Hurricane Sandy made landfall on October 29, 2012.m Thus, we continue to recommend that hospitals not rely solely on methods dependent on functioning Internet connectivity.
Although we did not encounter any prolonged periods of extreme latency due to cellular network congestion during our testing, as has been described for SMS text messages sent to smart phones,n,7 we remain concerned about the impact of APN transmission over cellular pathways when these networks become congested. On nearly 1 of 3 days, there was at least 1 iPhone APN message with prolonged latency, so this is not just a theoretical concern. Under iOS versions up to 6.1, the Apple mobile devices preferentially use cellular pathways (if available) to deliver APN messages, even if there is an available WLAN connection.o There is no user option to change the order of priority, which is designed to maximize battery life. Thus, we recommend that processes relying on APN notices for critical messaging within a hospital environment use the internal WLAN, not a cellular network. This can be accomplished by either using a device without a cellular connection or by temporarily disabling the cellular service (e.g., by putting the device in “airplane mode” and then turning Wi-Fi back onp).
A final concern about the use of APN notifications is that although messages are encrypted during transmission using the highly secure Transport Layer Security protocol, Apple has the ability view messages in clear text.q Messages are stored on the APN servers for a variable period of time before being deleted, and Apple explicitly advises to “never include sensitive data in the payload.”p There are similar security issues for text messages sent to alphanumeric pagers or cell phones via third parties, since these are also logged by the vendor and, even more concerning, may be transmitted over the Internet in clear text. Thus, the content of automated messages needs to be carefully considered, regardless of the delivery method, and protected health information not transmitted, since inadvertent disclosure has serious implications.r
Our study has several limitations. We only studied 1 cellular carrier (AT&T) in 2 different cities with the iPhone, so it is possible that performance with other carriers and/or different cities will be different. Hospital WLAN networks may also differ in their performance, potentially adding to latency due to bandwidth contention or geographical coverage issues. Devices were kept in 1 location with high signal strength during all study periods, so the impact on APN transmission of moving between cell towers or WLAN access points could not be determined. This might decrease performance when deployed in a more real-world environment.
Another limitation was that we did not test the APN system for message delivery to the iPhone under high cellular network traffic conditions. The APN system uses different pathways than those used for SMS communication (which shares bandwidth with voice communication), so this scenario may or may not impact performance.
Also, our study only evaluated the technical performance of the APN system. We did not account for human factors such as the time for recipients to read or respond to messages or how long it would take to notice them after they arrived.8,10
Finally, we did not evaluate performance of the Google Cloud Messaging service,s which provides similar functionality as the APN service for devices running the Android operating system (Google, Mountain View, CA). Thus, the conclusions we reached regarding APN performance may not be applicable to the Google Cloud Messaging pathway.
In conclusion, our paper demonstrates that the Apple APN system can be sufficiently robust to deliver time-critical messages to iOS devices. However, since our study was a best-case assessment, at most what we have shown is that the APN system is suitable for testing at individual sites, unlike many of the other (rejected) devices that we have investigated. Furthermore, regardless of the system used to deliver critical messages to anesthesia personnel, backup communication systems need to be available in the event of service disruption.
Franklin Dexter is the Statistical Editor and Section Editor of Economics, Education, and Policy for the Journal. The manuscript was handled by Maxime Cannesson, Section Editor of Technology, Computing, and Simulation and Dr. Dexter was not involved in any way with the editorial process or decision.
Name: Brian S. Rothman, MD.
Contribution: This author helped design the study, conduct the study, analyze the data, and write the manuscript.
Attestation: Brian S. Rothman has seen the original study data, reviewed the analysis of the data, and approved the final manuscript.
Name: Franklin Dexter, MD, PhD.
Contribution: This author helped design the study, conduct the study, analyze the data, and write the manuscript.
Attestation: Franklin Dexter has seen the original study data, reviewed the analysis of the data, and approved the final manuscript.
Name: Richard H. Epstein, MD, CPHIMS.
Contribution: This author helped design the study, conduct the study, analyze the data, and write the manuscript.
Attestation: Richard H. Epstein has seen the original study data, reviewed the analysis of the data, approved the final manuscript, and is the author responsible for archiving the study files.
a Escobar SK, Escobar ED, Whitten C, Griffin JD. Texas iPad anesthesia education domain: http://www.utswanesthesia.com/tipad/. 2011 Annual Meeting of the American Society of Anesthesiologists. A163.
b Escobar ED, Tighe BS, van Oostrom H, Robicsek S. 2011 Annual Meeting of the American Society of Anesthesiologists. A668. Accessed February 5, 2013.
c Some of the anesthesia training programs providing iPads to residents are George Washington University, Mount Sinai Medical Center, Northwestern University, Thomas Jefferson University, University of Chicago, University of Florida, University of Michigan, University of Tennessee, and Vanderbilt University (retrieved from Google using the search terms anesthesia iPad resident, and resident iPad deployment, performed on February 4, 2013). The number of anesthesia programs providing such devices to their residents is growing rapidly.
d Personal communications, Matthew D. McEvoy (Medical University of South Carolina), Larry Chu (Stanford University), 2013.
e Apple Inc. Local and Push Notification Programming Guide. Available at: http://developer.apple.com/library/ios/#documentation/NetworkingInternet/Conceptual/RemoteNotificationsPG/Introduction.html. Accessed January 31, 2013.
f Hafeneger, S. PushMeBaby source code. Available at: https://github.com/stefanhafeneger/PushMeBaby. Accessed February 5, 2013.
g The device token is not the same as the unique device identifier, which is a hardware identifier. The device token is derived from the iOS instance on the device, and migrates to a new device if the user restores from a backup. The device token allows the APN service to locate the specific device to which messages are to be sent.
h United States Navy. US Naval Observatory Master Clock Time. Available at: http://tycho.usno.navy.mil/simpletime.html. Accessed February 5, 2013.
i National Institute of Standards and Technology. Available at: http://nist.time.gov. Accessed March 26, 2013.
j Meng X, Zerfos P, Samanta V, Wong S, Lu S. Analysis of the Reliability of a Nationwide Short Message Service, Proceedings of IEEE INFOCOM, 2007. Available at: http://www.cs.ucla.edu/wing/publication/papers/Meng.INFOCOM07.pdf. Accessed February 3, 2013.
k The text paging frequency from the decision support systems at both Thomas Jefferson University Hospital and Vanderbilt Medical Center are approximately 300 per day, so performance would be expected to be better than this.
l Joint ITU-T/OASIS Workshop and Demonstration of Advances in ICT Standards for Public Warning, Oct 2006. Available at: http://www.isoc.org/pubpolpillar/docs/Highlights-and-Actions-2a1.pdf. Accessed January 25, 2013.
m http://arstechnica.com/information-technology/2012/10/hurricane-sandy-also-disrupts-cellular-networks-and-wired-internet/. Accessed January 29, 2013.
n Elliott A-M. Texters to Experience 6 Hour Delays on New Year’s Eve. December 2007. Available at: http://www.pocket-lint.com/news/news.phtml/11895/. Accessed January 25, 2013.
o Apple Inc. Unable to use Apple Push Notification service (APNs). Available at: http://support.apple.com/kb/TS4264. Accessed February 26, 2013.
p Apple Inc. iOS: Understanding Airplane Mode. Available at: http://support.apple.com/kb/HT1355. Accessed January 28, 2013.
q Dhanjani, N. Apple iOS Push Notifications: Security Implications, Abuse Scenarios, and Countermeasures. Available at: http://software-security.sans.org/blog/2011/02/07/apple-ios-push-notifications-security-implications-abuse-scenarios-and-countermeasures. Accessed January 25, 2013.
r AISHealth. Pervasive HIPAA failings net surgeons the first OCR sanctions against physicians. Report on Patient Privacy 2012;12:1,7–11 Available at: http://aishealth.com/archive/hipaa0512-01. Accessed January 25, 2013.
s Google Inc. Google Cloud Messaging for Android. Available at: http://developer.android.com/google/gcm/index.html. Accessed February 4, 2013.
1. Tanaka PP, Hawrylyshyn KA, Macario A. Use of tablet (iPad®) as a tool for teaching anesthesiology in an orthopedic rotation. Rev Bras Anestesiol. 2012;62:214–22
2. Lacquiere DA, Courtman S. Use of the iPad in paediatric anaesthesia. Anaesthesia. 2011;66:629–30
3. Bhansali R, Armstrong J. Smartphone applications for pediatric anesthesia. Paediatr Anaesth. 2012;22:400–4
4. Lane JS, Sandberg WS, Rothman B. Development and implementation of an integrated mobile situational awareness iPhone application VigiVU™ at an academic medical center. Int J Comput Assist Radiol Surg. 2012;7:721–35
5. Rothman B, Sandberg WS, St Jacques P. Using information technology to improve quality in the OR. Anesthesiol Clin. 2011;29:29–55
6. Epstein RH, Dexter F. Implications of resolved hypoxemia on the utility of desaturation alerts sent from an anesthesia decision support system to supervising anesthesiologists. Anesth Analg. 2012;115:929–33
7. Epstein RH, Dexter F, Rothman B. Communication latencies of wireless devices suitable for time-critical messaging to anesthesia providers. Anesth Analg. 2013;116:911–8
8. Ledolter J, Dexter F, Epstein RH. Analysis of variance of communication latencies in anesthesia: comparing means of multiple log-normal distributions. Anesth Analg. 2011;113:888–96
9. Law AM, Kelton WD Simulation Modeling and Analysis. 19912nd ed New York, NY McGraw-Hill, Inc:551–3
10. Epstein RH, Dexter F, Ehrenfeld JM, Sandberg WS. Implications of event entry latency on anesthesia information management decision support systems. Anesth Analg. 2009;108:941–7