Critically ill patients generate vast quantities of data that clinicians must gather, interpret, and synthesize to facilitate effective clinical decision making. Daily interprofessional rounds are an integral component of critical care delivery as they provide an opportunity for ICU physicians, nurses, and pharmacists to converge, share data, and formulate the patient’s daily care plan (1,2). Prior to rounds, each team member engages in their own “prerounding” or data gathering and cognitive processing in preparation to verbally share information (3). Medical errors are now the third leading cause of death in the United States (4). Previous research shows that incomplete data gathering and processing leads to diagnostic error and subsequent patient harm (5,6). Thus, to minimize diagnostic error, communicated data should be accurate, complete, and current. Surprisingly, little is known about the quality of communicated data during rounds.
There are many reasons to question the accuracy of data communicated during ICU rounds. First, despite the implementation of interprofessional rounds, diagnostic error in the ICU remains a common occurrence. In one study, 28% of ICU deaths demonstrated at least one missed, wrong, or delayed diagnosis with 6% of these potentially fatal (7). Several studies demonstrate that data sharing during patient handoffs is rife with data communication failures (8–10). Poor handoff communication may occur when there is no standardized format, allowing for wide variability in content and quality of information transfer (11). A lack of standardized ICU rounds data reporting practices and individual clinician variations in ICU prerounding workflow may also affect the accuracy of the reported data.
The ICU environment also makes data gathering and communication challenging. With over 1,300 new data points/ICU patient per day, clinicians face information overload and may suffer from cognitive fatigue (12–14). Despite evidence suggesting that physicians who have been in clinical practice longer integrate data more successfully than those with less clinical experience, historic precedence in academic centers relegates the majority of data gathering to the least experienced physician on the team (15,16). Interruptions and patient care emergencies are also commonplace in the ICU leading to clinician workflow disruptions that impair memory and task completion efficiency (17).
Finally, the health record user interface also influences clinician ability to extract and synthesize data. Paper-based patient records are often disorganized, illegible, and location bound, making serial data assessment and integration time consuming and inefficient (18,19). Paper records also facilitate data omission and transcription errors given clinicians must manually reproduce information in order to transport it beyond the patient’s chart (20). Electronic health records (EHRs) offer a data management solution by providing a central repository of categorized, legible data that can be accessed by multiple users simultaneously during ICU rounds (21). EHRs also support the creation of rounding tools, which through macros, can automatically populate data, saving time and eliminating transcription errors (22,23). Unfortunately, EHR use has yet to consistently improve ICU outcomes, which may in part relate to problems with EHR implementation, novel patient safety issues created by the EHR, and suboptimal EHR design and usability (24–27). The extent to which EHRs hinder data gathering and information sharing on ICU rounds also remains unclear (21).
For all of these reasons, we hypothesized that communication of patient data during ICU rounds is prone to errors and inaccuracies. Using the EHR, we created a methodology to evaluate physician data gathering and data communication accuracy on ICU rounds.
The study was conducted in a single 26-bed closed medical ICU (MICU) at an urban tertiary care academic medical center. Daily rounds occur in the hallway outside the patient’s room and follow a structured rounding script. Trainees present interval data, and then solicit input from nursing, pharmacy, and respiratory therapy before presenting the plan. At least two EHR-equipped mobile computers are present on rounds at all times. No changes to this process occurred during the study period. Real-time attending physician use of the EHR on rounds is optional but encouraged. Our institution has used EPICcare (Epic Systems, Verona, WI) since 2008 and there were no major system upgrades during the study period. The study was approved by the Oregon Health and Sciences University Institutional Review Board.
Two senior ICU fellows (K.A., E.D.) audited the accuracy of laboratory data communication on daily ICU rounds. To ensure adequate training in the study methodology, personnel piloted data collection, including use of a templated data collection tool (Supplemental Digital Content 1, http://links.lww.com/CCM/C103) and direct observation by the senior author during a 1-month run-in period. Data collection commenced in August 2013 and occurred on weekdays with even sampling between the first and second half of the attending’s time on service (phase 1). Interim analysis after 3 months revealed a very high percentage of omitted laboratory data and that trainees uniformly used their prerounding notes (artifact) as a presentation aid. To understand at what point in clinician workflow loss of data fidelity occurred, the protocol was revised to include obtaining deidentified photocopies of presenters’ prerounding artifacts (phase 2, February 2014 to June 2014). We planned enrollment at 200 patients since a preventable life-threatening or fatal adverse event occurs on approximately one of every 200 ICU patient days (28). To avoid contamination of the prerounding process, investigators arrived minutes prior to the start of rounds. Investigators refrained from participating in rounds discussions. Information on patient census, duration of rounds, order of patient presentation, day relative to attending’s time on service, attending EHR use on rounds, patient disposition, and presenter level of training were also collected.
We audited the communication of 26 laboratory tests relevant to the management of critically ill patients (Supplemental Table 1, Supplemental Digital Content 2, http://links.lww.com/CCM/C104). We limited our study to laboratory data given their instant and accurate population of the EHR from the primary system, their overall ubiquitous presence on ICU patients, and the ease of real-time validation of communication accuracy. Specific laboratories were selected to provide broad representation of multiple organ systems and a balance of frequently and less frequently ordered tests. At our institution, trainees are expected to present all new data since the conclusion of rounds the day prior. Thus, all laboratory results from noon the day prior were eligible for analysis. When serial laboratories were present we audited only the most recent set. Using laboratory values displayed within the patient’s EHR on predefined laboratory screens as a gold standard, observers listened for the oral communication of laboratories during rounds. Spoken values were compared with EHR values captured on printed EHR screenshots taken just prior to the beginning of presentations (data collection protocol, Supplemental Digital Content 3, http://links.lww.com/CCM/C105).
Verbal laboratory communication was scored as accurate when any one of the following conditions was met: 1) laboratory value was accurately reported and not described; 2) laboratory value was omitted but accurately described; and 3) laboratory value was accurately reported and described. Laboratories were considered “misrepresented” if they failed to meet any of these criteria. For every laboratory misrepresentation event, we documented whether or not it was detected, and if so, by whom.
Misrepresentations were classified into one of several categories (Table 1). Omissions did not include laboratories described as part of a laboratory set even if not explicitly named. Analysis of the prerounding artifact allowed for further determination of which omissions were due to “artifact creation failure” versus “artifact usage failure.” “Misinterpreted” data were laboratories that were incorrectly described according to the clinical judgment of the observers.
A chi-square test was used to assess for associations between categoric variables and communication accuracy and misrepresentation detection. Relative risk was calculated to assess strength of associations. For continuous variables, linear regression and a Pearson correlation and determination of R2 was performed. A p value of less than 0.05 was considered significant. All data were analyzed with Graphpad Prism (GraphPad Software Inc., La Jolla, CA) and Microsoft Office Excel 2010 (Microsoft Corporation, Redmond, WA).
We observed 34 MICU rounds yielding 301 patient rounding audits (n = 90 phase 1; n = 211 phase 2) and 4,945 laboratory test observations (Supplemental Table 1, Supplementary Digital Content 2, http://links.lww.com/CCM/C104). Interns most commonly presented patients. Eight of eleven attendings regularly viewed the EHR during rounds and 58% of all observations included attending EHR use.
Presenters created and used a paper artifact for 94.3% of presentations. However, the format of artifacts varied (Supplemental Fig. 1, Supplemental Digital Content 2, http://links.lww.com/CCM/C104) both by structure and presenters’ reliance on macros to import data; 82.4% of presenters created an artifact from within the EHR but most of these paper printouts also contained handwritten notes in the margins, and in some cases handwritten duplications of laboratories already present, and 17.6% of artifacts were manually created outside the EHR, most of which were entirely handwritten.
Overall, trainees accurately reported only 61.1% of the laboratories with no observed difference in accuracy between data collection periods (Table 2). Most commonly trainees reported the laboratory value without any description (48.9%); 24.6% of laboratories were only described and 26.5% of laboratories values were both reported and described. Ninty-six percent of patients had at least one inaccurately communicated laboratory for an average of 6.3 laboratory misrepresentations/patient. The majority of misrepresentations were omissions (Fig. 1A). Artifact analysis revealed that 40.3% of omissions were artifact importation failures and 59.7% were artifact usage failures.
The accuracy of laboratory communication varied by individual laboratory tests (Figs. 2 and 3A). The frequency of test ordering correlated strongly with communication accuracy. Infrequently ordered laboratory tests were most likely to be misrepresented (p < 0.00001). Misrepresentations also appeared to cluster by ordering panel such as the blood gas and liver panel tests (Fig. 2).
The prerounding artifact was strongly linked with accurate laboratory communication on rounds. Overall, any laboratory extracted from the EHR and present on the artifact was more likely to be accurately communicated on rounds compared with a laboratory absent from the artifact (p < 0.0001) (Table 1). Additionally, the more consistently a type of laboratory test was found on trainee artifacts, the more likely it was that communication of the same laboratory on rounds was accurate (p < 0.00001) (Fig. 3B). Despite the protective association between artifact use and communication accuracy, artifacts were incomplete and contained only 78.5% of audited laboratories. Yet, the more frequently a laboratory was ordered, the more likely it was to be included on the artifact (R2 = 0.82; p < 0.00001). Whether the presenter used an EHR-generated or manually generated artifact had no correlation with communication accuracy (Table 2).
We next looked at the effect of sociotechnologic factors on laboratory communication accuracy (Table 2; and Supplemental Fig. 2, Supplemental Digital Content 2, http://links.lww.com/CCM/C104). There was no difference in laboratory communication accuracy based on presenter level of training or what day of an attending’s time on service audits occurred. Laboratory communication accuracy was worse when ICU census was greater than 14 patients, when the attending did not use the EHR, on patients presented later in rounds and when presentations lasted more than 20 min/patient.
Although inaccurate laboratory data communication was common, the interprofessional rounding team recognized only 7.8% misrepresentations. Attending physicians accounted for 56.4% of detected misrepresentations followed by nurses (17.6%), fellows (9.5%), residents (9.1%), and pharmacists (7.4%). An exception to poor team recognition of laboratory misrepresentations was the detection of “pending” type misrepresentations (Fig. 1B).
Similar to laboratory communication accuracy on rounds, environmental and human factors had a minor impact on the ICU team’s ability to detect laboratory misrepresentations (Table 2; and Supplemental Fig. 2, Supplemental Digital Content 2, http://links.lww.com/CCM/C104). Teams detected more data misrepresentations later versus earlier in the academic year, on patients presented by medical students compared with residents and on patients transferring out of the ICU. Misrepresentation detection was incrementally worse when the attending did not use the EHR, on patients presented at the end of rounds and when presentations lasted less than 10 min/patient.
In this study, we developed a simple and reproducible methodology to assess the accuracy of data communication on ICU rounds by studying the intersection of what is present in the EHR with what is generated on paper and finally verbalized on ICU rounds. At our institution, despite well-established EHR use and structured interprofessional ICU rounds, we discovered that laboratory misrepresentation was a pervasive phenomenon. It occurred on almost every patient and multiple times within the same presentation, involving nearly 40% of the laboratories studied. These results are consistent with the reports of communication failures during patient handoffs, which are now acknowledged as a universal patient safety issue (11). Thus, we suspect that data misrepresentation on ICU rounds at other institutions will be equally prevalent. This study provides a framework and methodology to facilitate future research.
Disappointingly, despite multiple studies supporting the positive impact of interprofessional ICU rounding, team-based rounding failed to compensate for individual clinician data communication failures (29–32). Furthermore, teams disproportionately relied on the attending physician to detect data misrepresentation. Possible explanations include unequal EHR access for real-time data viewing to recognize errors, inability to simultaneously listen, process, and verify data, individuals’ unwillingness to prolong rounds, deference to physicians, or perhaps a lack of active engagement by other participants on rounds.
Another key finding of our study was that data omissions were the most common form of data misrepresentation. Data omissions by clinicians may be unintentional (incomplete data gathering and processing) or deliberate (selective data communication); yet both have the potential to negatively impact patient care. Some may argue that intentionally culling data from rounds presentations is justified or even desirable because doing so might improve rounding efficiency and reduce clinician data overload. Yet, other team members on rounds may be unable to, in real time and merely by listening to an oral dialogue, distinguish between appropriate data filtering and lapses in data gathering. Furthermore, the problem with allowing individual providers, especially inexperienced trainees, to selectively present and omit data in a nonstandardized way, is that it allows one individual’s cognitive biases to contaminate the entire group, potentially leading to team consensus around a misdiagnosis (33). Finally, the reduction of unnecessary daily tests in the ICU is a goal of the “Choosing Wisely Campaign” (34). If nonessential test results are never brought to the team’s attention on rounds, wasteful test ordering behaviors will likely persist.
The presence of data on presenter artifacts was the strongest predictor of accurate data communication. This suggests that using a prerounding tool as a presentation aid is an effective data management strategy in our ICU where physicians are expected to gather, interpret, and communicate the entire 24-hour dataset on rounds. Continued use of paper artifacts and reliance on a single individual despite the ability to directly view data on computer screens during rounds may represent a failure to incorporate EHR technology. Alternatively, it represents a coping strategy for an EHR system that does not automatically provide an effective visual display of data needed for daily rounds. Our prior work showed that in order to recognize patient safety issues in simulated MICU patient cases, clinicians had to visit over 30 different EHR screens (15,35). Thus, the value of the prerounding artifact may be that it gives clinicians a standardized data collection script and creates a single visual display of all rounding data that are otherwise geographically fragmented within the EHR.
Conversely, artifact collection and analysis highlighted the many limitations of electronically generated prerounding tools. Many trainees printed incomplete daily progress note templates that included automatically imported laboratory fields, which may account for the extremely low number of “erroneous” misrepresentations attributable to transcription errors. However, over 20% of audited laboratory data never appeared on artifacts and these templates consistently lacked data fields for less frequently ordered laboratories. Printouts of electronic prerounding tools are also temporally static and fail to automatically incorporate new data resulting after artifact creation. This explains the observation of “pending” and “old data” misrepresentations comprising 14% of misrepresentations. Prospectively designing and ensuring unit-wide adoption of a single, comprehensive prerounding template might mitigate the lack of standardization in prerounding data gathering. Requiring clinicians to present off an electronic prerounding tool that automatically refreshes new data might reduce communication of outdated information. However, these strategies would require further testing including validation of the efficacy of data importation macros. Additionally, reducing the time clinicians spend gathering data and limiting their ability to freely annotate the artifact might also have negative unintended consequences on cognitive processing.
Some of our findings validate existing interprofessional rounds best practices and highlight vulnerable conditions that warrant additional ICU team vigilance (29). For example, data fidelity was worse when census exceeded 14 patients or after the 14th patient presented, further supporting 14 patients as a critical census threshold beyond which the quality of ICU care declines (36).
Our study has important limitations. First, these data represent the experience of a single ICU at one academic institution and need replication in other institutions with different workflows and rounding paradigms. However, the importance of our study is that for the first time, it highlights the potential extent of data misrepresentations on rounds and the multiple variables that may contribute to this. The simple, low-tech methodology used will allow for each institution, with its own unique technology and workflow, to assess both the frequency of data misrepresentation and the impacts of any modifications made to data gathering or rounding processes. A second limitation is the potential Hawthorne effect of in-person observers. However, if the presence of observers on rounds artificially improves communication, then one would predict that our results actually underestimate the frequency of misrepresentations. Third, our study did not link laboratory miscommunication with patient outcomes. Prospectively it is difficult to predict what data ultimately prove critical to effective decision making on individual patients. However, our methodology could be used in EHR rounds simulation exercises in which the diagnosis and desired clinical decisions are known (37). Finally, we limited our communication audit to a select group of laboratory data; thus, we cannot comment on patterns of miscommunication in other important domains such as vital signs, ventilator data, medications, or imaging.
We developed a methodology to assess ICU physician ability to extract laboratory data from the EHR and accurately present it on interprofessional rounds. At our institution, we found a high frequency of data misrepresentation, especially information omission that was largely unchecked by the rounding team but partially prevented by the use of a prerounding tool. This represents a patient safety issue that may be more widespread than currently recognized and deserves additional study.
1. Burger C: Multidisciplinary rounds: A method to improve quality and safety in critically ill patients. Northeast Florida Med 2007; 58:1619
2. Giri J, Ahmed A, Dong Y, et al.: Daily intensive care unit
rounds: A multidisciplinary perspective. Appl Med Inform 2013; 33:6373
3. Malhotra S, Jordan D, Patel VL: Workflow modeling in critical care: Piecing your own puzzle. AMIA Annu Symp Proc 2005:480484
4. Makary MA, Daniel M: Medical error-the third leading cause of death in the US. BMJ 2016; 353:i2139
5. Zwaan L, Thijs A, Wagner C, et al.: Does inappropriate selectivity in information use relate to diagnostic errors and patient harm? The diagnosis of patients with dyspnea. Soc Sci Med 2013; 91:3238
6. Graber ML, Franklin N, Gordon R: Diagnostic error in internal medicine. Arch Intern Med 2005; 165:14931499
7. Winters B, Custer J, Galvagno SM Jr, et al.: Diagnostic errors in the intensive care unit
: A systematic review of autopsy studies. BMJ Qual Saf 2012; 21:894902
8. Pickering BW, Hurley K, Marsh B: Identification of patient information corruption in the intensive care unit
: Using a scoring tool to direct quality improvements in handover. Crit Care Med 2009; 37:29052912
9. Bomba DT, Prakash R: A description of handover processes in an Australian public hospital. Aust Health Rev 2005; 29:6879
10. Venkatesh AK, Curley D, Chang Y, et al.: Communication of vital signs at emergency department handoff: Opportunities for improvement. Ann Emerg Med 2015; 66:125130
11. Colvin MO, Eisen LA, Gong MN: Improving the patient handoff process in the intensive care unit
: Keys to reducing errors and improving outcomes. Semin Respir Crit Care Med 2016; 37:96106
12. Manor-Shulman O, Beyene J, Frndova H, et al.: Quantifying the volume of documented clinical information in critical illness. J Crit Care 2008; 23:245250
13. Pickering BW, Dong Y, Ahmed A, et al.: The implementation of clinician designed, human-centered electronic medical record viewer in the intensive care unit
: A pilot step-wedge cluster randomized trial. Int J Med Inform 2015; 84:299307
14. McKenzie MS, Auriemma CL, Olenik J, et al.: An observational study of decision making by medical intensivists. Crit Care Med 2015; 43:16601668
15. March CA, Steiger D, Scholl G, et al.: Use of simulation to assess electronic health record safety in the intensive care unit
: A pilot study. BMJ Open 2013; 3:e002549
16. Wiener S: Ward rounds revisited – the validity of the data base. J Med Ed 1976; 49:351356
17. Grundgeiger T, Sanderson P, MacDougall HG, et al.: Interruption management in the intensive care unit
: Predicting resumption times and assessing distributed support. J Exp Psychol Appl 2010; 16:317334
18. Powsner SM, Tufte ER: Graphical summary of patient status. Lancet 1994; 344:386389
19. Tange HJ: The paper-based patient record: Is it really so bad? Comput Methods Programs Biomed 1995; 48:127131
20. Perez T, Bass EJ, Helms A, et al.: Comparison of patient data in parallel records: The sign-out sheet and the electronic medical record. In: IEEE International Conference on Systems, Man and Cybernetics: 2010. 2010, Istanbul, Turkey, Kudret Press & Digital Printing Co., pp 18841888
21. Gurses AP, Xiao Y: A systematic review of the literature on multidisciplinary rounds to design information technology. J Am Med Inform Assoc 2006; 13:267276
22. Van Eaton EG, McDonough K, Lober WB, et al.: Safety of using a computerized rounding and sign-out system to reduce resident duty hours. Acad Med 2010; 85:11891195
23. Black R, Woolman P, Kinsella J: Variation in the transcription of laboratory data in an intensive care unit
. Anaesthesia 2004; 59:767769
24. Thompson G, O’Horo JC, Pickering BW, et al.: Impact of the electronic medical record on mortality, length of stay, and cost in the hospital and ICU: A systematic review and metaanalysis. Crit Care Med 2015; 43:12761282
25. Han YY, Carcillo JA, Venkataraman ST, et al.: Unexpected increased mortality after implementation of a commercially sold computerized physician order entry system. Pediatrics 2005; 116:15061512
26. Warden GL, Bagian JP: Health IT and Patient Safety: Builder Safer Systems for Better Care. 2011Washington, DC, Institute of Medicine, National Academies Press
27. Lowry SZ, Quinn MT, Ramaiah M, et al.: NISTIR 7804. Technical Evaluation, Testing and Validation of Usability of Electronic Health Records. Edited by National Institute of Standards and Technology. 2012, Washington, DC, U.S. Department of Commerce, pp 1108
28. Rothschild JM, Landrigan CP, Cronin JW, et al.: The Critical Care Safety Study: The incidence and nature of adverse events and serious medical errors in intensive care. Crit Care Med 2005; 33:16941700
29. Lane D, Ferri M, Lemaire J, et al.: A systematic review of evidence-informed practices for patient care rounds in the ICU. Crit Care Med 2013; 41:20152029
30. Leape LL, Cullen DJ, Clapp MD, et al.: Pharmacist participation on physician rounds and adverse drug events in the intensive care unit
. JAMA 1999; 282:267270
31. Young MP, Gooder VJ, Oltermann MH, et al.: The impact of a multidisciplinary approach on caring for ventilator-dependent patients. Int J Qual Health Care 1998; 10:1526
32. Kim MM, Barnato AE, Angus DC, et al.: The effect of multidisciplinary care teams on intensive care unit
mortality. Arch Intern Med 2010; 170:369376
33. Ogdie AR, Reilly JB, Pang WG, et al.: Seen through their eyes: Residents’ reflections on the cognitive and contextual components of diagnostic errors in medicine. Acad Med 2012; 87:13611367
34. Halpern SD, Becker D, Curtis JR, et al.; Choosing Wisely Taskforce; American Thoracic Society; American Association of Critical-Care Nurses; Society of Critical Care Medicine: An official American Thoracic Society/American Association of Critical-Care Nurses/American College of Chest Physicians/Society of Critical Care Medicine policy statement: The Choosing Wisely® Top 5 list in Critical Care Medicine. Am J Respir Crit Care Med 2014; 190:818826
35. Gold JA, Stephenson LE, Gorsuch A, et al.: Feasibility of utilizing a commercial eye tracker to assess electronic health record use during patient simulation. Health Informatics J 2016; 22:744757
36. Neuraz A, Guérin C, Payet C, et al.: Patient mortality is associated with staff resources and workload in the ICU: A multicenter observational study. Crit Care Med 2015; 43:15871594
37. Stephenson LS, Gorsuch A, Hersh WR, et al.: Participation in EHR based simulation improves recognition of patient safety issues. BMC Med Educ 2014; 14:224