Institutional members access full text with Ovid®

Share this article on:

Assessment of Automating Safety Surveillance From Electronic Health Records: Analysis for the Quality and Safety Review System

Fong, Allan*; Adams, Katharine*; Samarth, Anita; McQueen, Laura; Trivedi, Manan; Chappel, Tahleah; Grace, Erin; Terrillion, Susan; Ratwani, Raj, M.*

doi: 10.1097/PTS.0000000000000402
Original Article: PDF Only

Background and Objectives In an effort to improve and standardize the collection of adverse event data, the Agency for Healthcare Research and Quality is developing and testing a patient safety surveillance system called the Quality and Safety Review System (QSRS). Its current abstraction from medical records is through manual human coders, taking an average of 75 minutes to complete the review and abstraction tasks for one patient record. With many healthcare systems across the country adopting electronic health record (EHR) technology, there is tremendous potential for more efficient abstraction by automatically populating QSRS. In the absence of real-world testing data and models, which require a substantial investment, we provide a heuristic assessment of the feasibility of automatically populating QSRS questions from EHR data.

Methods To provide an assessment of the automation feasibility for QSRS, we first developed a heuristic framework, the Relative Abstraction Complexity Framework, to assess relative complexity of data abstraction questions. This framework assesses the relative complexity of characteristics or features of abstraction questions that should be considered when determining the feasibility of automating QSRS. Questions are assigned a final relative complexity score (RCS) of low, medium, or high by a team of clinicians, human factors, and natural language processing researchers.

Results One hundred thirty-four QSRS questions were coded using this framework by a team of natural language processing and clinical experts. Fifty-five questions (41%) had high RCS and would be more difficult to automate, such as “Was use of a device associated with an adverse outcome(s)?” Forty-two questions (31%) had medium RCS, such as “Were there any injuries as a result of the fall(s)?" and 37 questions (28%) had low RCS, such as “Did the patient deliver during this stay?" These results suggest that Blood and Hospital Acquired Infections–Clostridium Difficile Infection (HAI-CDI) modules would be relatively easier to automate, whereas Surgery and HAI–Surgical Site Infection would be more difficult to automate.

Conclusions Although EHRs contain a wealth of information, abstracting information from these records is still very challenging, particularly for complex questions, such as those concerning patient adverse events. In this work, we developed a heuristic framework, which can be applied to help guide conversations around the feasibility of automating QSRS data abstraction. This framework does not aim to replace testing with real data but complement the process by providing initial guidance and direction to subject matter experts to help prioritize, which abstraction questions to test for feasibility using real data.

From the *National Center for Human Factors in Healthcare, Medstar Health; †Clinovations; and ‡Agency for Healthcare Research and Quality, U.S. Department of Health and Human Services, Washington, District of Columbia.

Correspondence: Allan Fong, MS, Research Scientist, 3007 Tilden St, NW Suite 7L, Washington, DC 20008 (e-mail:

The authors disclose no conflict of interest.

This study is funded under the Contract Number HHSA290201400008I/Task Order #1, from the Agency for Healthcare Research and Quality, U.S. Department of Health and Human Services (DHHS).

The authors of this article are responsible for its contents. Statements in the article should not be construed as an endorsement by Agency for Healthcare Research and Quality or the U.S. Department of Health and Human Services.

Identifiable information, on which this report, presentation, or other form of disclosure is based, is confidential and protected by federal law, Section 903(c) of the Public Health Service Act, 42 USC 299a-1(c). Any identifiable information that is knowingly disclosed is disclosed solely for the purpose for which it has been supplied. No identifiable information about any individual supplying the information or described in it will be knowingly disclosed except with the prior consent of that individual.

Copyright © 2018 Wolters Kluwer Health, Inc. All rights reserved