Objectives: Numerous studies show that follow-up of abnormal cancer screening results, such as mammography and Papanicolaou (Pap) smears, is frequently not performed in a timely manner. A contributing factor is that abnormal results may go unrecognized because they are buried in free-text documents in electronic medical records (EMRs), and, as a result, patients are lost to follow-up. By identifying abnormal results from free-text reports in EMRs and generating alerts to clinicians, natural language processing (NLP) technology has the potential for improving patient care. The goal of the current study was to evaluate the performance of NLP software for extracting abnormal results from free-text mammography and Pap smear reports stored in an EMR.
Methods: A sample of 421 and 500 free-text mammography and Pap reports, respectively, were manually reviewed by a physician, and the results were categorized for each report. We tested the performance of NLP to extract results from the reports. The 2 assessments (criterion standard versus NLP) were compared to determine the precision, recall, and accuracy of NLP.
Results: When NLP was compared with manual review for mammography reports, the results were as follows: precision, 98% (96%–99%); recall, 100% (98%–100%); and accuracy, 98% (96%–99%). For Pap smear reports, the precision, recall, and accuracy of NLP were all 100%.
Conclusions: Our study developed NLP models that accurately extract abnormal results from mammography and Pap smear reports. Plans include using NLP technology to generate real-time alerts and reminders for providers to facilitate timely follow-up of abnormal results.
From the *Division of General Medicine and Clinical Epidemiology, Department of Medicine, School of Medicine, †The North Carolina Translational and Clinical Sciences Center, and ‡Department of family Medicine, School of Medicine, University of North Carolina, Chapel Hill, North Carolina.
Correspondence: Carlton R. Moore, MD, MS, Division of General Medicine and Clinical Epidemiology, Department of Medicine, School of Medicine, University of North Carolina, 4023 Old Clinic Bldg, Campus Box 7110, Chapel Hill, NC 27599-7110 (e-mail: firstname.lastname@example.org).
The authors disclose no conflict of interest.
Supported by National Institutes of Health grant 1UL1TR001111 through the North Carolina Translational and Clinical Sciences Institute.