Institutional members access full text with Ovid®

Share this article on:

Feasibility of Real-Time Satisfaction Surveys Through Automated Analysis of Patients' Unstructured Comments and Sentiments

Alemi, Farrokh PhD; Torii, Manabu PhD; Clementz, Laura MA; Aron, David C. MD, MS

Quality Management in Healthcare: January/March 2012 - Volume 21 - Issue 1 - p 9–19
doi: 10.1097/QMH.0b013e3182417fc4
Original Articles

This article shows how sentiment analysis (an artificial intelligence procedure that classifies opinions expressed within the text) can be used to design real-time satisfaction surveys. To improve participation, real-time surveys must be radically short. The shortest possible survey is a comment card. Patients' comments can be found online at sites organized for rating clinical care, within e-mails, in hospital complaint registries, or through simplified satisfaction surveys such as “Minute Survey.” Sentiment analysis uses patterns among words to classify a comment into a complaint, or praise. It further classifies complaints into specific reasons for dissatisfaction, similar to broad categories found in longer surveys such as Consumer Assessment of Healthcare Providers and Systems. In this manner, sentiment analysis allows one to re-create responses to longer satisfaction surveys from a list of comments. To demonstrate, this article provides an analysis of sentiments expressed in 995 online comments made at the Web site. We focused on pediatrician and obstetrician/gynecologist physicians in District of Columbia, Maryland, and Virginia. We were able to classify patients' reasons for dissatisfaction and the analysis provided information on how practices can improve their care. This article reports the accuracy of classifications of comments. Accuracy will improve as the number of comments received increases. In addition, we ranked physicians using the concept of time-to-next complaint. A time-between control chart was used to assess whether time-to-next complaint exceeded historical patterns and therefore suggested a departure from norms. These findings suggest that (1) patients' comments are easily available, (2) sentiment analysis can classify these comments into complaints/praise, and (3) time-to-next complaint can turn these classifications into numerical benchmarks that can trace impact of improvements over time. The procedures described in the article show that real-time satisfaction surveys are possible.

Department of Health Systems Administration (Dr Alemi) and Imaging Science and Information Systems Center (Dr Torii), Georgetown University, Washington, District of Columbia; and VA HSR&D Quality Enhancement Research Initiative Center for Implementation Practice & Research Support (Ms Clementz and Dr Aron), Louis Stokes Cleveland Department of Veterans Affairs Medical Center, Cleveland, Ohio.

Correspondence: Farrokh Alemi, PhD, Department of Health Systems Administration, Georgetown University, 3700 Reservoir Rd, Washington, DC 20007 (

F.A. started Rapid Improvement Inc, a company focused on providing real-time satisfaction surveys using time to next complaint and sentiment analysis. L.C. and D.A. received grant support from VA Health Services Research and Development Service Quality Improvement Research Initiative. The authors extend special thanks to Laura Johnson, who assisted in preparation of the data. Contents of this article do not necessarily represent the views of the Department of Veterans Affairs or the US government.

Authors declare no other conflict of interest.

©2012Lippincott Williams & Wilkins, Inc.