Quantitative Patient-Reported Experience Measures Derived From Natural Language Processing Have a Normal Distribution and No Ceiling Effect

Abstract

Background and Objectives: 

Patient-reported experience measures have the potential to guide improvement in health care delivery. Many patient-reported experience measures are limited by the presence of strong ceiling effects that limit their analytical utility.

Methods: 

We used natural language processing to develop 2 new methods of evaluating patient experience using text comments and associated ordinal and categorical ratings of willingness to recommend from 1390 patients receiving specialty or nonspecialty care at our offices. One method used multivariable analysis based on linguistic factors to derive a formula to estimate the ordinal likelihood to recommend. The other method used the meaning extraction method of thematic analysis to identify words associated with categorical ratings of likelihood to recommend with which we created an equation to compute an experience score. We measured normality of the 2 score distributions and ceiling effects.

Results: 

Spearman rank-order correlation analysis identified 36 emotional and linguistic constructs associated with ordinal rating of likelihood to recommend, 9 of which were independently associated in multivariable analysis. The calculation derived from this model corresponded with the original ordinal rating with an accuracy within 0.06 units on a 0 to 10 scale. This score and the score developed from thematic analysis both had a relatively normal distribution and limited or no ceiling effect.

Conclusions: 

Quantitative ratings of patient experience developed using natural language processing of text comments can have relatively normal distributions and no ceiling effect.