Is the Promise of PROMs Being Realized? Implementation Experience in a Large Orthopedic Practice : American Journal of Medical Quality

Secondary Logo

Journal Logo


Is the Promise of PROMs Being Realized? Implementation Experience in a Large Orthopedic Practice

Ziegenfuss, Jeanette Y. PhD1; Grossman, Elizabeth S. MPH2; Solberg, Leif I. MD2; Chrenka, Ella A. MS2; Werner, Ann BS2; Asche, Stephen E. MA2; Norton, Christine K. MA3; Nelson, Andy MPH2; Reams, Megan OT4; Whitebird, Robin R. PhD5

Author Information
doi: 10.1097/JMQ.0000000000000079
  • Free



Patient-reported outcome measures (PROMs) are increasingly being collected as part of care, especially in orthopedics. PROMS are designed to assess patient-centered health outcomes such as quality of life, functional status, mental health, health symptoms, and experience of care.1 The primary use of PROMs thus far has involved their aggregate use for public reporting of outcomes at the provider or care delivery system level to inform patient choice or drive local quality improvement initiatives. Evidence from trials suggests well-implemented, systematic use of PROMs can improve patient-provider communication and patient satisfaction,2–4 as well as monitoring of treatment response and detection of unrecognized problems.2 Routine use of PROMS has also been shown to increase the frequency of discussion of patient outcomes during consultations, improve symptom control, patient satisfaction,5 and decision making at the patient level.6–10

The potential for using health information technology to advance the implementation of PROMs is central to improving the delivery of patient-centered care and increasing patient engagement, but challenges still exist in workflow, electronic data collection, integration in the electronic health record (EHR), and creating user-friendly data displays.11 Furthermore, successful integration of PROMS data in clinical workflows in health care systems would require substantial investment in time and resources to engage clinicians in the use of the data12 and extensive planning, infrastructure development, and ongoing monitoring of improvement processes in clinical team settings.8 Traditionally, integrated electronic platforms for collecting PROMS in orthopedics have been implemented primarily in academic medical centers for epidemiological and research purposes13–15 with varying degrees of PROMS completion rates.16

While there is agreement that PROMs have the potential to guide clinical decision-making and enhance preoperative and postoperative counseling in orthopedic surgery and beyond,13,17,18 their integration is complex and dependent on multilevel factors.19 There is little existing empirical evidence with respect to successful integration.20,21 This case study describes the extent to which PROMs are used by surgical clinical care teams when they are systematically collected and displayed in the EHR for patients undergoing total hip or knee replacement. Understanding their use in a system where the seemingly large barriers of technological integration and patient receptivity to survey completion have been overcome, presents a best-case scenario for understanding the realization of the promise of PROMs without further intervention.


PROMs Initiative

In 2014, leaders of a large Midwestern health system with vertically integrated health care delivery and insurance began an initiative to implement a long-term, organization-wide PROMs system. The medical group has 1800 multispecialty physicians working in over 90 clinic and hospital sites throughout the Minneapolis-St. Paul metropolitan area and western Wisconsin. During the initial phase, organizational leaders supported a pilot project in its 58-surgeon orthopedics department to systematically collect PROMs from patients and make them available to physicians in the EHRs for use during clinical care. After demonstrating feasibility in that setting, similar methods would then be spread across the care delivery system. This description of that natural experiment was completed as part of a learning health system study funded by the Agency for Healthcare Research and Quality and conducted in collaboration with health system orthopedic leadership to foster and assess the integration of PROMs in practice and is reported in part elsewhere (cites).22,23 This project is called PROM Optimization through Technology and Engagement (PROMOTE) and it has been reviewed, approved, and monitored by the health system’s affiliated Institutional Review Board.

PROMs Data Collection

The health care system has systematically collected PROMs in orthopedic care for public quality measure reporting for all total knee arthroplasty (TKA) and total hip arthroplasty (THA) cases since April 2017. PROMs measures used include the Oxford Knee/Hip Score,24 the PROMIS 10,25 the KOOS JR (7 questions), and the/HOOS JR (6 questions).26,27 Two Consumer Assessment of Healthcare Provider and Systems overall satisfaction questions and an adaptation of the 3-item CollaboRATE Shared Decision-Making scale28 were added in April 2018 as part of the PROMOTE study. Initial measures are collected preoperatively at the time that surgery is scheduled. Follow-up surveys are sent at 3 months (survey window: 9–20 wk post-surgery) and 1 year (survey window: 9–15 mo post-surgery).

Patients with an email address on record (approximately 70% for knee and 67% for hip surgery) are sent an initial baseline survey invitation upon scheduling their TKA or THA. The email is sent from a health system-specific outcomes email address with the subject line “[clinic name] Hip (or Knee) Replacement Initial Outcome Questionnaire.” Initial nonresponders are sent up to 2 reminder emails 5 days apart until they complete the survey, actively refuse, or come into an orthopedic clinic for a visit. If the patient has not completed a survey when they arrive at the clinic for an appointment, the front desk staff provides them with a tablet and their personalized survey link queued up so they can complete the PROMs survey while waiting.

PROM Data Made Available in the Electronic Medical Record

Beginning in August 2018, the organization integrated individual patient-level outcomes into the electronic medical record and systematically made the data available to the care team for patient care, quality improvement, and research. In 2019, at the request of orthopedic leadership, reports were developed by the research team to monitor how often PROMs data were accessed in the electronic medical record by provider and surgery type, reported here in aggregate. To encourage use of PROMs data in tandem with or during visits, orthopedics leadership utilized various strategies to generate awareness. These included sending email communications to total joint surgeons from the Medical Director of Joint Care Services with educational information about the PROMs instruments used for hip and knee patients, scoring details for each measure, and a tip sheet including how to access patient outcome data in the electronic medical record for use during care delivery. Department administrative leaders systematically held individual conversations with surgeons to demonstrate how to access and use PROMs in the EHR. Follow-up email communication from Medical Director Leadership reinforced messages about availability and encouraged use while welcoming surgeons’ suggestions about what else they needed to be able to access and use the data.


For this analysis, patients selected were all scheduled with surgeries from January 2019 to March 2020. Data was drawn in August 2021, after the last allowable survey completion date of June 30, 2021. This 15-month period represents the months between stabilization of the initial roll out of EHR integration in August 2018 through the time before elective procedures were temporarily discontinued in this health system due to the onset of the COVID-19 pandemic. Patients were presented with the opportunity to complete a PROMs questionnaire at up to 3-time points relative to their procedure as described above (preoperatively and 3 and 12 mo postoperatively). All surgeons and other clinical staff associated with these patients’ care within the care delivery system were able to view the patient-level responses and scores in the EHR at any point before, during, or after a patient encounter.


Over this period, response rates for each procedure and time point were calculated as the total number of submitted surveys divided by the number of initial survey invitations (American Associate for Public Opinion Research maximum response rate, response rate 6).29 For those patients that had more than 1 surgery scheduled, only the first procedure in the date range is included in describing response rates and patient characteristics.

EHR view rates of PROMs are calculated on an encounter level and aggregated by month. All orthopedic-associated encounters for patients that had scheduled or completed a total hip or knee replacement surgery in the 15 months between January 2019 and March 2020 were identified and multiple scheduled surgeries were allowed per patient. Any view of the PROMs data within the EHR in the 5 days before or after the encounter by a physician, physician assistant, nurse, medical assistant, athletic trainer, scribe, or tech was defined as a PROM view and reported as the percentage of encounters with a PROM view. The number of views by attributed clinician and professional role was also documented.


During this 15-month period, 2400 TKA surgeries were scheduled for 22 individual surgeons, an average of 109 TKAs/surgeon. Similarly, 1545 THA surgeries were scheduled for 20 individual surgeons an average of 75 THAs/surgeon. Over 16 000 encounters were associated with these surgeries attributed to 26 unique surgeons with an average of 4.2 encounters per surgery. PROMs data was viewed in the EHR a total of 156 times with monthly rates of views as a portion of encounters ranging from 0% to 2.8%, averaging views for only 0.9% of all encounters (Figure). Of the 156 views over this period, the majority were by physicians (71%), followed by athletic trainers, physician assistants, nurses, and scribes that accounted for an additional 5%–6% of the views each (Table 1). Seventy-seven percent of the 156 views were attributed to encounters related to just 4 surgeons. Of the 26 individual surgeons, 11 had no PROMs data views and 3 had just one view associated with attributed encounters.

Table 1. - Total Views of PROMs Data by Role, January 2019–March 2020.
Clinical role n (%)
Athletic trainer 8 (5)
Licensed practical nurse 1 (1)
Medical assistant 6 (4)
Orthopedic assistant 1 (1)
Physician 111 (71)
Physician assistant 10 (6)
Registered nurse 9 (6)
Scribe 9 (6)
Tech 1 (1)
Total 156
Abbreviation: PROMs, patient-reported outcome measures.

Total number of THA and TKAs associated with encounters by month and associated PROMs data view rates. Abbreviations: PROMs, patient-reported outcome measures; THA, total hip arthroplasty; TKA, total knee arthroplasty.

Considering just one record per patient (n = 3389) and removing canceled surgeries and those rescheduled outside of the window considered in this analysis, the overall patient survey response rates were 68% at baseline, 58% at 3 months, and 55% at 12 months. At baseline, the average patient age was 65.7, 58% were female, 89% were White, and 1.4% Hispanic. About half of these patients were commercially insured with 6% and 45%, respectively, having Medicaid and Medicare as a payor (see Table 2).

Table 2. - PROMs Response Rates and Patient Demographic Characteristics, Completed TKA and THA, January 2019 to March 2020.
All surgeries, January 2019 to March 2020
Total TKA and THA 3389
PROMs response rate
 Baseline 68%
 3 mo 58%
 12 mo 55%
Age 65.7 ± 9.9
Sex—female 57.8%
 Black/AA 89.2%
 Asian 6.4%
 American Indian/Alaska 0.7%
 Native 0.5%
 Native Hawaiian/PI 0.1%
 Some other 0.5%
 More than 1 0.2%
 Unknown 2.4%
Ethnicity—Hispanic 1.3%
Medicaid 5.8%
Medicare 44.2%
Abbreviations: PROMs, patient-reported outcome measures; THA, total hip arthroplasty; TKA, total knee arthroplasty.


PROMs survey response rates were relatively high for total hip and knee angioplasties and the data were successfully integrated into the EHR for use as part of patient care. However, the use of these PROMs data among surgeons and care teams was nearly nonexistent and largely clustered by surgeon, suggesting that a critical step in the pathway to the promise of using PROMs to improve the individual patient encounter was not being realized. Thus, a system-wide approach that is championed by leadership, fully integrated into clinic systems, and embraced by patients as indicated by high response rates, was not sufficient for full integration of PROMs in practice.

While these findings are limited to 2 types of surgeries in 1 department within a single health system, being from a care system and state that have a long history of supporting measurement and quality reporting, it in many ways may represent a best-case scenario. If the use of PROMs for clinical care was not taken up here, this problem might be even more likely to be present in other systems without these supporting factors, suggesting the need to create a system to elevate the data’s use, operationalize the organizational priority for PROMs, and increase the value of measurable patient engagement. Others have found that time to discuss PROMs and their potential redundancy with other tools to assess patient concerns could be barriers to use.30 An implementation science framework may best guide and organize intervention to drive change in use.19 Specifically, the Consolidated Framework for Implementation Research with its 5 multilevel domains (the intervention, outer setting, inner setting, characteristics of the individuals, and the implementation process itself) is useful for PROMs.19,31 The study has since implemented interviews with clinical staff and surgeons about the use of individual-level PROMs data during encounters to better understand the barriers and facilitators to use. Logistical barriers and lack of time as well as concerns about validity and reliability of measures and that patients may not understand the scores were reported in those interviews.32 Without system-level change to integrate use in care and help ensure that the resources invested at both the care system and patient levels are optimized, the use of PROMs to improve clinical care might be better off abandoned, while aggregate data can still be used to document and improve overall care approaches. Worse, if pursued for clinical care integration and not universally completed or discussed, inconsistent and limited use could further introduce disparities in the provision of care. It is important to distinguish this clinical use from the use of PROMs to support quality reporting, which can still be accomplished without clinician action.


Patient-reported outcome measures responders, providers.

Conflicts of Interest

Dr Ziegenfuss, Ms Grossman, Dr Solberg, Ms Chrenka, Ms Werner, Mr Asche, Mr Nelson, and Ms Reams report a potential conflict of interest as employees of a care system/health plan interested in this information. All the other authors have no conflicts of interest to disclose.


Dr Solberg received financial support for the research, authorship, and/or publication of this article from grant number R18HS025618 from the Agency for Healthcare Research and Quality.

Author Contributions

All authors contributed to discussion from which the scope of the article was determined. Drs Ziegenfuss and Solberg wrote the article. Ms Werner acquired data. Ms Chrenka and Mr Asche analyzed data. Dr Solberg, Ms Norton, Mr Nelson, Ms Reams, and Ms Whitebird provided critical review and all authors approved the final article. Dr Solberg obtained funding for this study.


The content is solely the responsibility of the authors and does not necessarily represent the official views of the Agency for Healthcare Research and Quality.


1. Weldring T, Smith SM. Patient-reported outcomes (PROs) and patient-reported outcome measures (PROMs). Health Serv Insights. 2013;6:61–68.
2. Chen J, Ou L, Hollis SJ. A systematic review of the impact of routine collection of patient reported outcome measures on patients, providers and health organisations in an oncologic setting. BMC Health Serv Res. 2013;13:211.
3. Santana MJ, Feeny D. Framework to assess the effects of using patient-reported outcome measures in chronic care management. Qual Life Res. 2014;23:1505–1513.
4. Valderas JM, Kotzeva A, Espallargues M, et al. The impact of measuring patient-reported outcomes in clinical practice: a systematic review of the literature. Qual Life Res. 2008;17:179–193.
5. Kotronoulas G, Kearney N, Maguire R, et al. What is the value of the routine use of patient-reported outcome measures toward improvement of patient outcomes, processes of care, and health service outcomes in cancer care? A systematic review of controlled trials. J Clin Oncol. 2014;32:1480–1501.
6. Baumhauer JF. Patient-reported outcomes - are they living up to their potential? N Engl J Med. 2017;377:6–9.
7. Ehlers AP, Khor S, Cizik AM, et al. Use of patient-reported outcomes and satisfaction for quality assessments. Am J Manag Care. 2017;23:618–622.
8. Gerhardt WE, Mara CA, Kudel I, et al. Systemwide implementation of patient-reported outcomes in routine clinical care at a children’s hospital. Jt Comm J Qual Patient Saf. 2018;44:441–453.
9. Oliver BJ, Nelson EC, Kerrigan CL. Turning feed-forward and feedback processes on patient-reported data into intelligent action and informed decision-making: case studies and principles. Med Care. 2019;57(suppl 5 suppl 1):S31–S37.
10. Chrenka EA, Solberg LI, Asche SE, et al. Is shared decision-making associated with better patient-reported outcomes? A longitudinal study of patients undergoing total joint arthroplasty. Clin Orthop Relat Res. 2022;480:82–91.
11. Hsiao CJ, Dymek C, Kim B, et al. Advancing the use of patient-reported outcomes in practice: understanding challenges, opportunities, and the potential of health information technology. Qual Life Res. 2019;28:1575–1583.
12. Foster A, Croot L, Brazier J, et al. The facilitators and barriers to implementing patient reported outcome measures in organisations delivering health related services: a systematic review of reviews. J Patient Rep Outcomes. 2018;2:46.
13. Lizzio VA, Dekhne MS, Makhni EC. Electronic patient-reported outcome collection systems in orthopaedic clinical practice. JBJS Rev. 2019;7:e2.
14. Zwiers R, Weel H, Mallee WH, et al.; Ankle Platform Study Collaborative – Science of Variation Group. Large variation in use of patient-reported outcome measures: a survey of 188 foot and ankle surgeons. Foot Ankle Surg. 2018;24:246–251.
15. Hjollund NHI, Valderas JM, Kyte D, et al. Health data processes: a framework for analyzing and discussing efficient use and reuse of health data with a focus on patient-reported outcome measures. J Med Internet Res. 2019;21:e12412.
16. Gold HT, Karia RJ, Link A, et al. Implementation and early adaptation of patient-reported outcome measures into an electronic health record: a technical report. Health Informatics J. 2020;26:129–140.
17. Mou D, Sisodia RC, Castillo-Angeles M, et al. The surgeon’s perceived value of patient-reported outcome measures (PROMs): an exploratory qualitative study of 5 different surgical subspecialties. Ann Surg. 2022;275:500–505.
18. Forsberg HH, Nelson EC, Reid R, et al. Using patient-reported outcomes in routine practice: three novel use cases and implications. J Ambul Care Manage. 2015;38:188–195.
19. Stover AM, Haverman L, van Oers HA, et al.; ISOQOL PROMs/PREMs in Clinical Practice Implementation Science Work Group. Using an implementation science approach to implement and evaluate patient-reported outcome measures (PROM) initiatives in routine care settings. Qual Life Res. 2021;30:3015–3033.
20. Briffa N. The employment of patient-reported outcome measures to communicate the likely benefits of surgery. Patient Relat Outcome Meas. 2018;9:263–266.
21. Davidson GH, Haukoos JS, Feldman LS. Practical guide to assessment of patient-reported outcomes. JAMA Surg. 2020;155:432–433.
22. Solberg LI, Chrenka E, Asche SE, et al. Adjusting for variation in patient-reported outcome measures is needed to improve care after total knee arthroplasty. J Am Acad Orthop Surg. 2021;30:e164–e172.
23. Whitebird RR, Solberg LI, Ziegenfuss JY, et al. Personalized outcomes for hip and knee replacement: the patients point of view. J Patient Rep Outcomes. 2021;5:116.
24. Oxford Hip Score. Threshold for Hip Replacement. Accessed March 7, 2022.
25. CODE Technology. PROMIS Global-10. Accessed March 7, 2022.
26. Hospital for Special Surgery. Knee injury and Osteoarthritis Outcome Score for Joint Replacement (KOOS, JR.). English version 1.0. Accessed March 7, 2022.
27. Hospital for Special Surgery. Hip dysfunction and Osteoarthritis Outcome Score for Joint Replacement (HOOS, JR.). English version 1.0. Accessed March 7, 2022.
28. Barr PJ, Thompson R, Walsh T, et al. The psychometric properties of CollaboRATE: a fast and frugal patient-reported measure of the shared decision-making process. J Med Internet Res. 2014;16:e2.
29. The American Association for Public Opinion Research. Standard Definitions: Final Dispositions of Case Codes and Outcome Rates for Surveys, 9th Edition. Accessed March 7, 2022.
30. Philpot LM, Barnes SA, Brown RM, et al. Barriers and benefits to the use of patient-reported outcome measures in routine clinical care: a qualitative study. Am J Med Qual. 2018;33:359–364.
31. Damschroder LJ, Aron DC, Keith RE, et al. Fostering implementation of health services research findings into practice: a consolidated framework for advancing implementation science. Implement Sci. 2009;4:50.
32. Whitebird RR, Solberg LI, Ziegenfuss JY, et al. What do orthopaedists believe is needed for incorporating patient-reported outcome measures into clinical care? A qualitative study. Clin Orthop Relat Res. 2021;480:680–687.

implementation science; orthopedics; patient-reported outcome measures; quality improvement; surveys and questionnaires

© 2022 the American College of Medical Quality