Secondary Logo

Journal Logo

Patient Outcomes in Orthopaedic Trauma

How to Evaluate if Your Treatment Is Really Working?

Nauth, Aaron MD, MSc*; Wasserstein, David MD, MSc, MPH, FRCSC; Tornetta, Paul III MD; Cole, Peter A. MD§,║,¶; Obremskey, William T. MD**; Attum, Basem MD, MS**,††; Slobogean, Gerard P. MD‡‡

Journal of Orthopaedic Trauma: June 2019 - Volume 33 - Issue - p S20–S24
doi: 10.1097/BOT.0000000000001470
Supplement Article

Summary: Outcomes are critical to gauge the success of our treatments and, in particular, surgical interventions in orthopaedic trauma. Patient-reported outcomes have evolved to become the primary measurement of success in surgery. This article reviews the concepts relevant to understanding these outcomes including general health outcomes, extremity- and disease-specific outcomes, minimum clinically important difference, economic analysis of treatment cost/benefit, and the impact of psychosocial factors on outcomes. An understanding of these concepts is important to allow for effective interpretation and critical analysis of the literature as well as to facilitate the practice of evidence-based medicine.

*Orthopaedic Division, Department of Surgery, St. Michael's Hospital, Division of Orthopaedic Surgery, University of Toronto, Toronto, ON, Canada;

University of Toronto, Toronto, ON, Canada;

Boston University Medical Center, Boston, MA;

§HealthPartners Medical Group, Bloomington, MN;

Orthopaedic Department, Regions Hospital, St. Paul, MN;

University of Minnesota, Minneapolis, MN;

**Vanderbilt University Medical Center, Nashville, TN;

††Department of Orthopedic Surgery, UCSD, San Diego, CA; and

‡‡Department of Orthopaedics, University of Maryland School of Medicine, Baltimore, MD.

Reprints: Aaron Nauth, MD, MS, Orthopaedic Division, Department of Surgery, St. Michael's Hospital, University of Toronto, 55 Queen St, Suite 800, Toronto, ON, Canada M5C 1R6 (e-mail:

P. Tornetta receives royalties from Smith Nephew and Wolters Kluwer; P. A. Cole receives institutional research grants from Depuy Synthes and Stryker, institutional educational grants from COTA, AONA, OMeGA, Stryker, Zimmer Biomet, Acumed, Depuy Synthes, and ownership interest in BoneFoam, LLC. The remaining authors report no conflict of interest.

Accepted February 15, 2019

Back to Top | Article Outline


Patient-reported outcomes (PROs) are critical to frame our understanding of how well our treatments work. All outcome measures must meet certain burdens to be considered useful. They should be consistent and reproducible while providing information that is both prognostic and relevant. PROs may fall into the realm of general health, disease-specific measures, or anatomic measures. Examples of general health measures are the 36-Item Short Form Health Survey (SF-36), the Patient-Reported Outcomes Measurement Information System (PROMIS) Global-10, or the EQ-5D.1 These scores may be affected by multiple disease states or even physiologic considerations. Pairing these measures with anatomic or disease-specific measures, such as the Western Ontario and McMaster Universities Osteoarthritis Index score or the Short Musculoskeletal Functional Assessment, provides the best overall assessment of an individual's outcome. Only then can we understand how well a patient is functioning, and what true impact their injury and treatment has had. As an example, in the Study to Prospectively Evaluate Reamed Intramedullary Nails in Patients with Tibial Fractures (SPRINT), some components of general health were unaffected after recovery from a tibial fracture, whereas the Short Musculoskeletal Functional Assessment and bother index demonstrated persistent and substantial disability at 1 year.2

Many typical anatomic measures, such as the American Orthopaedic Foot and Ankle Society score and the Majeed3 score (pelvic outcome score), are summed scores that add values in different arenas such as ability to walk, range of motion of a joint, and pain. The resultant score is then used as the overall assessment. These types of scores can be problematic in several ways. First, they are subject to floor and ceiling effects. By example, patients with a bad pelvic fracture who has minimal pain in their pelvis and a good outcome from their pelvic injury but cannot walk due to a severe tibial plateau fracture might have the same Majeed score as patients with substantial pelvic pain and poor outcome from their pelvic injury. Similarly, when patients have a poor outcome, additional problems will not necessarily move them further down on a summed score. Finally, measures that have a subscale for pain may be unduly influenced by this one parameter. This has been shown in both lower- and upper-extremity fractures where as much as 80% of the variation in summed scores has been shown to be due to differences in pain scores.4

More recently, the National Institutes for Health has funded the PROMIS outcome measure.1 This measure is more responsive than other measures and is less likely to suffer from floor and ceiling effects. It has been validated in many areas of medicine and has multiple scales that can be used including a general health measure, a depression score, a pain interference score, as well as a physical function score. There are limited data thus far in fracture care, but this research is currently being performed and holds great promise for understanding how our care affects patients. The advantages of this measure are its open availability for all researchers, and a 5-minute administration time. Work is being done to cross-tabulate the results of the PROMIS score with legacy measures to be able to compare future work with prior research in many areas.

In summary, the patient-based evaluation of any lower-extremity injury should include a general health measure as well as an anatomic or disease-specific score. Pain should be reported separately, if possible, and efforts should be made to avoid summed scoring. In the future, more adaptive PROs such as the PROMIS score may help us to evaluate not only our patients' outcomes, but also be used in comparative effectiveness evaluations using quality-adjusted life-year (QALY).

Back to Top | Article Outline


When evaluating the outcomes of an upper-extremity program, there are 2 components of any clinical outcomes program that must be in place: the tools that measure clinical outcomes, and the apparatus to capture and store this information. Consider the tools first because this influences the infrastructure that will be used to collect and store data. Consider what information must be collected to be confident that their patient's outcome is reliably measured. In regard to an upper-extremity outcomes program, it would be widely accepted that motion, strength, and a patient self-assessment survey be completed. In addition, if fracture work is being assessed, then radiographs should be included in the necessary results.

Although it is the patients' perception of their outcome that seems most important, no surgeon would be pleased with a clinical result in which the patients lost half their motion or had strength substantially diminished. Likewise, if the goal of an intervention is to fix a fracture, then a satisfactory reduction and union is paramount. It is true in contemporary surgical practice that functional outcomes are indeed held high on the priority scale, but this reality should not imply that motion, strength, and radiographic healing are no longer important to understand. In fact, it will likely be these latter variables that help to inform or interpret the result of a functional outcome tool. Therefore, all are important.

The self-assessment survey should include a general-outcomes instrument and an anatomy (joint or region)-specific instrument. These should be easy and efficient surveys for the patient to improve compliance. General health outcomes have been described above. For outcomes specific to the upper extremity, there are multiple options and as long as they are validated, they can provide utilitarian results. Examples include the American Shoulder and Elbow Score, the Disability of Arm Shoulder and Hand, the Constant–Murley Score, the Oxford Shoulder Score, and the Oxford Elbow Score. These are free and accessible, although some require authorized permission. Many of these tools have evolved to shortened versions such as the Quick-Disability of Arm Shoulder and Hand, which may improve compliance and mitigate patient burden.

The infrastructure to capture the outcome metrics includes both the mechanism for collecting data and the repository in which these data are stored. The mechanism for collection usually includes documentation of the history and physical examination by the clinician themselves, but this can be costly, time-consuming, and inefficient. It is generally agreed on for most conditions that the clinician should assess objective data on strength, motion, and radiographic healing. These variables should be recorded in an accessible database, which ideally can be queried.

It is the process of data transfer and management that is ideally performed by physician extenders (such as therapists, physician assistants, or nurse practitioners). Individual models should be evaluated in the context of the subspecialty, flow of care, and resources in which the clinician practices. Each model should optimize revenue to the system without compromising the quality of care or interaction between the physician and the patient.

A relatively recent phenomenon is the evolution of self-reported PROs in which the patient self-reports by interacting electronically with a data-capture mechanism. Data-capture mechanisms may include an interaction with a Smart-Pad in the office, or responding to an email, text, or telephone encounter. The advantage of these newer mechanisms is that the information can be entered and downloaded remote from the office visit, which may be more cost-effective, present less of a burden to office personnel, and consume less time from both the physician and the patient. The databases provide the opportunity for query and analysis for the purposes of research or benchmarking. Many of these systems allow for linkage to electronic medical record systems.

In conclusion, it should be recognized that physicians are working in a rapidly evolving landscape in medicine, and it is imperative for us to contribute to innovation and to be active in practice management and proactive in change management. In actuality, there is a tremendous advantage to the proper implementation and use of these new and future technologies and processes that will allow us to take better care of more patients. If executed properly, we will be in a better place to understand our patient's outcome of treatment, and to assess our patient outcome program.

Back to Top | Article Outline


Functional outcome instruments and quality-of-life measures are popular outcome tools for clinical research because of their inherent benefits. Most of these instruments collect PROs, which ensures the patients' perspective is directly measured. These instruments can facilitate disease- or extremity-specific outcome assessments. The instruments also report outcomes on a continuous numeric scale that ideally detect small changes in outcome and improve statistical power. Despite these advantages, the primary limitation to the utility of numeric outcome instruments is determining the clinical importance of the reported results. As a sample size becomes larger, the ability to statistically determine a difference in numeric outcomes between 2 groups increases. However, whether a statistically detected difference is important to patients or practitioners requires further information. The minimal clinically important difference (MCID) is designed to improve the interpretation of outcome instruments.

The MCID is defined as the smallest numeric difference in the outcome instrument, which patients perceive as beneficial. Therefore, differences between treatment options that exceed the MCID would mandate, in the absence of troublesome side effects or excessive costs, a change in the patient's management.5 The MCID is derived by applying a calculation technique to data obtained from the outcome instrument of interest. The 3 most commonly used MCID calculation techniques are based on: (1) the statistical distribution of the data (often 1/2 of the SD); (2) Delphi or consensus opinion; or (3) anchor questions that compare a change in the patient's outcome.6 When using an anchor method, patients are frequently asked a separate question to determine if they have “improved” from their baseline assessment. The mean difference between the baseline score and the repeat assessment among respondents who self-report improved function becomes the MCID.

The primary limitation of using a published MCID is the value is specific to the patient population and period used for its calculation. This limits its transferability to other populations or instruments. Furthermore, because the MCID is typically calculated from anchor methods that rely on improvements after a health intervention, its use in trauma can be problematic since we do not expect patients to “get better” beyond their preinjury baseline and patients must rely on estimates of their preinjury function.7 As a result, there are 2 common methods to apply the MCID for comparative studies: researchers can either conclude superiority of one treatment if the group's mean score exceeds the alternative treatment by a magnitude greater than the MCID; or alternatively, the proportion of individuals that recover or improve greater than the MCID in each treatment group can be compared.8 In general, most orthopaedic trauma studies rely on the MCID from an unrelated, nontrauma population and compare the mean scores between treatment groups.

Despite the challenges of using an MCID for most outcome instruments in orthopaedic trauma studies, the concept of an MCID has been an important advancement for patient-centered research. Whenever possible, clinicians and researchers should consider the MCID and how it was derived when interpreting the results of clinical trials comparing treatment options in orthopaedic trauma.

Back to Top | Article Outline


In an era of “value-based” health care, an understanding of the basic principles of economic analysis is a requirement to participate in the planning and delivery of care. Not only should an intervention have clinical benefit, but it must also be administered in a cost-efficient manner. Economic analysis is a means of quantifying the cost related to a health intervention in the context of how much benefit is achieved. Some expertise in clinical outcomes is needed to define an appropriate benefit for an intervention—this could include things such as length of stay, readmission, or death. In addition, some expertise in economics is also needed to properly define costs. Finally, comparing the cost-effectiveness of interventions, especially across disciplines in medicine and public health, requires a standardized set of methods termed the reference case analysis.

To conduct an economic analysis, costs must be assigned. From the perspective of society, all costs of an intervention are considered. From the perspective of the payer, only the costs for which the payer is responsible are included. Although variation exists, the standard is generally a societal perspective. Costs are further classified as either direct or indirect. Direct costs are those that involve the consumption of goods or services, such as medical care, implants, drugs, and rehabilitation appliances. Indirect costs are not consumable per se, but still measurable and may include things such as time. Indirect costs that are not easily accounted for are often termed intangible costs and include things such as pain or suffering. Assigning value to intangible costs is subjective and can lead to considerable variation in economic analysis modeling.

Cost-effectiveness analysis (CEA) is performed based on an outcome of interest. For most analyses, the defined outcome is not something binary such as death. Selecting an appropriate outcome can be complex, and there is a need for some commonality in measuring what is deemed an improvement in health or function, specifically to be able to compare interventions across disciplines. A benefit can also be selected based on perspective. For example, institutions may be interested in interventions that reduce length of stay or hospital readmission. However, from the perspective of the patient, most commonly there is a requirement for a measure of change in quality of life with the intervention. The most common metric used is a QALY with the reference being 1 year spent in perfect health. In states of disease or injury, the health-related quality of life refers to the proportion of a QALY (reduced from the value 1) and is often calculated from a self-reported outcome score such as the EQ-5D, which can also be derived from the Short-Form 36 questionnaire.9 When quality-of-life measures are used, the CEA is more accurately termed cost utility analysis (CUA). Finally, comparing the cost per QALY gained between interventions gives an incremental cost-effectiveness ratio (ICER). There is no universal ICER (cost per QALY gained) amount that defines a good “value” intervention. The National Institute for Health and Clinical Excellence in the United Kingdom and the World Health Organization have suggested values between $20,000 and $50,000 per QALY gained as a threshold.

CUA/CEA is increasingly common in the orthopaedic literature. However, the current literature often suffers from methodological flaws, with most published cost analyses using data from randomized controlled trials (RCTs).10 CUA is possible from RCT data when an SF-36 has been collected; however, RCTs are not typically designed with cost in mind, which can affect the assumptions made when constructing the CUA model. The CUA value produced from trial data is also only relevant to the interventions compared and in the patient population studied (ie, it follows the external validity of the trial).

A few recent articles from the orthopaedic trauma literature can be examined further to highlight some of these concepts. Performing CEA in areas of orthopaedic trauma where the benefit of surgical intervention (vs. non-surgical) is unclear, may be of limited value. Not surprisingly, the cost utility of what is typically a more expensive surgical intervention is unlikely to be superior when the clinical benefit is small or nonexistent. Recall that for a costlier intervention (larger numerator) to have a better ICER, the benefit (denominator) must be significantly greater. Handoll et al11 analyzed the cost utility of surgery in proximal humerus fractures from the ProFHER multicenter randomized trial. The trial did not demonstrate a clinical benefit for operative repair, and neither did the CEA.11

In a comparison of total hip and hemiarthroplasty for femoral neck fracture based on an older randomized trial, Keating et al12 developed a CEA model. In their model, the costs, taken from administrative data, were very similar between interventions. However, the utility value [health related quality of life (HRQoL)] of hemiarthroplasty was set at 75% of a total hip based on the trial results. Total hip replacement proved to be more cost-effective in this study because for a similar cost, it produced greater improvement in QALY. Whether the results of that trial hold true, and in what specific populations of elderly patients, means that this specific CEA may lack long-term relevance.

In some instances, if the costs include time and lost productivity as indirect costs, surgical intervention may outperform nonoperative management despite the direct cost of administering the surgery, because there is an earlier restoration of function. Davis et al13 examined the cost utility of open reduction and internal fixation of an undisplaced scaphoid waist fracture and found that open reduction and internal fixation was effective compared with casting—a finding driven primarily by time to recovery. However, the authors used a survey of medical students, a high functioning group, to define the time trade off. The value of time lost can be unique between individuals (eg, professional athlete vs. retired elderly patient) and decisions on how to value time will significantly influence the conclusions in these models.

In summary, cost utility or cost-effective analysis is an increasingly important component of health care system planning and decision making. However, the assumptions made, baseline data used, and the perspective can all significantly influence both the costs and benefits, producing wide variation in ICER. Careful interpretation of any CUA model assumptions is required to properly determine the validity of study findings.

Back to Top | Article Outline


When assessing outcomes in orthopaedic trauma, it is important to understand and appreciate the maladaptive responses to trauma. Depression, posttraumatic stress disorder (PTSD), and other impairments of psychosocial function are extremely common after trauma and represent substantial confounders of outcome. PTSD is defined as a disorder in which the patient persistently reexperiences a traumatic event and develops behaviors of avoidance, numbing of general responsiveness, and hyperarousal causing clinically significant distress or impairment that impairs function.14 Depression is defined as persistent sadness, decreased ability to experience pleasure, and decreased interest in usual activities that may result from the traumatic experience or chronic disability as a result of injury.

Risk factors for both PTSD and depression include young age, female sex, poor education, low socioeconomic class, alcohol and drug abuse, pain, and cognitive deficit. Pain perception is a possible modifiable risk factor that has been shown to influence the development of PTSD. Norman et al15 reported that patients with an increase of 1/2 of an SD on the visual analog scale for pain at 24–48 hours had a 5-fold increase of PTSD at 4 months and 7-fold increase at 8 months after discharge. Castillo et al reported a recurring and persistent relationship between pain and psychological distress at 2 years. Interestingly, both pain and psychological stress exacerbated each other during the chronic stage of trauma.16 Previous research has demonstrated that cognitive impairments and psychological dysfunction after trauma is quite common. In addition, these aspects of patient impairment are often significantly undertreated. Jackson et al17 found that 55% of patients with an injury severity score of >15 had a cognitive deficit, 40% had clinically significant symptoms of depression, and 26% presented with symptoms of PTSD. Bell et al18 reported that 68.4% of 500 injury survivors screened positive for depression, whereas only 22.2% of those patients reported receiving treatment for depression. Similarly, 44.4% of patients screened positive for PTSD, but only 9.8% received treatment.

This research highlights the fact that comprehensive postinjury care should include interventions to reduce pain and emotional stress. The expanded role for the orthopedic team is consistent with the development of integrated, collaborative care models emphasizing the need for interactions between informed patients (who take charge of their recovery) and proactive providers to support and encourage patient activation. These models of care delivery build off of a biopsychosocial approach to medicine and have been shown to be effective in managing complex medical conditions where pain and disability are common. Developing approaches for ensuring informed and activated patients is challenging, but is arguably the most important ingredient in a collaborative care approach to trauma management. Critical components of patient activation are: (1) access to good information and resources; (2) nurturing of problem-solving skills and high self-efficacy through self-management strategies; and (3) peer support.

The role of the surgeon and the trauma team is to perform routine screening for PTSD, depression, and overall distress, to refer patients who meet clinical criteria for a diagnosable condition, and to provide a patient- and family-centered environment promoting peer support and self-management to empower patients. One model resource is the Trauma Survivors Network (, which was developed by the American Trauma Society. This program has many components including peer visitation, family classes, peer support, and self-management classes. These classes include the use of structured techniques of cognitive-behavioral theory that help survivors alter cognitive, emotional, and behavioral responses to pain and impairment, which can be used in the treatment of arthritis and chronic diseases. The role of cognitive-behavioral theory in the management of trauma patients is becoming more widely recognized and is the subject of ongoing research in orthopaedic trauma.

The psychosocial aspects of trauma should not be overlooked because they are prevalent, undertreated, and have been shown to significantly affect outcome. Properly educating health care providers on how to screen and refer patients to the appropriate resources is of the utmost importance for optimizing patient outcomes.

Back to Top | Article Outline


An understanding of the important role that PROs play in evaluating treatment success or in comparing treatments in orthopaedic trauma is critical to effectively interpreting the literature and practicing evidence-based medicine. In addition, recognizing the importance of psychosocial factors and the important role that orthopaedic surgeons can play in addressing these issues is paramount to effective patient care and optimized outcomes.

Back to Top | Article Outline


1. Cella D, Yount S, Rothrock N, et al. The Patient-Reported Outcomes Measurement Information System (PROMIS): progress of an NIH Roadmap cooperative group during its first two years. Med Care. 2007;45(5 Suppl 1):S3–S11.
2. Lin CA, Swiontkowski M, Bhandari M, et al. Reaming does not affect functional outcomes after open and closed tibial shaft fractures: the results of a randomized controlled trial. J Orthop Trauma. 2016;30:142–148.
3. Majeed SA. Grading the outcome of pelvic fractures. J Bone Joint Surg Br. 1989;71:304–306.
4. Tornetta P III, Qadir R, Sanders R. Pain dominates summed scores for hindfoot and ankle trauma. J Orthop Trauma. 2013;27:477–482.
5. Jaeschke R, Singer J, Guyatt GH. Measurement of health status. Ascertaining the minimal clinically important difference. Control Clin Trials. 1989;10:407–415.
6. Rai SK, Yazdany J, Fortin PR, et al. Approaches for estimating minimal clinically important differences in systemic lupus erythematosus. Arthritis Res Ther. 2015;17:143.
7. Norman GR, Stratford P, Regehr G. Methodological problems in the retrospective computation of responsiveness to change: the lesson of Cronbach. J Clin Epidemiol. 1997;50:869–879.
8. Beaton DE, Boers M, Wells GA. Many faces of the minimal clinically important difference (MCID): a literature review and directions for future research. Curr Opin Rheumatol. 2002;14:109–114.
9. Vavken P, Bianchi T. Brief: cost-effectiveness analyses in orthopaedics. Clin Orthop Relat Res. 2011;469:2395–2398.
10. Coyle S, Kinsella S, Lenehan B, et al. Cost-utility analysis in orthopaedic trauma; what pays? A systematic review. Injury. 2018;49:575–584.
11. Handoll H, Brealey S, Rangan A, et al. The ProFHER (PROximal Fracture of the Humerus: Evaluation by Randomisation) trial—a pragmatic multicentre randomised controlled trial evaluating the clinical effectiveness and cost-effectiveness of surgical compared with non-surgical treatment for proximal fracture of the humerus in adults. Health Technol Assess. 2015;19:1–280.
12. Keating JF, Grant A, Masson M, et al. Randomized comparison of reduction and fixation, bipolar hemiarthroplasty, and total hip arthroplasty. Treatment of displaced intracapsular hip fractures in healthy older patients. J Bone Joint Surg Am. 2006;88:249–260.
13. Davis EN, Chung KC, Kotsis SV, et al. A cost/utility analysis of open reduction and internal fixation vs. cast immobilization for acute nondisplaced mid-waist scaphoid fractures. Plast Reconstr Surg. 2006;117:1223–1235; discussion 1236–1228.
14. American Psychiatric Association. Task Force on DSM-IV. Diagnostic and Statistical Manual of Mental Disorders DSM-IV-TR. 4th ed. Washington, DC: American Psychiatric Association; 2000.
15. Norman SB, Stein MB, Dimsdale JE, et al. Pain in the aftermath of trauma is a risk factor for post-traumatic stress disorder. Psychol Med. 2008;38:533–542.
16. Castillo RC, Wegener ST, Heins SE, et al. Longitudinal relationships between anxiety, depression, and pain: results from a 2-year cohort study of lower extremity trauma patients. Pain. 2013;154:2860–2866.
17. Jackson JC, Obremskey W, Bauer R, et al. Long-term cognitive, emotional, and functional outcomes in trauma intensive care unit survivors without intracranial hemorrhage. J Trauma. 2007;62:80–88.
18. Bell TM, Vetor AN, Zarzaur BL. Prevalence and treatment of depression and posttraumatic stress disorder among trauma patients with non-neurological injuries. J Trauma Acute Care Surg. 2018;85:999–1006.

PROs; MCID; economic analysis; depression; PTSD; orthopaedic trauma; fracture

Copyright © 2019 Wolters Kluwer Health, Inc. All rights reserved.