The COVID-19 pandemic has thrust changes on our specialty worldwide that heretofore have never taken place before. One such change is the delay or prohibition of doing surgical procedures that are considered “elective” with the aim of preserving personal protective equipment for front-line medical personnel and other resources, eg, intensive care units, for patients afflicted with COVID-19 disease by the severe acute respiratory syndrome coronavirus 2. Although some hospitals around the world have been overwhelmed and responded as best they can, other hospitals have had the challenge of planning in advance for a potential surge in patients using only inexact predictive modeling tools in an effort to preserve the appropriate portion of their resources to treat this pandemic.1,2 Once the exponential increase in COVID-19 diagnoses has decreased or plateaued, the next phase in planning is how to begin to restore and resume doing the backlog of surgical procedures that has been accrued. For some facilities, this number may run in the thousands and prioritization is problematic.
How to prioritize orthopaedic surgical procedures on a comprehensive scale has not taken place in the lifetime of most surgeons in healthcare facilities in developed countries. Some attempts, such as the recent MeNTS score, to address the prioritization are either too rigid or nonspecific regarding orthopaedic surgery or may have unintended consequences of scoring emergency procedures too low because of severe comorbidities (eg, compartment syndrome in a patient with multiple trauma ranks lower in priority than carpal tunnel release in a healthy patient).3-5 Furthermore, their reliability has not been tested. Although guidance is available for the triage of large-scale disasters or war time injuries, very limited information exists published for assessing the triage of all nonemergent orthopaedic surgeries as a whole. Most available information pertains to the triage of surgical wait times for specific procedures, eg, joint arthroplasty,6-12 or other services in demand such as cataract surgery13,14 or allied health care, the impact of delays15,16 or the ethical implications.17 Proposals to study this topic have also been reported.18 The severity of the COVID-19 pandemic will vary by locale, and resources for doing surgeries will expand and possibly contract, with possible second waves of infections. Orthopaedic surgeons need to be able to define the levels of urgency for all procedures, based on data and tested for reliability, rather than relying on nonspecific, untested, guidelines.
The primary aim of this study was to assess the reliability of a newly developed prioritization list used for the phased reinstatement of orthopaedic surgical procedures during the COVID-19 pandemic and afterward. The secondary aim was to report its implementation.
This study adhered to the Guidelines for Reporting Reliability and Agreement Studies19 including a workflow schematic diagram (Figure 1). A consensus-based, orthopaedic surgery-specific, tiered prioritization list reflecting various levels of medical necessity and urgency was created by a committee of orthopaedic surgeons covering all subspecialties and representing academic, multispecialty, and private community-based practices. To evaluate the applicability and reliability of the prioritization list, cases were independently selected by nonstudy-affiliated personnel from the academic institution's existing surgery waitlist and the previous month's emergency cases. The primary determinant for the prioritization into the tiered list was medical urgency, and the secondary determinant was a potential loss of function or pain suffered by the patient. The tier definitions needed to be specific and rigid enough to discriminate between level of urgency among procedures but flexible enough to allow for outliers and rare cases that do not fit into a specific tier or contain other mitigating factors to avoid patient harm. Furthermore, the list had to be acceptable to a large group of orthopaedic surgeons across all different healthcare settings. The four tiers correspond to four phases of restoration of hospital operations based on COVID-19 disease burden, current resources, and supply chain predictions (Figure 2). These are (1) tier 1—high priority cases including life- or limb-threatening cases or cases that cannot wait for greater than 14 to 21 days without notably associated morbidity, (2) tier 2—intermediate priority including cases that are not life or limb threatening but maintain the potential for future morbidity or mortality if surgery is not done within 30 days, (3) tier 3—low priority cases that can be delayed for more than 30 days, but still are time sensitive, without notably associated morbidity to patient or impact on patient outcome, and (4) tier 4—the lowest priority cases without medical urgency, including those that are truly elective, and can be done when hospital functions return to a normal state. The principles that guided tier assignment included questions of time sensitivity, morbidity, case complexity, and loss of function while allowing for a mitigation process for outlier cases or appeals. Specifically, how long can a procedure be delayed without causing permanent injury? How long can a procedure be delayed without notably increasing the surgical risk or morbidity with the understanding that a certain amount of risk is required because of the current pandemic? The loss of function because of pain and physical disability was considered acceptable as long as it was temporary, and not permanent. Finally, a mitigation process for appealing to the service line chief to change tier assignment was established for outlier or rare cases that fall outside the range of current definitions. A committee of 25 individuals (22 orthopaedic surgeons, 1 biomechanical engineer, 1 anesthesiologist, and 1 surgery director registered nurse) from the greater Minneapolis/St-Paul area, doing surgery within a large healthcare system comprising multiple hospitals and ambulatory surgery sites, developed the tier assignments based on consensus derived after group meetings. The additional individuals besides orthopaedic surgeons follows recommendations by the American College of Surgeons.5 The committee consisted of representatives across all orthopaedic subspecialties and a variety of practice types including two large single specialty private practices, a hospital-based multispecialty practice and a large academic practice. Current Procedural Terminology from the AAOS CODE-X 2019 database (American Academy of Orthopaedic Surgery, Chicago) were used to generate and comprehensively identify all possible orthopaedic procedure types. The surgeons' knowledge bases were used to determine indications for these procedures. The description of procedures contained in each of the tier groups is outlined in the prioritization list depicted in Figure 3.
An effort was made to ensure broad representation across all tiers in all subspecialties including adult reconstruction, oncology, foot and ankle, shoulder and elbow, hand, pediatric orthopaedics, orthopaedic traumatology, spine, and sports orthopaedics. The most recent orthopaedic surgeon's notes and all relevant imaging were extracted from patient charts and deidentified before randomization and rating. The appropriate sample size according to Donner et al20 with an 80% power and 5% significance and a P0 of 0.4 and P1 > 0.6 to prove at least substantial reliability, with a minimum of 3 raters, was 52 cases. To generate adequate number of cases per subspecialty, 63 cases were abstracted from the medical record and evaluated for the purposes of this study. Cases were scored into five categories: tiers 1 to 4 based on the level of urgency or unable to assess based on the data provided. The average of all 4 raters during the second round of rating was used to evaluate case distribution among tiers. Fleiss kappa was used to assess intrarater and interrater reliability for the four raters.21 The second round of ratings was used for further statistical analyses. Agreement was defined as poor (0.0 to 0.20), fair (0.21 to 0.40), moderate (0.41 to 0.6), substantial (0.61 to 0.8), and almost perfect (0.81 to 1.0).22
Fleiss kappa interrater agreement was found to be 0.63 (95% confidence interval [CI] 0.57 to 0.69) and 0.72 (95% CI 0.66 to 0.78) for the first and second round of ratings, respectively, signifying substantial agreement between raters. The distributions of cases within tiers 1, 2, 3, and 4 were 35%, 14%, 27%, and 24%, respectively (Table 2). In 27 of 63 cases, a perfect agreement among all raters at both time points was recorded. In an additional 10 cases, overall agreement was noted with only one outlier present. When evaluating the interrater agreement per tier, the best agreement was found in cases rated as tier 1 (k = 0.83, 95% CI 0.73 to 0.93), with near-perfect agreement between raters. The lowest agreement was found in cases classified as tier 2 (k = 0.46, 95% CI 0.36 to 0.56), with moderate agreement between raters (Table 1). When classified by subspecialty, oncology (k = 0.81, 95% CI 0.59 to 1.03) and foot and ankle (k = 0.79, 95% CI 0.49 to 1.10) had the highest interrater agreement, whereas spine (k = 0.39, 95% CI 0.21 to 0.57) and sports orthopaedics (k = 0.48, 95% CI 0.30 to 0.66) were found to have the only slight and moderate agreements, respectively (Table 2). The intrarater reliabilities among the 4 raters were 0.61 (rater 1), 0.91 (rater 2), 1.00 (rater 3), and 0.89 (rater 4).
Table 1 -
Case Distribution by Rater and Interrater Reliability per Tier (for Second Rating)
|Rater and Statistical Test
|Cohen Kappa (95% CI)
Table 2 -
Interrater Agreement per Orthopaedic Subspecialty
||No. of Cases
| Foot and ankle
| Paediatric orthopaedics
| Shoulder and elbow
| Orthopaedic traumatology
| Sports orthopaedics
CI = confidence interval No. = number
Within 3 weeks of inception to implementation, each participating surgical facility site noted 100% adoption and usage by the orthopaedic surgeons without any resistance or refusal. After the drafting and agreement to use the tier definitions for prioritizing surgeries on a waiting list, the acceptance was assessed by ease of adoption and utilization by all orthopaedic surgeons. The scheduling administrative staff at each surgical facility within the health system were also surveyed. Initial reviews by orthopaedic surgeons revealed a request for including a method using pain and disability in the ranking for patients with severe disruption of their lives. This resulted in the inclusion of a validated patient reported outcome measure or a validated waiting list tool (New Zealand wait list score23) for usage to gauge and quantitate the disruption for a given patient.
The current health crisis has resulted in notably scaled back operational capacity for (elective) orthopaedic procedures to adequately care for those with COVID-19 disease. As a result, a backlog in orthopaedic surgical care developed. Although many countries have had waitlists arise as the demand for services exceeds capacity, the United States has heretofore not had to deal with this problem or ration care. Despite the presence of general guidelines, no prioritization schema comprehensively covering all orthopaedic surgical procedures has ever been developed and tested for reliability. The present study proposes a consensus-based, stratified prioritization list, in four tiers, that is statistically reliable and applicable to the phased reinstatement of orthopaedic surgical procedures during either the COVID-19 pandemic or other situations where resources are severely constrained.
One of the limitations of the current study is that the prioritization list is a newly developed tool and discrepancies or outliers may be identified at a later stage. Furthermore, a learning curve is present as evidenced by the improved agreement in the second round of scores. By continued data collection after implementation of the list, these could be addressed to further improve it. A second potential limitation is that the reliability of the list may be dependent on the quality of documentation and the medical knowledge of the raters. However, by clearly specifying the different orthopaedic surgical procedures in combination with a description of urgency (acute, chronic, and impending), we believe that this has been adequately mitigated as proven by the calculated intrarater and interrater agreements. Further research is needed to evaluate the validation of the proposed prioritization list and attempts should be made to further improve it based on the data collected. Unfortunately, because no benchmark exists for prioritizing orthopaedic procedures, an external validation study can only be done after the existing tier definitions have been in force for a period of time to allow for determination of any adverse impact. As for prioritization within tier 4, more formal inclusion of additional factors such as pain, mobility, progression, and influence on work and independence may be included similar to the tools used in waitlist prioritization for elective surgeries in Canada8 and New Zealand.23
Once the COVID-19 pandemic allows for the expansion in number of orthopaedic surgeries done, it is important to allocate the limited resources, such an testing, staffing, equipment, and personal protection, in a manner that adequately reflects its medical necessity and urgency. Ideally, such an allocation tool must allow for both expansion and contraction of services. Recently, recommendations were made (GRADE weak) that elective surgery should be risk stratified and possibly deferred on the basis of the patients' age (≥75 years), morbid obesity, diabetes, uncontrolled hypertension, chronic pulmonary disease, obstructive sleep apnea, chronic heart disease, and immunocompromised state.24 Patients with substantial comorbidities and risk factors should be scheduled after healthier patients have been treated, and experience has been amassed from the establishment of screening, prevention, and treatment protocols.24,25 Multiple states across the United States have made nonspecific recommendations on delaying orthopaedic surgical procedures (WA, TN, MI, MA, FL).26 Others have recommended a list of acute orthopaedic injuries defined as “surgically necessary that are considered urgent.”27 Although the above recommendations are sensible, no specific practical prioritization list is available that encompasses most orthopaedic surgical procedures of different levels of medical urgency and necessity, and furthermore, to our knowledge, no such ranking tool has ever been tested for reliability. In the present study, such a specific tiered prioritization list was designed to protocolize the most common orthopaedic procedures and their indications. A group of orthopaedic surgeons covering all subspecialties and representing multiple different types of hospital systems reached consensus on these tiers. By using this particular approach, it is believed to be applicable across university hospitals, community hospitals, and private practice groups. Because of the lack of scientific evidence on the topic, most recommendations are currently derived from expert opinions, and controversy exists around which orthopaedic procedures are considered elective.25 Therefore, the consensus-based approach reaching a defined prioritization list was deemed appropriate. Another proposal to prioritize orthopaedic surgical cases has been by means of a Surgical Review Committee consisting of surgery, anesthesiology and nursing representatives.28 The use of the tiered list presented here is potentially less time consuming and more reproducible across different settings. To assess the practical applicability of the proposed prioritization list, the reliability was tested and found to be substantial (k = 0.72). Although not perfect, the list is specific and rigid enough to adequately discriminate between most levels of urgency among procedures while flexible enough to accommodate nearly all clinical situations and avoid any unintended adverse consequences. When evaluating the prioritization list by subspecialty, procedures in spine, shoulder and elbow, and sports domains may need further refinement with the caveat that the sample size in this study to analyze by subspecialty may have been rather limited. A strength of the proposed prioritization list is that using it to rate cases is reproducible as depicted by an almost perfect intrarater agreement (k range 0.61 to 1.0).
The development and implementation of any change in workflow is beset with challenges. Some of the costs of adoption are related to training requirements, difficulties in communication, resistance to change, unexpected deficiencies, and enhancing electronic medical record functionality. Previous experience with changes in categorizing surgeries has borne some of these out.29 Moreover, the decision-making associated with prioritization and triage is viewed by some as a moral dilemma.30 The COVID-19 pandemic has brought about so many constantly changing policies and procedures that physicians have become accustomed to such fluidity. Traditional barriers have been lowered, and surgeons have gained understanding in the need for adapting to circumstances beyond their control. No doubt exists that this environment has facilitated the rapid adoption and implementation of our tiered prioritization for surgeries.
One of the limitations of the current study is that the prioritization list is a newly developed tool, and discrepancies or outliers may be identified at a later stage. Furthermore, a learning curve is present as evidenced by the improved agreement in the second round of scores. By continued data collection after implementation of the list, these could be addressed to further improve it. A second potential limitation is that the reliability of the list may be dependent on the quality of documentation and the medical knowledge of the raters. However, by clearly specifying the different orthopaedic surgical procedures in combination with a description of urgency (acute, chronic, and impending), we believe that this has been adequately mitigated as proven by the calculated intrarater and interrater agreements. No doubt that many hospitals have already instituted similar policies potentially reducing the need for the current prioritization list. However, this is related to the phase of the COVID-19 pandemic they are experiencing locally, and many institutions both in the United States and internationally still have not yet begun restoration of normal hospital functions. Furthermore, if resources are again constrained during a second wave resurgence, then subsequent plans for restoration can now be based on processes with proven reliability. Further research is needed to evaluate the validation of the proposed prioritization list and attempts should be made to further improve it based on the data collected. In addition, our study was not designed to address how to allocate surgical time among the various orthopaedic and nonorthopaedic surgical services, an acknowledged challenge. Unfortunately, because no benchmark exists for prioritizing orthopaedic procedures, an external validation study can only be done after the existing tier definitions have been in force for a period of time to allow for determination of any adverse impact. As for prioritization within tier 4, more formal inclusion of additional factors such as pain, mobility, progression, and influence on work and independence may be included similar to the tools used in waitlist prioritization for elective surgeries in Canada6 and New Zealand.21
The proposed consensus-based tiered prioritization list for orthopaedic surgical procedures is both adoptable and reliable during the phased reinstatement of orthopaedic surgical procedures during the COVID-19 pandemic and afterward. Further refinements, focused on the classification for tiers 2, 3, and the spine subspecialty, may enhance utility.
The authors thank the following individuals for their assistance and involvement in this project: Peter Kelly, MD, Director Surgical Services, M Health Fairview (Figure 1). Brian Gager, FACHE, Service line executive, M Health Fairview (Figure 1). Consensus group committee: Drs. Chris Coetzee, Corey Wulf, Elizabeth Arendt, Jack Drogt, and others (21 anonymous).
1. Buggey H, Wheeler T: Cleveland Clinic and SAS share COVID-19 predictive models to help hospitals plan for current and future needs, in Models Help Forecast Bed and Ventilator Needs and Predict Impacts on Critical Supply Chains, Finance and Staffing to Optimize Hospital Preparedness Before, During and After Regional Peaks. Cleveland, OH, Cleveland Clinic, 2020.
2. American Hospital Association: A compendium of models that predict the spread of COVID-19, in COVID-19 Models: Forecasting the Pandemic's Spread. Chicago, IL, American Hospital Association, 2020.
3. CMS: Centers for Medicare & Medicaid Services: Non-emergent, elective medical services, and treatment recommendations.
Available at: https://www.cms.gov/files/document/cms-non-emergent-elective-medical-recommendations.pdf
. Accessed Accessed May 17, 2020.
4. Prachand VN, Milner R, Angelos P, et al.: Medically necessary, time-sensitive procedures: Scoring system to ethicallyand efficiently manage resource scarcity and provider risk during the COVID-19 pandemic. J Am Coll Surg 2020;231:281-288.
5. American College of Surgeons: COVID-19: Guidance for triage of non-emergent surgical procedures. Available at: https://www.facs.org/covid-19/clinical-guidance/triage
. Accessed Accessed March 17, 2020.
6. Arnett G, Hadorn DC; Steering Committee of the Western Canada Waiting List Project: Developing priority criteria for hip and knee replacement: Results from the Western Canada waiting list project. Can J Surg 2003;46:290-296.
7. Coleman B, McChesney S, Twaddle B: Does the priority scoring system for joint replacement really identify those in most need? N Z Med J 2005;118:U1463.
8. Conner-Spady B, Estey A, Arnett G, et al.: Prioritization of patients on waiting lists for hip and knee replacement: Validation of a priority criteria tool. Int J Technol Assess Health Care 2004;20:509-515.
9. Conner-Spady B, Estey A, Arnett G, et al.: Determinants of patient and surgeon perspectives on maximum acceptable waiting times for hip and knee arthroplasty. J Health Serv Res Policy 2005;10:84-90.
10. Conner-Spady BL, Arnett G, McGurran JJ, Noseworthy TW; Steering Committee of the Western Canada Waiting List Project: Prioritization of patients on scheduled waiting lists: Validation of a scoring system for hip and knee arthroplasty. Can J Surg 2004;47:39-46.
11. Conner-Spady BL, Johnston G, Sanmartin C, et al.: Patient and surgeon views on maximum acceptable waiting times for joint replacement. Healthc Policy 2007;3:102-116.
12. Desmeules F, Dionne CE, Belzile ÉL, Bourbonnais R, Frémont P: The impacts of pre-surgery wait for total knee replacement on pain, function and health-related quality of life six months after surgery. J Eval Clin Pract 2012;18:111-120.
13. Conner-Spady BL, Sanmugasunderam S, Courtright P, McGurran JJ, Noseworthy TW; Steering Committee of the Western Canada Waiting List Project: Determinants of patient satisfaction with cataract surgery and length of time on the waiting list. Br J Ophthalmol 2004;88:1305-1309.
14. Ma Quintana J, Escobar A, Bilbao A; IRYSS-Appropriateness Cataract Group: Explicit criteria for prioritization of cataract surgery. BMC Health Serv Res 2006;6:24.
15. Ackerman IN, Bennell KL, Osborne RH: Decline in health-related quality of life reported by more than half of those waiting for joint replacement surgery: A prospective cohort study. BMC Musculoskelet Disord 2011;12:108.
16. Desmeules F, Dionne CE, Belzile E, Bourbonnais R, Frémont P: The burden of wait for knee replacement surgery: Effects on pain, function and health-related quality of life at the time of surgery. Rheumatology (Oxford) 2010;49:945-954.
17. Pitt DF, Noseworthy TW, Guilbert J, Williams JR: Waiting lists: Management, legalities and ethics. Can J Surg 2003;46:170-175.
18. Déry J, Ruiz A, Routhier F, et al.: Patient prioritization tools and their effectiveness in non-emergency healthcare services: A systematic review protocol. Syst Rev 2019;8:78.
19. Kottner J, Audige L, Brorson S, et al.: Guidelines for reporting reliability and agreement studies (GRRAS) were proposed. Int J Nurs Stud 2011;48:661-671.
20. Donner A, Eliasziw M: Sample size requirements for reliability studies. Stat Med 1987;6:441-448.
21. Ranganathan P, Pramesh CS, Aggarwal R: Common pitfalls in statistical analysis: Measures of agreement. Perspect Clin Res 2017;8:187-191.
22. Landis JR, Koch GG: The measurement of observer agreement for categorical data. Biometrics 1977;33:159-174.
23. Gwynne-Jones DP, Iosua EE, Stout KM: Rationing for total hip and knee arthroplasty using the New Zealand Orthopaedic Association score: Effectiveness and comparison with patient-reported scores. J Arthroplasty 2016;31:957-962.
24. Parvizi J, Gehrke T, Krueger CA, et al.; The International Consensus Group (ICM) and Research Committee of the American Association of Hip and Knee Surgeons (AAHKS): Resuming elective orthopaedic surgery during the COVID-19 pandemic. Guidelines developed by the International Consensus Group (ICM). J Bone Joint Surg Am 2020;102:1205-1212.
25. Prada C, Chang Y, Poolman R, Johal H, Bhandari M: COVID-19 evidence based scoping review. Available at: https://myorthoevidence.com/
. Accessed May 17, 2020.
26. Sarac NJ, Sarac BA, Schoenbrunner AR, et al.: A review of state guidelines for elective orthopaedic procedures during the COVID-19 outbreak. J Bone Joint Surg Am 2020;102:942-945.
27. DePhillipo NN, Larson CM, O'Neill OR, LaPrade RF: The orthopaedic forum guidelines for ambulatory surgery centers for the care of surgically necessary/time-sensitive orthopaedic cases during the COVID-19 pandemic. J Bone Joint Surg Am 2020;102:933-936.
28. American College of Surgeons: Create a surgical review committee for COVID-19-related surgical triage decision making. Chicago, IL: American College of Surgeons; 2020. Available at: https://www.facs.org/covid-19/clinical-guidance/review-committee
. Accessed March 24, 2020.
29. Henley MB: ICD 10: “What orthopedic surgeons should know, how it will affect them and the cost of implementation?” Sports Med Arthrosc Rev 2013;21:142-147.
30. Carroll CT, Aziz KT, Humbyrd CJ: Elective reconstructive surgery during a pandemic: A moral dilemma. J Bone Joint Surg Am 2020;102:e51.