Journal Logo

Scientific Articles

Novel Biomarkers for Diagnosing Periprosthetic Joint Infection from Synovial Fluid and Serum

Keemu, Hannes MD1,a; Vaura, Felix2; Maksimow, Anu MD, PhD1; Maksimow, Mikael PhD2; Jokela, Aleksi1; Hollmén, Maija PhD2; Mäkelä, Keijo MD, PhD1

Author Information
doi: 10.2106/JBJS.OA.20.00067
  • Open
  • SDC
  • Disclosures


Periprosthetic joint infection (PJI) is a devastating complication after total joint arthroplasty (TJA), occurring in approximately 1% to 2% of all cases1. Diagnosis of PJI is usually multifactorial and relies on the clinical picture, radiographs, serologic tests, and synovial fluid analysis2. Clinical signs of PJI include local redness, swelling, wound drainage, pain, and fever. Joint synovial fluid analysis is the cornerstone of the diagnostic algorithm for confirmation of PJI3-5. Although synovial biomarkers have generally shown superior accuracy compared with serum biomarkers, serum tests are less invasive and remain the first-line screening tool6,7. An ideal biomarker would be accurate, easy to identify, quick to analyze, and cost-effective to detect8.

Several potential synovial fluid biomarkers (e.g., alpha defensin, leucocyte esterase, interleukin [IL]-6, and IL-8)8,9 and serum biomarkers (e.g., IL-6 and D-dimer)10 have been introduced to detect PJIs, with varying results. This study aimed to compare the synovial fluid and serum cytokine and chemokine patterns in TJA patients with and without symptoms of suspected PJI, in order to identify promising biomarkers for further research—i.e., comparing their diagnostic ability with existing ones such as alpha defensin.

Materials and Methods

The study was approved by the local Ethical Committee of the Hospital District of South-West Finland. With approval of the Ethical Committee, a register under the name The Role of Immunological Biomarkers in Diagnosing Prosthetic Joint Infections was formed and is held at the Department of Orthopaedics and Traumatology, Turku University Hospital.

Patient Cohort

From October 2016 to December 2018, synovial fluid and serum samples were collected from 35 patients who were admitted to the Arthroplasty Unit of the Department of Orthopaedics and Traumatology at the Turku University Hospital, Turku, Finland. Ten of these were healthy patients (52 to 84 years of age, 3 male and 7 female) with an elective TJA for osteoarthritis and served as the control group. Fifteen patients (57 to 90 years of age, 7 male and 8 female) who were admitted to the Arthroplasty Unit due to clinical suspicion of PJI and had a positive synovial fluid bacterial culture served as the study group (Table I). The remaining 10 patients with a negative synovial fluid bacterial culture were excluded.

TABLE I - Demographic Data
All Control Group PJI Group P Value
No. 25 10 15
Female (no. [%]) 15 (60) 7 (70) 8 (53) 0.68*
Age(yr) 72.7 ± 10.3 72.7 ± 10.3 72.7 ± 10.7 0.99
CRP§(mg/L) 17.0 (215.0) 0.8 (2.0) 117.0 (235.0) 6.8×10−5#
ESR§(mm/hr) 19.0 (29.8) 13.0 (14.5) 54.5 (38.0) 0.009#
WBC(×10 9 /L) 8.8 ± 4.5 6.2 ± 1.6 10.5 ± 5.0 0.0063
*Fisher exact test.
The values are given as the mean and standard deviation.
T test.
§The values are given as the median, with the width of the interquartile range in parentheses.
#Mann-Whitney U test.

Sampling and Analyses

One orthopaedic surgeon (H.K.) was responsible for collecting the synovial fluid samples. Nine mL of whole blood was collected in a serum sample tube and left to clot for 3 hours at room temperature. The sample was centrifuged at 2,000 ×g for 10 minutes, after which the serum was extracted and stored at −70°C until analysis. All samples were handled identically, and violations of this protocol would lead to the exclusion of the sample.

The synovial fluid sample was missing for 1 patient in the control group. All other patients had both a synovial fluid sample and a serum sample, for a total of 49 samples.

All analyses were performed with the same magnetic bead suspension array kit, Bio-Plex Pro Human Cytokine 37-plex (Bio-Rad Laboratories), according to the manufacturer’s instructions. The analyzed cytokines and their detection limits are shown in Table II.

TABLE II - Quantification Limits of the 37 Analyzed Biomarkers*
Acronym Full Name LLOQ (pg/mL) ULOQ (pg/mL)
APRIL/TNFSF13 A proliferation-inducing ligand; also known as tumor necrosis factor ligand superfamily member 13 3,848.5 1,823,800.0
BAFF/TNFSF13B B-cell activating factor; also known as tumor necrosis factor ligand superfamily member 13B 251.6 637,748.4
sCD30/TNFRSF8 Soluble CD30; also known as tumor necrosis factor receptor superfamily member 8 1.7 2,331.0
sCD163 Soluble CD163 137.9 2,446,500.0
Chitinase 3-like 1 Chitinase 3-like 1 19.3 142,584.1
gp130/sIL-6Rβ Glycoprotein 130; also known as soluble interleukin-6 receptor β 18.6 137,794.5
IFN-α2 Interferon-α2 1.7 242.2
IFN-β Interferon-β 12.7 219.3
IFN-γ Interferon-γ 3.8 425.0
IL-2 Interleukin-2 1.4 57.3
sIL-6Rα Soluble interleukin-6 receptor α 4.2 62,171.9
IL-8 Interleukin-8 5.4 64,430.1
IL-10 Interleukin-10 1.8 252.9
IL-11 Interleukin-11 0.2 1,930.6
IL-12 (p40) Interleukin-12 p40 homodimer 1.9 438.0
IL-12 (p70) Interleukin-12 p70 heterodimer 0.4 30.3
IL-19 Interleukin-19 3.7 254.1
IL-20 Interleukin-20 30.3 5,651.8
IL-22 Interleukin-22 6.6 3,053.2
IL-26 Interleukin-26 13.9 938.4
IL-27 (p28) Interleukin-27 p28 chain 6.0 369.8
IL-28A/IFN-λ2 Interleukin-28A; also known as interferon-λ2 4.4 214.8
IL-29/IFN-λ1 Interleukin-29; also known as interferon-λ1 87.7 872.3
IL-32 Interleukin-32 1.8 523.9
IL-34 Interleukin-34 126.1 2,548.4
IL-35 Interleukin-35 46.0 1,934.2
LIGHT/TNFSF14 Homologous to lymphotoxin, exhibiting inducible expression and competing with herpes simplex virus glycoprotein D for herpesvirus entry mediator, a receptor expressed by T lymphocytes; also known as tumor necrosis factor ligand superfamily member 14 34.4 1,749.6
MMP-1 Matrix metalloproteinase-1 121.1 574,322.7
MMP-2 Matrix metalloproteinase-2 195.7 600,817.5
MMP-3 Matrix metalloproteinase-3 70.9 438,588.7
Osteocalcin Osteocalcin 12.3 13,171.9
Osteopontin Osteopontin 142.0 1,907,200.0
Pentraxin-3 Pentraxin-3 13.0 241,496.6
sTNF-R1 Soluble tumor necrosis factor receptor 1 6.2 79,655.7
sTNF-R2 Soluble tumor necrosis factor receptor 2 12.5 3,692,800.0
TSLP Thymic stromal lymphopoietin 4.4 1,197.9
TWEAK/TNFSF12 tumor necrosis factor-like weak inducer of apoptosis; also known as tumor necrosis factor ligand superfamily member 12 1.5 1,674.9
*LLOQ = lower limit of quantification, and ULOQ = upper limit of quantification.

All 49 samples were analyzed using a single multiplex plate, the Bio-Plex 200 System, and Bio-Plex Manager 6.0 software (Bio-Rad Laboratories). The Bio-Plex Pro Human Cytokine 37-plex was chosen because the analyzed biomarkers form a representative repertoire of cytokines, some of which are suspected to be involved in PJI.

In addition to the study serum samples, the following analyses were routinely performed for all patients: plasma C-reactive protein (P-CRP), white blood-cell count (B-WBC), and blood erythrocyte sedimentation rate (B-ESR). Blood and synovial fluid bacterial culture specimens were taken from all 25 patients.

Statistical Analyses

Differences in sample characteristics between the study groups were compared using a 2-sample t test (age and blood leucocytes), Mann-Whitney U test (CRP and ESR), or Fisher exact test (sex).

First, biomarker concentrations were compared between the control and PJI groups to identify promising biomarkers. Biomarker concentrations that were out of range, either above the upper limit of quantification (ULOQ) or below the lower limit of quantification (LLOQ), were imputed as ULOQ or as LLOQ/√2, respectively. Only biomarkers with <10% out-of-range measurements in each group were included in the analysis. A Mann-Whitney U test was used to compare differences in concentrations, and both a Bonferroni correction and a false-discovery-rate correction (Benjamini-Hochberg)11 were used to adjust for multiple comparisons.

After promising biomarkers were identified, their ability to distinguish the control group from the study group was modeled with logistic regression and the model fit was quantified with receiver operating characteristic (ROC) curve analysis12. The Youden index was maximized to identify the optimal specificity and sensitivity for each biomarker. All statistical tests were 2-sided, p < 0.05 was considered significant, and R version 3.6.1 (R Core Team; R Foundation for Statistical Computing) was used for all computations.


Description of the Patient Cohort

The study population consisted of 25 participants with 13 hip and 12 knee arthroplasties, of whom 15 (60%) were female. All were of Caucasian origin. The mean age (and standard deviation) was 72.7 ± 10.3 years (range, 52 to 90 years) (Table I). Median P-CRP was 0.8 mg/L in the control group and 117.0 mg/L in the study group. Mean B-WBC was 6.2 ×109/L in the control group and 10.5 ×109/L in the study group. Median B-ESR was 13.0 mm/hr in the control group and 54.5 mm/hr in the study group (Table I).

Synovial Fluid Biomarkers

The differences in synovial fluid biomarker concentrations between the control and study groups are presented in Figure 1. The sTNF-R2, sTNF-R1, BAFF/TNFSF13B, pentraxin-3, MMP-2, sCD163, MMP-1, and sCD30 concentrations were highly significantly greater in the PJI group, whereas the osteocalcin concentration was highly significantly lower in the PJI group (p < 0.002) (Fig. 1). The IL-10, TSLP, IL-22, IL-19, IL-20, IL-35, APRIL, and MMP-3 concentrations were also significantly greater in the PJI group (p < 0.05) (Fig. 1).

Fig. 1:
The significance of differences in synovial fluid and serum biomarker concentrations (pg/mL) between the study and control groups, expressed as −log10(p). The p value is from the Mann-Whitney U test. Larger bars indicate a larger difference, and only biomarkers with p < 0.1 are shown. Bars with a black dot correspond to differences for which the p value would remain significant after a Benjamini-Hochberg correction for the false discovery rate (FDR). + indicates that the biomarker had a larger concentration in the PJI group compared with the control group, while − indicates the opposite. The blue vertical line corresponds to p = 0.05, and the red line corresponds to the Bonferroni significance level of p = 0.002.

Serum Biomarkers

The differences in serum biomarker concentrations between the control and study groups are presented in Figure 1. The sTNF-R1 concentration was highly significantly greater in the PJI group, whereas the TWEAK and osteocalcin concentrations were highly significantly smaller in the PJI group (p < 0.002). The sTNF-R2, BAFF/TNFSF13B, APRIL, TSLP, and pentraxin-3 concentrations were also significantly greater in the PJI group, while the MMP-2 and sIL-6Ra concentrations were significantly smaller (p < 0.05).

Box plots comparing the concentrations of the 9 most promising synovial biomarkers between the PJI and control groups are presented in Figure 2. Box plots of 8 additional biomarkers of preliminary interest are presented in the Supplementary data 1 section of the Appendix.

Fig. 2:
Box plots comparing the concentrations of the 9 most promising synovial biomarkers between the PJI and control groups. For clarity, the natural logarithms of the biomarker concentrations are plotted. The boxes indicate the interquartile range (IQR), the horizontal line within a box indicates the median, and the whiskers indicate points closer to the box than 1.5 times the IQR width.

Box plots comparing the concentrations of the 3 most promising serum biomarkers between the PJI and control groups are presented in Figure 3. Box plots of 7 additional biomarkers of preliminary interest are presented in the Supplementary data 2 section of the Appendix.

Fig. 3:
Box plots comparing the concentrations of the 3 most promising serum biomarkers between the PJI and control groups. For clarity, the natural logarithms of the biomarker concentrations are plotted. The boxes indicate the interquartile range (IQR), the horizontal line within a box indicates the median, and the whiskers indicate points closer to the box than 1.5 times the IQR width.

Sensitivity and Specificity of Biomarkers for Detecting PJI

For synovial fluid, the area under the ROC curve (AUC) was 1.00 for sTNF-R2, 0.96 for BAFF/TNFSF13B, and 0.99 for sTNF-R1 (Fig. 4, Table III). The optimal specificities for sTNF-R1, sTNF-R2, and BAFF were all 1.00. The optimal sensitivity was 1.00 for sTNF-R2, 0.87 for BAFF, and 0.93 for sTNF-R1 (Table III). The AUCs for 8 additional synovial biomarkers are presented in the Supplementary data 3 and 4 sections of the Appendix.

Fig. 4:
Receiving operating characteristic (ROC) curves for the 9 most promising synovial biomarkers.
TABLE III - Diagnostic Performance and P Value for Each Biomarker Reaching the Bonferroni Significance Level of P < 0.002
AUC (95% CI) Optimal Specificity and Sensitivity P Value (Control Vs. PJI)
Synovial fluid
 sTNF-R2 1.00 1.00, 1.00 6.5×10−5
 sTNF-R1 0.99 (0.95-1.00) 1.00, 0.93 1.1×10−4
 BAFF 0.96 (0.90-1.00) 1.00, 0.87 2.2×10−4
 Pentraxin-3 0.94 (0.84-1.00) 0.89, 0.93 4.3×10−4
 Osteocalcin 0.94 (0.84-1.00) 1.00, 0.87 4.3×10−4
 MMP-2 0.94 (0.82-1.00) 1.00, 0.93 4.3×10−4
 sCD163 0.92 (0.81-1.00) 0.67, 1.00 8.4×10−4
 MMP-1 0.90 (0.77-1.00) 1.00, 0.80 0.0013
 sCD30 0.90 (0.77-1.00) 0.89, 0.87 0.0013
 TWEAK 0.93 (0.82-1.00) 1.00, 0.80 4.2×10−4
 sTNF-R1 0.90 (0.77-1.00) 1.00, 0.73 0.0011
 Osteocalcin 0.89 (0.75-1.00) 1.00, 0.80 0.0014

For serum biomarkers, the AUC was 0.93 for TWEAK, 0.90 for sTNF-R1, and 0.89 for osteocalcin (Fig. 5). The optimal specificities for TWEAK, sTNF-R1, and osteocalcin were all 1.00. The optimal sensitivity was 0.80 for TWEAK, 0.73 for sTNF-R1, and 0.80 for osteocalcin (Table III). The AUCs for 7 additional serum biomarkers are presented in the Supplementary data 3 and 5 sections of the Appendix.

Fig. 5:
Receiving operating characteristic (ROC) curves for the 3 most promising serum biomarkers.


TNF-α was originally identified as a factor that causes rapid necrosis of transplantable tumors in mice13. Currently, it is considered a pro-inflammatory cytokine, involved in the innate immune response14. In fact, bacterial components are the main inducers of TNF-α secretion by tissue macrophages via pattern recognition receptors and NF-κB (nuclear factor kappa-light-chain-enhancer of activated B cells) signaling15. TNF-α was assessed previously for PJI diagnosis and found to have high specificity (0.94) but low sensitivity (0.43)16. It has been stated that TNF-α is an unstable biomarker, and that it is not as sensible as IL-6 for detecting PJI13. The panel used in the current study did not include TNF-α. However, soluble TNF receptors (sTNF-R1 and TNF-R2), which were included in the panel, had high sensitivity and specificity for detecting PJI using synovial fluid, and sTNF-R1 and to a lesser extent sTNF-R2 also had promising diagnostic performance using serum. The shedding of both sTNF-R1 and sTNF-R2 is induced by TNF-α to neutralize its effects and prevent overt tissue damage. Thus, measuring the soluble forms of TNF-α receptors may provide a stabler way to detect bacteria-induced events in joints with periprosthetic infection.

Synovial BAFF had high sensitivity and specificity for detecting PJI, and serum BAFF also showed some potential. Serum TWEAK was able to diagnose PJI well, although synovial fluid TWEAK was not. We are not aware of any previous reports of the use of any of these cytokines in PJI diagnosis. MMPs are enzymes that are capable of degrading extracellular matrix proteins, but they can also process bioactive molecules involved in cytokine inactivation17. Synovial fluid MMP-2 was able to distinguish patients with PJI from healthy controls. Marazzi et al. previously assessed the use of serum MMP-9 for detecting PJI, and considered that it had very low diagnostic potential18. Serum MMP-2 was somewhat promising in the current study, although the other MMPs were not.

Osteocalcin is a non-collagenous hormone found in bone and dentin; it is secreted by osteoblasts and plays a role in the body’s metabolic regulation19. Serum osteocalcin had good sensitivity and excellent specificity for detecting PJI in the current study, and synovial fluid osteocalcin was also promising. Osteopontin is a phosphoprotein with signaling and adhesive functions. It can act as either an extracellular matrix component in mineralized tissues or a soluble cytokine in serum20. Marazzi et al. previously reported that serum osteopontin is elevated in patients with PJI compared with patients with aseptic loosening of TJA components. Their ROC analysis revealed that osteopontin had good diagnostic values for PJI18. However, these previous findings were not supported by our data. Furthermore, synovial fluid osteopontin was also a poor detector of PJI in the current study.

Missing data complicate statistical analysis, especially when the data are not missing at random. In our study, we imputed variables with <10% missing data. When the level of missing data increases, so does the error of the imputation. However, incorporating information about the LLOQ and ULOQ into the analysis through imputation reduces bias. While setting a cutoff at 10% missing data is arguably arbitrary, we considered it to be a valid compromise in our analyses.

As this study was exploratory in nature, formal power calculations were not performed in advance. In future studies, the promising biomarkers from this study could be compared with previously studied well-performing biomarkers, such as alpha defensin. Based on the results for sTNF-R1 and osteocalcin in our study, together with previously reported results for alpha defensin21, an estimated 80 patients and 80 controls would be needed to achieve a power of 0.8 with a significance level of 0.05 when assessing the equivalence between our best-performing 2 biomarkers and alpha defensin. These power calculations were performed using pROC22, which is based on the work of Obuchowski and McClish23.

In conclusion, an early accurate PJI diagnosis would be highly beneficial for an orthopaedic clinician, and the best biomarkers in our study could in theory have value as quick preoperative PJI diagnostic tests. The current project was a preliminary study to identify and assess several potential biomarkers. The strength of our study was that we were able to assess both serum and synovial fluid samples from a relatively large patient cohort. We report high sensitivity and specificity for several new synovial fluid biomarkers for detecting PJI. Synovial fluid sTNF-R2, in particular, is a promising new biomarker. Further research is needed to assess the clinical importance of our findings and compare the diagnostic potential of the newly identified biomarkers with current ones, such as alpha defensin.


Supporting material provided by the authors is posted with the online version of this article as a data supplement at (


1. Kuiper JW, Willink RT, Moojen DJ, van den Bekerom MP, Colen S. Treatment of acute periprosthetic infections with prosthesis retention: review of current concepts. World J Orthop. 2014 Nov 18;5(5):667-76.
2. Parvizi J, Gehrke T; International Consensus Group on Periprosthetic Joint Infection. Definition of periprosthetic joint infection. J Arthroplasty. 2014 Jul;29(7):1331. Epub 2014 Mar 21.
3. Della Valle CJ, Sporer SM, Jacobs JJ, Berger RA, Rosenberg AG, Paprosky WG. Preoperative testing for sepsis before revision total knee arthroplasty. J Arthroplasty. 2007 Sep;22(6)(Suppl 2):90-3. Epub 2007 Jul 26.
4. Parvizi J, Ghanem E, Sharkey P, Aggarwal A, Burnett RS, Barrack RL. Diagnosis of infected total knee: findings of a multicenter database. Clin Orthop Relat Res. 2008 Nov;466(11):2628-33. Epub 2008 Sep 10.
5. Squire MW, Della Valle CJ, Parvizi J. Preoperative diagnosis of periprosthetic joint infection: role of aspiration. AJR Am J Roentgenol. 2011 Apr;196(4):875-9.
6. Deirmengian C, Kardos K, Kilmartin P, Cameron A, Schiller K, Parvizi J. Diagnosing periprosthetic joint infection: has the era of the biomarker arrived? Clin Orthop Relat Res. 2014 Nov;472(11):3254-62.
7. Saleh A, Ramanathan D, Siqueira MBP, Klika AK, Barsoum WK, Rueda CAH. The diagnostic utility of synovial fluid markers in periprosthetic joint infection: a systematic review and meta-analysis. J Am Acad Orthop Surg. 2017 Nov;25(11):763-72.
8. Lee YS, Koo KH, Kim HJ, Tian S, Kim TY, Maltenfort MG, Chen AF. Synovial fluid biomarkers for the diagnosis of periprosthetic joint infection: a systematic review and meta-analysis. J Bone Joint Surg Am. 2017 Dec 20;99(24):2077-84.
9. Deirmengian C, Kardos K, Kilmartin P, Cameron A, Schiller K, Parvizi J. Combined measurement of synovial fluid α-defensin and C-reactive protein levels: highly accurate for diagnosing periprosthetic joint infection. J Bone Joint Surg Am. 2014 Sep 3;96(17):1439-45.
10. Saleh A, George J, Faour M, Klika AK, Higuera CA. Serum biomarkers in periprosthetic joint infections. Bone Joint Res. 2018 Jan;7(1):85-93.
11. Benjamini Y, Hochberg Y. Controlling the false discovery rate: a practical and powerful approach to multiple testing. J R Stat Soc B. 1995;57(1):289-300.
12. McFadden D. Conditional logit analysis of qualitative choice behavior. In: Zarembka P, editor. Frontiers in econometrics. Academic Press; 1974. p 105-42.
13. Vicenti G, Bizzoca D, Nappi V, Pesce V, Solarino G, Carrozzo M, Moretti F, Dicuonzo F, Moretti B. Serum biomarkers in the diagnosis of periprosthetic joint infection: consolidated evidence and recent developments. Eur Rev Med Pharmacol Sci. 2019 Apr;23(2)(Suppl):43-50.
14. Stahelova A, Mrazek F, Smizansky M, Petrek M, Gallo J. Variation in the IL1B, TNF and IL6 genes and individual susceptibility to prosthetic joint infection. BMC Immunol. 2012 May 8;13:25.
15. Wajant H, Siegmund D. TNFR1 and TNFR2 in the control of the life and death balance of macrophages. Front Cell Dev Biol. 2019 May 29;7:91.
16. Bottner F, Wegner A, Winkelmann W, Becker K, Erren M, Götze C. Interleukin-6, procalcitonin and TNF-α: markers of peri-prosthetic infection following total joint replacement. J Bone Joint Surg Br. 2007 Jan;89(1):94-9.
17. Van Lint P, Libert C. Chemokine and cytokine processing by matrix metalloproteinases and its effect on leukocyte migration and inflammation. J Leukoc Biol. 2007 Dec;82(6):1375-81. Epub 2007 Aug 20.
18. Marazzi MG, Randelli F, Brioschi M, Drago L, Romanò CL, Banfi G, Massaccesi L, Crapanzano C, Morelli F, Corsi Romanelli MM, Galliera E. Presepsin: a potential biomarker of PJI? A comparative analysis with known and new infection biomarkers. Int J Immunopathol Pharmacol. 2018 Jan-Dec;31:394632017749356. Epub 2017 Dec 18.
19. Lee NK, Sowa H, Hinoi E, Ferron M, Ahn JD, Confavreux C, Dacquin R, Mee PJ, McKee MD, Jung DY, Zhang Z, Kim JK, Mauvais-Jarvis F, Ducy P, Karsenty G. Endocrine regulation of energy metabolism by the skeleton. Cell. 2007 Aug 10;130(3):456-69.
20. Vaschetto R, Nicola S, Olivieri C, Boggio E, Piccolella F, Mesturini R, Damnotti F, Colombo D, Navalesi P, Della Corte F, Dianzani U, Chiocchetti A. Serum levels of osteopontin are increased in SIRS and sepsis. Intensive Care Med. 2008 Dec;34(12):2176-84. Epub 2008 Sep 20.
21. De Vecchi E, Romanò CL, De Grandi R, Cappelletti L, Villa F, Drago L. Alpha defensin, leukocyte esterase, C-reactive protein, and leukocyte count in synovial fluid for pre-operative diagnosis of periprosthetic infection. Int J Immunopathol Pharmacol. 2018 Mar-Dec;32:2058738418806072.
22. Robin X, Turck N, Hainard A, Tiberti N, Lisacek F, Sanchez JC, Müller M. pROC: an open-source package for R and S+ to analyze and compare ROC curves. BMC Bioinformatics. 2011 Mar 17;12(1):77.
23. Obuchowski NA, McClish DK. Sample size determination for diagnostic accuracy studies involving binormal ROC curve indices. Stat Med. 1997 Jul 15;16(13):1529-42.

Supplemental Digital Content

Copyright © 2021 The Authors. Published by The Journal of Bone and Joint Surgery, Incorporated. All rights reserved.