Journal Logo

Statistical Consideration for Bilateral Cases in Orthopaedic Research

Park, Moon Seok, MD1; Kim, Sung Ju, MS2; Chung, Chin Youb, MD1; Choi, In Ho, MD3; Lee, Sang Hyeong, MD1; Lee, Kyoung Min, MD1

doi: 10.2106/JBJS.I.00724
Scientific Articles
Free

Background: Statistical independence means that one observation is not affected by another; however, the principle of statistical independence is violated if left and right-side measures within a subject are considered to be independent, because they are usually correlated and can affect each other. The purpose of the present study was to analyze the violation of statistical independence in recent orthopaedic research papers and to demonstrate the effect of statistical analysis that considered the data dependency within a subject.

Methods: First, all original articles that had been published in The Journal of Bone and Joint Surgery (American Volume) over a two-year period were evaluated. The analysis was designed to identify articles that included bilateral cases and possible violations of statistical independence. Second, a demonstrative logistic regression without consideration of statistical independence was performed and was compared with a statistical analysis that considered data dependency within a subject. Radiographs of 1200 hips in 600 patients were used to examine the differences in terms of odds ratios (with 95% confidence intervals) of the risk factors for hip osteoarthritis.

Results: Four hundred and eighty-six original articles were reviewed, and 151 articles (including forty-one articles involving the hip, thirty-four involving the knee, twenty-one involving the foot or ankle, nineteen involving the shoulder, ten involving the hand or wrist, nine involving the elbow, and seventeen involving other structures) were considered to include bilateral cases. Of the 486 articles that were reviewed, 120 articles (25%) (including thirty-six articles involving the hip, twenty-six involving the knee, fifteen involving the foot or ankle, fourteen involving the shoulder, seven involving the elbow, six involving the hand or wrist, and sixteen involving other structures) were found to have possibly violated statistical independence. Demonstrative statistical analysis showed that logistic regression was not robust to the violation of statistical independence. The 95% confidence intervals of the odds ratios for the risk factors showed narrower ranges (1.13 to 2.68 times) when data dependency within a subject was not considered.

Conclusions: Researchers need to consider statistical independence when performing statistical analysis, particularly in studies involving bilateral cases. If data dependency within a subject is not considered, studies involving bilateral cases can bias results, depending on the context of those studies.

1Department of Orthopaedic Surgery, Seoul National University Bundang Hospital, 300 Gumi-Dong, Bundang-Gu, Sungnam, Kyungki 463-707, South Korea. E-mail address for K.M. Lee: oasis100@empal.com

2Health Economics and Outcome Research, Market Access, Novartis Korea, 84-11 Namdaemun 5th Street, Chung-Gu, Seoul 100-753, South Korea

3Department of Orthopaedic Surgery, Seoul National University Children's Hospital, 28 Yongun-Dong, Chongno-Gu, Seoul 110-744, South Korea

Given the trend toward evidence-based medicine1-7, medical researchers commonly use statistical analyses to demonstrate study hypotheses objectively. However, one of the most important assumptions in statistical analysis when comparing independent groups is that the observations are independent of each other8. Nevertheless, investigators in orthopaedic research frequently encounter cases of bilateral involvement in which the left and right sides are possibly correlated with each other. The principle of statistical independence could be violated if, in these cases, the left and right-side measures are considered to be independent despite their correlation9. By definition, statistical independence means that each observation is not affected by another observation.

The effects of bilateral cases have raised concerns in orthopaedic research, depending on the hypothesis. First, if a study includes bilateral cases, researchers may be faced with a dilemma as to whether to analyze the joints or the patients. Joint-based analysis could cause the demographic data to be duplicated, and patient-based analysis is likely to miss the clinical implications associated with unilaterality and bilaterality. Second, the inclusion of both joints in statistical analysis violates the underlying principle of statistical independence and could bias the study results by exaggerating the levels of significance and narrowing the confidence intervals if the joints in same individuals are significantly related to each other10,11.

In the present study, we hypothesized that a considerable amount of orthopaedic research does not properly consider the principle of statistical independence in bilateral cases. Although reports may mention that right and left sides are closely correlated, this correlation between right and left sides should be accounted for when analyzing data9,11-13.

Therefore, the purpose of the present study was to assess the possible violation of statistical independence in orthopaedic research involving bilateral cases and to demonstrate the effect of statistical analysis that considers the data dependency within a subject.

Back to Top | Article Outline

Materials and Methods

The present study was approved by the institutional review board at our institute. It consisted of two parts: (1) a journal review and (2) a demonstration of the effects of the statistical analysis considering the data dependency within a subject.

The journal review was performed by two orthopaedic surgeons (M.S.P. and K.M.L.) to assess considerations of bilaterality when performing statistical analysis in orthopaedic research. Articles published in The Journal of Bone and Joint Surgery (American Volume) (JBJS-Am) were analyzed. The articles that were included met the following criteria: (1) publication between January 2007 and December 2008 and (2) the presentation of original research regarding either the upper or lower extremity. The articles that were excluded included (1) case reports, (2) Current Concepts Reviews, (3) editorials, (4) Instructional Course Lectures, (5) basic-science studies, (6) economic studies, (7) meta-analyses, and (8) studies of spinal diseases. For each article that met the above criteria, we recorded the number of subjects (persons) and cases (limbs, joints, muscles, and so on) along with the specific body parts involved (hips, knees, shoulders, and so on). The articles were categorized according to criteria established among three of the authors (M.S.P., S.H.L., and K.M.L.). Articles with the same number of subjects and cases were classified as unilateral studies, and articles with different numbers of subjects and cases were classified as “potentially bilateral studies” and were analyzed further. The articles in which the subject or case numbers were not indicated in the text were assigned to Category 1 (“possible bilateral cases, but not clarified in the text”). The statistical methods of the remaining studies were then reviewed further in the Materials and Methods section and were assigned to Category 2 (“bilateral cases considered to be independent cases in the statistical analysis”), Category 3 (“bilateral cases with one representative measurement [average or one selected side] included in the statistical analysis”), Category 4 (“bilateral cases separated from unilateral cases”), Category 5 (“bilateral cases statistically considered in terms of correlations between right and left sides”), Category 6 (“bilateral cases not subjected to statistical analysis” [i.e., only descriptive analysis performed]), or Category 7 (“bilateral cases in which the contralateral sides were used as controls”).

Of these seven categories, Categories 1 and 2 were considered to possibly violate the principle of statistical independence and Categories 3 through 7 were considered sound in this respect.

The second part of this study involved examining the odds ratios (with 95% confidence intervals) of the risk factors for hip osteoarthritis in elderly patients. Six hundred patients with an age of more than sixty-five years who had been evaluated with anteroposterior weight-bearing hip radiographs at our hospital from September 2005 to August 2006 were selected consecutively. Patients with total joint arthroplasty, previous osseous surgery, or severe joint destruction due to infection, trauma, or tumor were excluded. On the hip radiographs, the joint space width was assessed in three areas: (1) the lateral end of the subchondral sclerotic line, (2) a vertical line from the center of the femoral head, and (3) the medial end of the weight-bearing surface near the fovea. The minimum joint space width was selected as the smallest of these three measurements or as another measurement if the minimum joint space width was found outside the above three areas. Hip osteoarthritis was defined as a minimum joint space width of ≤2.5 mm14-16. The demographic data, including age, sex, and body-mass index, were recorded as candidate risk factors for hip osteoarthritis. After setting the cutoff values for the continuous variables (seventy years for age and 25 kg/m2 for body-mass index) with coding, the odds ratio (and 95% confidence interval) of each candidate risk factor for hip osteoarthritis was calculated with use of logistic regression, a generalized estimating equation17, and multilevel modeling18,19, the latter two of which considered the correlation between the right and left hips. The odds ratios (with 95% confidence intervals) were calculated for all 600 patients as well as for 300, 100, and fifty patients who were selected randomly by means of computer-generated random numbers with use of a uniform distribution. All radiographic images were acquired digitally with use of a picture archiving and communication system (PACS) (IMPAX; Agfa, Mortsel, Belgium). All radiographic measurements were carried out with use of PACS software by a single orthopaedic surgeon (K.M.L.) who was unaware of the clinical information.

For this part of the study, candidate risk factors were coded as binary data according to the cutoff values. The odds ratios of each candidate risk factor were calculated with use of logistic regression, the generalized estimating equation model, and multilevel modeling and are presented with the 95% confidence intervals. The level of significance was set at p < 0.05.

Back to Top | Article Outline

Source of Funding

There was no external funding source related to this investigation.

Back to Top | Article Outline

Results

Four hundred and eighty-six original articles were found in JBJS-Am during the index period. Of these, we excluded 135 articles concerning spinal diseases, meta-analyses, economic studies, and basic-science studies. Of the remaining 351 articles (including seventy-five articles on the knee, sixty-eight articles on the hip, fifty-five articles on the shoulder, thirty-eight articles on the foot or ankle, thirty-two articles on the elbow, twenty-two articles on the hand and wrist, and sixty-one other articles regarding multiple sites or specific bones and tendons), 200 articles included only unilateral cases. Finally, 151 articles (forty-one articles on the hip, thirty-four articles on the knee, twenty-one articles on the foot or ankle, nineteen articles on the shoulder, ten articles on the hand or wrist, nine articles on the elbow, and seventeen other articles regarding multiple sites or specific bones and tendons) were classified as potentially bilateral studies. Thirty-three studies did not clarify the numbers of subjects or cases. Eighty-seven studies analyzed sides as independent cases. Six studies analyzed representative data from a single subject by averaging or selecting one side. Four studies separated unilateral cases from bilateral cases and analyzed the data separately. Six studies considered the correlation between right and left-side data in the statistical analysis. Ten studies did not involve a statistical analysis, and five studies used contralateral sides as matched controls during statistical analysis. Therefore, of the 486 original articles, 120 articles (thirty-six articles on the hip, twenty-six articles on the knee, fifteen articles on the foot or ankle, fourteen articles on the shoulder, seven articles on the elbow, six articles on the hand or wrist, and sixteen other articles) were considered to have possibly violated statistical independence (Table I); in 77% of these studies, <30% of the total number of cases were bilateral (Fig. 1).

Fig. 1

Fig. 1

TABLE I - Categories of the 351 Original Articles Included in Present Study
Category No. of Articles Body Parts
Articles containing only unilateral cases 200 41 knee, 27 hip, 36 shoulder, 23 elbow, 17 foot/ankle, 12 wrist/hand, and 44 others
Articles containing bilateral cases 151 41 hip, 34 knee, 21 foot/ankle, 19 shoulder, 9 elbow, 10 wrist/hand, and 17 others
 Articles possibly violating statistical independence 120 36 hip, 26 knee, 15 foot/ankle, 14 shoulder, 7 elbow, 6 wrist/hand, and 16 others
  Not clarified 33
  Bilateral cases analyzed as independent 87
 Articles not violating statistical independence 31 8 knee, 6 foot/ankle, 5 hip, 5 shoulder, 2 elbow, 4 wrist/hand, and 1 other
  One representative measurement 6
  Unilateral cases separated from bilateral cases 4
  Statistically considered for correlation between right and left sides 6
  No statistical analysis 10
  Other side as matched control 5

Radiographs of the hip for the 258 male and 342 female patients were retrieved. The mean age of the 600 consecutive patients selected for radiographic measurements was 71.4 ± 5.0 years. The mean body-mass index was 24.4 ± 3.2 kg/m2. The overall prevalence of hip osteoarthritis was 13.3%. The prevalence of unilateral and bilateral hip osteoarthritis was 9.8% and 3.5%, respectively. The correlation of joint space width (in millimeters) between right and left hips within a subject was 0.761. When correlation between the right and left hips was not considered (that is, when the data were evaluated with logistic regression), the 95% confidence intervals of the odds ratios of each candidate risk factor were 1.13 to 2.68 times narrower than the ranges when correlation of the right and left hips was considered (that is, when the data were evaluated with the generalized estimating equation and multilevel modeling). The difference in the width of the 95% confidence intervals between logistic regression and generalized estimating equation (or multilevel modeling) appeared to decrease with increasing number of cases (Table II); however, this result needs to be investigated further to determine its significance.

TABLE II - Change in the Confidence Interval with Various Subject Numbers According to Statistical Method Used*
Logistic Regression*
Generalized Estimating Equation
Multilevel Model
Risk Factors Odds Ratio 95% Confidence Interval (A) Odds Ratio 95% Confidence Interval (B) Ratio (B/A) Odds Ratio 95% Confidence Interval (C) Ratio (C/A)
600 persons, 1200 joints
 Age 2.22 1.40 to 3.53 2.22 1.26 to 3.92 1.25 2.10 1.19 to 3.70 1.18
 Body-mass index 0.78 0.50 to 1.21 0.78 0.44 to 1.38 1.32 0.77 0.44 to 1.35 1.28
 Sex 1.95 1.24 to 3.07 1.95 1.07 to 3.54 1.35 1.96 1.12 to 3.45 1.27
300 persons, 600 joints
 Age 1.40 0.75 to 2.62 1.40 0.66 to 2.97 1.24 1.30 0.61 to 2.80 1.17
 Body-mass index 0.55 0.28 to 1.06 0.55 0.23 to 1.28 1.35 0.57 0.25 to 1.27 1.31
 Sex 2.79 1.39 to 5.60 2.79 1.15 to 6.73 1.33 2.62 1.15 to 5.97 1.14
100 persons, 200 joints
 Age 2.05 0.65 to 6.41 2.05 0.50 to 8.31 1.36 1.85 0.43 to 7.99 1.31
 Body-mass index 0.24 0.05 to 1.12 0.24 0.02 to 2.40 2.22 0.25 0.04 to 1.61 1.47
 Sex 3.04 0.91 to 10.17 3.04 0.57 to 16.21 1.69 2.51 0.57 to 11.05 1.13
50 persons, 100 joints
 Age 1.88 0.50 to 7.09 1.87 0.34 to 10.20 1.50 1.65 0.23 to 11.69 1.74
 Body-mass index 0.31 0.06 to 1.62 0.31 0.02 to 4.20 2.68 0.32 0.03 to 3.27 2.08
 Sex 3.02 0.81 to 11.28 3.02 0.42 to 21.86 2.05 2.56 0.39 to 16.79 1.57
*
Logistic regression was performed assuming that the left and right hips are independent. The generalized estimating equation and the multilevel model were used to consider the correlation between both hips.

Back to Top | Article Outline

Discussion

A considerable proportion (25%) of original orthopaedic research articles in one medical journal were found to have potentially violated the principle of statistical independence as they did not consider the possible data dependency within a subject. In our study of the prevalence of osteoarthritis, the risk factors for hip osteoarthritis showed narrower 95% confidence intervals for the odds ratios when the correlation between the right and left hip was not considered. This finding suggests that ignoring the data dependency within a subject could bias the study results. The aim of this second part of the study was not to examine the prevalence of or risk factors for hip osteoarthritis but to demonstrate the effects of correlation between the two sides on statistical analysis. These results should be discussed in terms of their statistical aspects and not their orthopaedic aspects.

Before the implications of this study are addressed, its limitations should be discussed. First, in our analysis, the studies that did not clarify the number of cases and subjects were considered to have violated statistical independence, which could have exaggerated the number of studies that actually violated the principle of statistical independence. Second, we evaluated articles from only one orthopaedic surgery journal. Third, the second part of this study, which demonstrated the effect of the method of statistical analysis and considered data dependency within a subject, could not represent all clinical or research situations. Therefore, the issue of statistical independence should be assessed on the basis of the specific research hypothesis being tested. Fourth, the second part of the study was performed in the specific setting of 100% bilateral cases with a correlation coefficient of 0.761, with specific numbers of subjects (fifty, 100, 300, and 600). We believe that the degree of the correlation, the proportion of bilateral cases, and the number of cases all can affect the study results. Consequently, these results may not be generalizable; however, we believe that the present study has shown the effect of violating the principle of statistical independence.

The use of bilateral cases in orthopaedic research has several different implications. First, statistical power and comparability could be enhanced if one side is assigned to a study group and the other side is assigned to a matched control group, because doing so could eliminate or reduce the effect of some confounding variables. Second, mixing subjects and cases can result in a disparity between subject-based results and joint-based results, sometimes leading to absurd conclusions. For instance, Andersen described a trial that resulted in the deaths of 22% of patients but only 16% of legs20. Third, the inclusion of unilateral and bilateral cases in the same group could violate the principle of statistical independence. If this principle is violated, the calculated p values or 95% confidence intervals may be spurious, within-group variabilities may be underestimated, and sample sizes may be inaccurately estimated21.

Despite the fact that several studies have raised these concerns9,11,13, data dependency has not been afforded sufficient attention.

One quarter of the articles that were examined in the present study may have violated the principle of statistical independence, and various numbers of cases and percentages of bilateral cases were reported in these articles. It is possible that both the percentage of bilateral cases and the number of study subjects can affect the robustness of the statistical analysis. However, this hypothesis was not tested in the present study. Additional study with use of a Monte Carlo simulation may identify the number of cases and the portion of bilateral cases that would be allowable so as not to introduce unacceptable bias. Several methods have been proposed, such as hierarchical modeling (also called multilevel modeling) and the use of a generalized estimating equation17 to adjust for any dependency within subjects. These methods are known to improve the statistical power of a study and provide guidance regarding the interpretation of site-specific results. We believe that investigators can obtain more precise statistical results by considering statistical independence in studies involving bilateral cases.

Investigation performed at Seoul National University Bundang Hospital, Sungnam, Kyungki, South Korea

Disclosure: The authors did not receive any outside funding or grants in support of their research for or preparation of this work. Neither they nor a member of their immediate families received payments or other benefits or a commitment or agreement to provide such benefits from a commercial entity.

Note: The authors thank Mi Seon Ryu for data collection.

Back to Top | Article Outline

References

1. Akai M. Evidence-based medicine for orthopedic practice. J Orthop Sci. 2002;7:731-42.
2. Barbier O, Hoogmartens M . Evidence-based medicine in orthopaedics. Acta Orthop Belg. 2004;70:91-7.
3. Bernstein J. Evidence-based medicine. J Am Acad Orthop Surg. 2004;12:80-8.
4. Bhandari M, Giannoudis PV . Evidence-based medicine: what it is and what it is not. Injury. 2006;37:302-6.
5. Fisher CG, Wood KB . Introduction to and techniques of evidence-based medicine. Spine (Phila Pa 1976). 2007;32(19 Suppl):S66-72.
6. Hurwitz S. Evidence-based medicine in orthopaedic surgery—a way to the future. Iowa Orthop J. 2003;23:61-5.
7. Kuhn JE, Dunn WR, Spindler KP . Evidence-based medicine for orthopedic surgeons. J Knee Surg. 2005;18:57-63.
8. Nasser R. Deciding whether there is statistical independence or not? J Math Stat. 2007;3:151-62.
9. Zhang Y, Glynn RJ, Felson DT . Musculoskeletal disease research: should we analyze the joint or the person? J Rheumatol. 1996;23:1130-4.
10. Glynn RJ, Rosner B . Comparison of alternative regression models for paired binary data. Stat Med. 1994;13:1023-36.
11. Sauerland S, Lefering R, Bayer-Sandow T, Brüser P, Neugebauer EA . Fingers, hands or patients? The concept of independent observations. J Hand Surg Br. 2003;28:102-5.
12. Glynn RJ, Rosner B . Accounting for the correlation between fellow eyes in regression analysis. Arch Ophthalmol. 1992;110:381-7.
13. Bryant D, Havey TC, Roberts R, Guyatt G . How many patients? How many limbs? Analysis of patients or limbs in the orthopaedic literature: a systematic review. J Bone Joint Surg Am. 2006;88:41-5.
14. Lau EM, Lin F, Lam D, Silman A, Croft P . Hip osteoarthritis and dysplasia in Chinese men. Ann Rheum Dis. 1995;54:965-9.
15. Ali-Gombe A, Croft PR, Silman AJ . Osteoarthritis of the hip and acetabular dysplasia in Nigerian men. J Rheumatol. 1996;23:512-5.
16. Ingvarsson T, Hägglund G, Lohmander LS . Prevalence of hip osteoarthritis in Iceland. Ann Rheum Dis. 1999;58:201-7.
17. Liang K-Y, Zeger SL . Longitudinal data analysis using generalized linear models. Biometrika. 1986;73:13-22.
18. Hox JJ. Multilevel analysis: techniques and applications. Mahwah, NJ: Lawrence Erlbaum Associates; 2002.
19. Kreft I, de Leeuw J . Introducing multilevel modeling. London: SAGE; 1998.
20. Andersen B. Methodological errors in medical research. Oxford: Blackwell Science; 1990.
21. Altman DG, Bland JM . Statistics notes. Units of analysis. BMJ. 1997;314:1874.
Copyright © 2010 by The Journal of Bone and Joint Surgery, Incorporated