Journal Logo

The Quality of Reporting of Orthopaedic Randomized Trials with Use of a Checklist for Nonpharmacological Therapies

Chan, Simon, BSc1; Bhandari, Mohit, MD1

doi: 10.2106/JBJS.F.01591
Scientific Articles
Free
Supplementary Content

Background: The Consolidated Standards of Reporting Trials statement for the reporting of randomized controlled trials has been limited by its applicability to surgical trials. In response, a Checklist to Evaluate a Report of a Nonpharmacological Trial was recently developed by the Consolidated Standards of Reporting Trials group to address reporting issues in surgical trials. We aimed (1) to apply the checklist for nonpharmacological therapies to orthopaedic randomized controlled trials across multiple journals from 2004 through 2005, and (2) to survey authors when methodological safeguards itemized in the checklist were not reported to determine whether they actually had been performed. We hypothesized that lack of reporting of a methodological safeguard did not necessarily mean it had not been conducted.

Methods: We searched for relevant orthopaedic randomized controlled trials across eight journals in the period from January 2004 through December 2005. We applied the Checklist to Evaluate a Report of a Nonpharmacological Trial to all eligible studies. We contacted authors to determine what methodological safeguards were actually used, especially when details remained unclear from the publication.

Results: We included eighty-seven randomized controlled trials from eighty-five scientific reports. In assessing the randomized controlled trials with the checklist for nonpharmacological therapies, seventy-three studies (84%) had unclear reporting of treatment allocation concealment. Only seventeen studies (20%) mentioned surgeon skill or experience. The blinding of patients, ward staff, rehabilitation staff, clinical outcome assessors, and nonclinical outcome assessors was unclear in forty-eight (55%), sixty-three (72%), sixty-four (74%), forty (46%), and thirty-three studies (38%), respectively. Authors from forty-three randomized controlled trials responded to our survey. The results of the survey showed that 41% (95% confidence interval, 25% to 58%) of the trials had adequate allocation concealment when this had been unclear from the report. Although the surgical experience of the investigators was rarely reported, most authors (70%) acknowledged that they had defined “surgical expertise criteria” such as minimum case criteria, specialized training, and clinical performance. The survey also showed that 28% to 40% of the trials had blinding of relevant groups despite the fact that the reporting of such blinding had been unclear in the publications.

Conclusions: The quality of reporting in the orthopaedic literature was highly variable. Readers should not assume that bias-reducing safeguards that were not reported in a randomized controlled trial did not occur. Our study reinforces the need for the consistent use of a tool like the Checklist to Evaluate a Report of a Nonpharmacological Trial to assess the methodology of surgical trials.

1 Hamilton Health Sciences-General Hospital, 237 Barton Street East, 7 North Wing, Suite 727, Hamilton, ON L8L 2X2, Canada. E-mail address for M. Bhandari: bhandam@mcmaster.ca

When making treatment recommendations, healthcare practitioners and practice-guideline developers look to evidence that allows them to make causal inferences about the effects of treatments. The most secure source for such evidence can be found in properly conducted randomized controlled trials (RCTs), i.e., level-I evidence. However, the strength of this evidence hinges on the methodological rigor with which the study was conducted. As an example, Altman et al. reported that a lack of allocation concealment (i.e., the assurance that investigators in a randomized trial cannot determine beforehand the treatment allocation of the next patient to be enrolled in their study) leads to an overestimation of treatment effects1.

The Consolidated Standards of Reporting Trials (CONSORT) statement2 was developed in the 1990s in an effort to improve the quality of reporting of published randomized trials in the medical literature. The CONSORT statement has been supported by a growing number of health-care journals and editorial groups since its introduction, but it is not optimized for use in evaluating surgical trials3. Recently, CONSORT developers recognized the limitations of the checklist in the evaluation of surgical interventions.

Surgical trials present unique methodological issues. It is often impossible to perform sham interventions or to blind patients and care providers4,5. The skill of the care providers can play a crucial role in the success of the treatment6. To evaluate these types of studies, a Checklist to Evaluate a Report of a Nonpharmacological Trial (CLEAR NPT) was recently developed by the CONSORT development team7.

Given the introduction and endorsement of the CLEAR NPT checklist for surgery, we aimed (1) to apply the checklist to published orthopaedic randomized trials across multiple journals from 2004 through 2005, and (2) to survey the authors of these randomized trials when items in the checklist were not reported to determine whether they actually had been conducted. In our secondary question, we hoped to determine whether “not reported” was consistent with “not conducted.” We hypothesized that lack of reporting of a methodological safeguard on the CLEAR NPT checklist did not necessarily mean it had not been conducted.

Back to Top | Article Outline

Materials and Methods

We conducted an observational study of published trials in the orthopaedic literature and a subsequent cross-sectional survey of authors to evaluate the quality of reporting. This study was approved by our university ethics board.

Back to Top | Article Outline

Eligibility Criteria

To qualify for inclusion in this investigation, a study had to meet the following criteria: (1) It had to be a nonpharmaceutical randomized trial dealing with an intervention for an orthopaedic problem performed on patients in an operating room, and (2) it had to have been published between January 1, 2004, and December 31, 2005, in the English language. Studies were excluded if they involved physiotherapy, manipulation, or acupuncture. Also excluded were animal, cadaver, and basic-science studies, as well as papers summarizing the results of a previously published study.

Back to Top | Article Outline

Study Identification

We searched for relevant trials across eight journals (four orthopaedic and four general medicine publications). These journals included The Journal of Bone and Joint Surgery (American Volume), The Journal of Bone and Joint Surgery (British Volume), Clinical Orthopaedics and Related Research, Acta Orthopaedica, British Medical Journal, Lancet, the Journal of the American Medical Association, and the New England Journal of Medicine, and they were searched for relevant studies published from January 2004 through December 2005. One investigator conducted an independent hand search of the issues of the eight journals published in 2004 and 2005. Any potentially relevant studies were retrieved for further review. Another hand search was independently conducted by three investigators who divided the eight journals among themselves. In this search, the scope was limited to all orthopaedic-related nonpharmaceutical randomized trials, as identified by review of the Materials and Methods section of each study.

To further ensure that all randomized trials published during the study period were identified, a professional librarian performed searches for orthopaedic-related randomized trials in all eight journals using both MEDLINE and EMBASE databases. The final potentially eligible list of trials from the four independent reviewers and a librarian were combined and reviewed for final inclusion in the study. Two investigators independently evaluated the abstracts of the combined search results for eligibility. Discrepancies were resolved by a consensus of the two reviewers. Consensus was achieved in all cases—those that were included and those that were excluded.

Back to Top | Article Outline

Data Abstraction

The details abstracted from each study included the following: journal name, category of trial, region of body, number of centers, and funding received.

Back to Top | Article Outline

The CLEAR NPT Checklist

A series of five papers were scored by both investigators using the CLEAR NPT checklist. The scoring of this test series was discussed, and one investigator subsequently reviewed all eligible studies for quality of reporting. The accuracy of the data abstraction was confirmed by a second investigator. Consensus was achieved on all papers. The papers were reviewed in random order as dictated by a computer-generated sequence of numbers. The checklist7 was utilized to score the quality of reporting (see Appendix). Briefly, this ten-item checklist contains questions designed to assess the reporting in nonpharmacological trials in areas crucial to internal validity, such as standardization, care-provider skill, blinding, compliance, cointerventions, and subject attrition. These questions are typically answered as yes, no, or unclear. While we adhered to the CLEAR NPT scoring guidelines as described by the authors, we also extended the scoring guidelines in attempts to achieve as much objectivity and reliability as possible.

In scoring the papers, we found it necessary to expand two questions found in the checklist. The question about outcome assessor blinding was made into two questions with regard to the blinding of the clinical outcome assessor and the nonclinical outcome assessor. The question on caregiver blinding was expanded into three questions with regard to the blinding of the surgeon, ward staff, and rehabilitation staff.

Back to Top | Article Outline

Author Survey to Resolve Uncertainties About Study Methodology

We developed a twenty-six-item questionnaire to be completed by the study authors to clarify the methodological issues probed by the CLEAR NPT (see Appendix). The questions focused on the assessment of investigator expertise and clarification of the methods of randomization, concealment, and blinding. An online version of the questionnaire was created with use of the SurveyMonkey online service (www.surveymonkey.com). The answers provided by the authors were used to provide details that were not reported or were unclear in the published study.

Fig. 1

Fig. 1

Questionnaires were e-mailed to the corresponding author of each included study. Our e-mails provided a description of our survey, as well as an Internet link to the online version of the questionnaire. If authors did not respond within three weeks, we sent follow-up e-mails at three and six weeks. Approximately eight weeks after our first e-mail, we attempted to make telephone contact with nonresponders. If we were unable to confirm receipt of our e-mail messages during a telephone call, we faxed the authors a copy of our questionnaire with an accompanying cover letter explaining the study. We were able to obtain a Japanese translation of our cover letter to fax to authors located in Japan.

Back to Top | Article Outline

Data Analysis

We summarized data descriptively. We used proportions with 95% confidence intervals to summarize categorical variables, and we used means and standard deviations to summarize continuous variables. For each aspect of methodological reporting, we determined the proportion of randomized controlled trials for which authors directly informed us that they had used the methodological safeguard, despite not explicitly reporting this in their study. Exact 95% confidence intervals were calculated for these proportions.

Back to Top | Article Outline

Results

Study Identification

Areview of the articles published from January 2004 through December 2005 revealed 613 potentially relevant reports. Application of the eligibility criteria eliminated 528 studies (Fig. 1). Eighty-five reports (13.9%) met the criteria as randomized trials concerning orthopaedic procedures that took place in the operating room. Agreement between reviewers with regard to the eligibility of the studies was moderate (κ = 0.60). Although the inclusion of orthopaedic trials seemed self-explanatory, our criteria of including only surgical interventions performed in an operating room led to some discrepancies among reviewers. Discrepancies were resolved by discussion and consensus.

Back to Top | Article Outline

Characteristics of the Randomized Trials

Two papers contained data from a total of four different randomized trials. Thus, we included eighty-seven randomized trials from eighty-five scientific reports. Seventy-seven authors were contacted in total. Four of them had authored two reports each, and two had authored three randomized controlled trial reports. Thirty randomized trials (34%) were found in the American Volume of The Journal of Bone and Joint Surgery. No eligible studies were found in either the New England Journal of Medicine or the Journal of the American Medical Association. A total of 9844 patients were included, with sample sizes ranging from eighteen to 552 participants. The average sample size (and standard deviation) across studies was 113 ± 102 participants. Sixty-seven studies (77%) were single center, forty-one studies (47%) focused on the treatment of degenerative disease, and the most studies (63%) involved problems affecting the hip or knee. The funding status was unclear in thirty-nine studies (45%) (Table I).

TABLE I - Characteristics of the Eighty-seven Randomized Trials
Characteristics No. (%) of Studies
Journal
    Acta Orthopaedica 9 (10)
    Clinical Orthopaedics and Related Research 20 (23)
    Journal of Bone and Joint Surgery (American Volume) 30 (34)
    Journal of Bone and Joint Surgery (British Volume) 25 (29)
    Lancet 2 (2)
    British Medical Journal 1 (1)
Category of trial
    Fracture treatment 18 (21)
    Treatment of degenerative disease 41 (47)
    Pain management 5 (6)
    Pediatrics 2 (2)
    Other 21 (24)
Region of body
    Upper extremity 13 (15)
    Lower extremity long bones 6 (7)
    Foot and ankle 8 (9)
    Spine 5 (6)
    Hip and knee 55 (63)
No. of centers
    1 67 (77)
    ≥2 17 (20)
    Unclear 3 (3)
Funding received by source
    Government 4 (5)
    Industry 12 (14)
    Foundation 1 (1)
    Association 1 (1)
    Combination 12 (14)
    Nonfunded 18 (21)
    Unclear 39 (45)

Back to Top | Article Outline

Quality of Reporting Assessed with the CLEAR NPT

The quality of reporting was assessed with the CLEAR NPT (Table II). Forty-one studies (47%) were unclear in describing the method of generating an allocation sequence. Of the seventy-three studies (84%) with unclear allocation concealment, thirty-three (45%) used envelopes as a means of allocation concealment, but the studies were inadequate with regard to the description of the envelopes. Only seventeen studies (20%) made any mention of surgeon skill or experience. Of those that did, most made simple statements of skill or training, leaving only four studies (5%) that provided objective measures of the skill of the surgeon (Table III). Blinding of the participants, ward staff, rehabilitation staff, clinical outcome assessors, and nonclinical outcome assessors was found to be unclear in forty-eight (55%), sixty-three (72%), sixty-four (74%), forty (46%), and thirty-three studies (38%), respectively. Forty-five studies (52%) made use of specific methods to avoid assessor bias. Due to the nature of our eligibility criteria, surgeon blinding was judged to be impossible in the vast majority of reports (eighty-six; 99%) (Table IV). Intention-to-treat analysis was judged to be used in thirty-eight studies (44%).

TABLE II - Quality of Reporting in Eighty-seven Trials Assessed with the Checklist to Evaluate a Report of a Nonpharmacological Trial
Yes (%) No (%) Unclear (%) Not Applicable*(%)
1. Was the generation of allocation sequences adequate? 47 6 47 0
2. Was the treatment allocation concealed? 3 13 84 0
3. Were details of the intervention administered to each group made available? 91 9 0 0
4. Was the experience or skill of the care providers in each arm appropriate? 20 0 80 0
5. Was participant (i.e., patient) adherence assessed quantitatively? 5 3 0 92
6. Were participants adequately blinded? 21 24 55 0
7.1. Were surgeons adequately blinded? 1 99 0 0
7.2. Were ward staff adequately blinded? 6 21 72 1
7.3. Were rehabilitation staff adequately blinded? 5 21 74 1
8. Were all other treatments and care (i.e., cointerventions) the same in each randomized group? 55 6 39 0
9. Were numbers of patients who dropped out and those lost to follow-up the same in each randomized group? 55 15 30 0
10. Were clinical outcome assessors adequately blinded to assess the primary outcomes? 21 11 46 22
11. Were nonclinical outcome assessors adequately blinded to assess the primary outcomes? 14 30 38 18
12. If outcome assessors were not adequately blinded, were specific methods used to avoid ascertainment bias (systematic differences in outcome assessment)? 52 5 25 18
13. Was the follow-up schedule the same in each group? 86 1 8 5
14. Were the main outcomes analyzed according to the intention-to-treat principle? 44 23 33 0
*
Some of the questions were not relevant, depending on the randomized controlled trial in question. As an example, some studies did not have nonclinical outcome assessors.

TABLE III - Methods Used to Ensure Skill of Care Provider
CLEAR NPT*(% of Trials) (N = 87) Author Report (% of Trials) (N = 43)
Minimum no. of case criteria 5 56
Results compared with good clinical practice outcomes 1 5
Minimum no. of case criteria and results compared with good clinical practice outcomes 0 9
Statement of skill (learning curve, orthopaedic specialist, or special training) 14 7
Uncertain 80 23
*
CLEAR NPT = Checklist to Evaluate a Report of a Nonpharmacological Trial.

TABLE IV - Data on Blinding in Trials
Checklist Items No Blinding, Not Feasible No Blinding, But It Was Feasible Total No. of Trials Without Blinding
6. Were participants adequately blinded? 20 (95%) 1 (5%) 21
7.1 Were surgeons adequately blinded? 86 (100%) 0 (0%) 86
7.2 Were ward staff adequately blinded? 17 1 18
7.3. Were rehabilitation staff adequately blinded? 17 1 18
10. Were clinical outcome assessors adequately blinded to assess the primary outcomes? 1 9 10
11. Were nonclinical outcome assessors adequately blinded to assess the primary outcomes? 25 (96%) 1 (4%) 26

Although not technically part of the CLEAR NPT, data analyst blinding and whether a sample size calculation was done a priori were also assessed. Three studies (3%) had data analyst blinding, while eighty-four (97%) were unclear. In regard to sample size calculation, twenty-eight studies (32%) performed a priori sample size calculations and one (1%) did not, while fifty-eight (67%) were unclear.

Back to Top | Article Outline

What Authors Did Not Tell Readers

We received responses regarding forty-three (49%) of the eighty-seven randomized controlled trials. Of seventy-seven authors, thirty-nine (51%) completed our questionnaire and six (8%) declined participation, leaving thirty-two (42%) who did not respond at all.

Table V presents the proportion of studies that had used a methodological safeguard, according to the authors who completed the survey, when the report itself had been unclear or ambiguous regarding that safeguard. Often, what the authors reported in our survey differed from what they had reported in the primary publication (Table VI). When concealment of allocation was unclear in the published report, almost half (41%) of the authors responded that they had in fact used sufficient methods to conceal the randomization. Although most of the authors did not report the experience (skill) of the surgeons in their reports, 70% of them acknowledged that they had set objective measures such as minimum case criteria and/or comparison with good clinical outcomes (Table III). In 28% to 40% of the studies, the authors specified that they had blinded relevant groups, despite the fact that they had not explicitly stated so in their articles. The majority of the authors used intention-to-treat analysis when it was unclear whether this was conducted in their studies. Data analysts had been blinded in seventeen studies (40%; 95% confidence interval, 22% to 53%), and a priori sample size calculations had been done in fourteen studies (67%; 95% confidence interval, 33% to 73%) when the randomized controlled trial reports had been unclear with regard to these methodological parameters.

TABLE V - Randomized Controlled Trials That Used But Failed to Report a Methodological Approach*
Randomization Used But Not Reported/Total That Used Randomization Percentage (95% Confidence Interval)
1. Was the generation of allocation sequences adequate? 13/21 62 (38-82)
2. Was the treatment allocation concealed? 15/37 41 (25-58)
4. Was experience or skill of care providers in each arm appropriate? 25/34 74 (56-87)
6. Were participants adequately blinded? 6/17 35 (14-62)
7.2 Were ward staff adequately blinded? 7/25 28 (12-49)
7.3 Were rehabilitation staff adequately blinded? 8/26 31 (14-52)
8. Were all other treatments and care (i.e., cointerventions) the same in each randomized group? 14/20 70 (46-88)
9. Were numbers of patients who dropped out and those lost to follow-up the same in each randomized group? 5/16 31 (11-59)
10. Were clinical outcome assessors adequately blinded to assess the primary outcomes? 7/20 35 (15-59)
11. Were nonclinical outcome assessors adequately blinded to assess the primary outcomes? 6/15 40 (16-68)
12. If outcome assessors were not adequately blinded, were specific methods used to avoid ascertainment bias (systematic differences in outcome assessment)? 8/12 67 (35-90)
13. Was the follow-up schedule the same in each group? 1/1 100 (3-100)
14. Were the main outcomes analyzed according to the intention-to-treat principle? 8/10 80 (44-97)
*
Data are based on results of questionnaires completed by the authors of forty-three of the eighty-seven trials.

TABLE VI - Methodology Before and After Direct Contact with Authors of Forty-three Trials
Yes (%) No (%) Unclear (%) Not Applicable (%)
Before After Before After Before After Before After
1. Was the generation of allocation sequences adequate? 47 72 5 9 49 19 0 0
2. Was the treatment allocation concealed? 5 40 9 53 86 7 0 0
4. Was experience or skill of care providers in each arm appropriate? 21 77 0 0 79 23 0 0
5. Was participant (i.e., patient) adherence assessed quantitatively? 5 35 7 9 0 56 88 0
6. Were participants adequately blinded? 23 42 37 51 40 7 0 0
7.1 Were surgeons adequately blinded? 0 12 100 81 0 7 0 0
7.2 Were ward staff adequately blinded? 5 23 35 70 58 7 2 0
7.3 Were rehabilitation staff adequately blinded? 2 28 35 65 60 7 2 0
8. Were all other treatments and care (i.e., cointerventions) the same in each randomized group? 47 79 7 12 47 7 0 2
9. Were numbers of patients who dropped out and those lost to follow-up the same in each randomized group? 44 47 19 16 37 37 0 0
10. Were clinical outcome assessors adequately blinded to assess the primary outcomes? 14 40 14 58 47 2 26 0
11. Were nonclinical outcome assessors adequately blinded to assess the primary outcomes? 21 42 30 53 35 5 14 0
12. If outcome assessors were not adequately blinded, were specific methods used to avoid ascertainment bias (systematic differences in outcome assessment)? 49 63 5 5 28 16 19 16
13. Was the follow-up schedule the same in each group? 93 93 2 0 2 7 2 0
14. Were the main outcomes analyzed according to the intention-to-treat principle? 51 70 26 19 23 12 0 0
15. Were data analysts adequately blinded? 5 40 0 53 95 7 0 0
16. Was sample size determined a priori? 37 67 2 23 60 9 0 0

Back to Top | Article Outline

Discussion

Principal Findings

In a review of eighty-seven randomized trials evaluating orthopaedic surgical therapies, we found that (1) the quality of reporting as measured by the CLEAR NPT was highly variable, (2) readers should not assume that investigators who do not describe concealment of randomization, blinding outcomes, or investigator skill in their reports did not actually use these safeguards to protect against bias (i.e., readers should not assume that “not reported” means “not performed”), and (3) the CLEAR NPT, a surgery-specific checklist, is a useful measure of surgical reporting.

The randomized controlled trial, a level-I study, can have a powerful and immediate impact on patient care. Randomization provides one single and uniformly important advantage over all other methodologies—it balances known and unknown prognostic factors between treatment groups. No degree of statistical manipulation in a nonrandomized study of two alternative treatments can correct an imbalance of patient prognostic factors. However, inadequate reporting of a randomized trial may leave readers unclear about the so-called validity of its methodology and thus may call into question the applicability of the results to patient care.

Back to Top | Article Outline

Strengths and Limitations

Our study has a number of strengths. Multiple investigators, including a librarian, were independently involved in the literature search. We achieved acceptable agreement in our eligibility decisions. We also used a surgery-specific checklist (CLEAR NPT) developed by the CONSORT development group. We were comprehensive in our attempts to contact authors. A Japanese cover letter was created for the benefit of the authors located in Japan.

Lack of clarity with regard to the reporting of blinding in the randomized controlled trial literature was reported by Montori et al., who found eight combinations of blinded groups among eighty-three randomized controlled trials reported as “double-blind.”8 The problem with the use of conventional terms such as “double blinding” was further demonstrated by Devereaux et al., who reported that physicians and textbooks varied greatly in their interpretations and definitions of single, double, and triple blinding9. For this reason, two items on the checklist were divided into multiple questions. The question on blinding of the care provider was subdivided into three questions concerning blinding of the surgeon, rehabilitation staff, and ward staff. The question on blinding of the outcome assessor was subdivided into blinding of the clinical outcome assessor and blinding of the nonclinical outcome assessor.

Although the CLEAR NPT checklist has been designed as an adjunct to the CONSORT statement for surgical therapies, its reliability has yet to be assessed. To improve the transparency and relative objectivity of the questions, we developed scoring guidelines to supplement those provided by the original CLEAR NPT paper (see Appendix). As an example, we created objective guidelines to determine whether the number of patients who dropped out and those lost to follow-up were the same between groups.

Although several attempts had been made to contact the authors, our survey response rate was just over 50%. It is plausible that authors simply did not understand the questions being posed and thus thought their study met all methodological standards. Alternatively, authors may have felt compelled to provide “socially desirable answers” (i.e., that all methodological safeguards were used in their randomized controlled trial). However, this concern is mitigated by the finding that authors acknowledged many methodological shortcomings with their studies.

Back to Top | Article Outline

Review of Relevant Literature

The CONSORT statement was developed to address the issue of quality of reporting in randomized controlled trials2. While the use of the CONSORT statement has been associated with improvements in the quality of reports of randomized controlled trials10,11, it is a tool that is not optimized for nonpharmacological trials, which present their own unique methodological issues4,5. Foley et al., in a review of eighty-one randomized controlled trials from the physiotherapy literature, found that the methodological quality for pharmacological interventions was substantially higher than that for nonpharmacological interventions12. A similar result was found in a review of the literature evaluating treatments for hip and knee osteoarthritis4. The CLEAR NPT was created with use of the Delphi consensus method in 2005 to assess the reporting of nonpharmacological studies7. Our findings support those of Jacquier et al., who reviewed a total of 158 nonpharmacological trials and reported a low quality of reporting as assessed by the CLEAR NPT13.

Devereaux et al. conducted a survey of 105 authors of published randomized controlled trials to identify whether those who failed to report concealment of randomization or blinding actually used these safeguards in their studies14. They reported that “readers should not assume that biasreducing procedures not reported in an RCT did not occur.” The authors in the full-text publications failed to report the presence or absence of concealment of randomization in 55% of the trials and the blinding of outcome assessors in 83%. When the authors were contacted directly, they frequently reported that they had used concealment of randomization (96%; 95% confidence interval, 87% to 100%) and blinding of outcome assessors (79%; 95% confidence interval, 69% to 87%), despite not having reported the use of these methodological safeguards in their studies14. We also found high variability in the reporting of concealment and blinding status in surgical trials (with unclear reporting in 84% and 46%, respectively). However, when the authors were asked, fewer reported that they had actually used these safeguards (41% and 35%, respectively).

Our study demonstrated that the proportion of orthopaedic randomized trials that failed to meet the criteria on a surgery-specific checklist ranged from 1% to 99%. Information was often inadequately reported, leaving assessments about the study methodology unclear in up to 84% of the articles. The quality of reporting in the orthopaedic literature has been previously evaluated15-17. Bhandari et al. reviewed 2468 studies, and seventy-two (2.9%) were identified as randomized trials17. More than half of the trials were limited by a lack of concealed randomization, lack of blinding of outcome assessors, or failure to report reasons for excluding patients. Those authors called for the application of standardized guidelines for the reporting of clinical trials in orthopaedics to improve quality. In a subsequent study, investigators applied the CONSORT criteria to a wide spectrum of surgical trials16. The studies adhered to an average (and standard deviation) of 36% ± 34% of critical CONSORT items, and the proportion of CONSORT items met by each trial was an average of 32% ± 29%. At the very least, authors of surgical randomized trials should endeavor to include sufficient detail to allow a careful review of the methodology.

The CLEAR NPT is an important advancement in the assessment of surgical trials over the CONSORT statement. Its focus on the reporting of interventions that require some degree of technical skill in their administration places a new value on the unique methodological considerations in conducting and reporting surgical trials. Our experience in applying the CLEAR NPT to the orthopaedic literature suggests the need for better surgical reporting. We also found that modifications to this checklist may be considered to enhance its reliability. Our work also suggests that readers of orthopaedic trials should not assume that if it was not reported, it was not done. In many circumstances, authors did use safeguards but simply did not report them. Authors, journal editors, and reviewers may consider adopting the CLEAR NPT as a standard checklist to improve the reporting of these trials.

Back to Top | Article Outline

Appendix

The annotated CLEAR NPT and the questionnaire sent to authors are available with the electronic versions of this article, on our web site at jbjs.org (go to the article citation and click on “Supplementary Material”) and on our quarterly CD-ROM (call our subscription department, at 781-449-9780, to order the CD-ROM). ▪

Disclosure: In support of their research for or preparation of this work, one or more of the authors received, in any one year, outside funding or grants of less than $10,000 from the Canadian Institutes of Health Research and of more than $10,000 from a Canada Research Chair and McMaster University. Neither they nor a member of their immediate families received payments or other benefits or a commitment or agreement to provide such benefits from a commercial entity. No commercial entity paid or directed, or agreed to pay or direct, any benefits to any research fund, foundation, division, center, clinical practice, or other charitable or nonprofit organization with which the authors, or a member of their immediate families, are affiliated or associated.

Investigation performed at the Division of Orthopaedic Surgery, McMaster University, Hamilton, Ontario, Canada

Back to Top | Article Outline

References

1. Altman DG, Schulz KF, Moher D, Egger M, Davidoff F, Elbourne D, Gotzsche PC, Lang T; CONSORT GROUP (Consolidated Standards of Reporting Trials). The revised CONSORT statement for reporting randomized trials: explanation and elaboration. Ann Intern Med. 2001;134: 663-94.
2. Begg C, Cho M, Eastwood S, Horton R, Moher D, Olkin I, Pitkin R, Rennie D, Schulz KF, Simel D, Stroup DF. Improving the quality of reporting of randomized controlled trials. The CONSORT statement. JAMA. 1996;276: 637-9.
3. Moher D, Schulz KF, Altman D; CONSORT Group (Consolidated Standards of Reporting Trials). The CONSORT statement: revised recommendations for improving the quality of reports of parallel-group randomized trials. JAMA. 2001;285: 1987-91.
4. Boutron I, Tubach F, Giraudeau B, Ravaud P. Methodological differences in clinical trials evaluating nonpharmacological and pharmacological treatments of hip and knee osteoarthritis. JAMA. 2003;290: 1062-70.
5. Boutron I, Tubach F, Giraudeau B, Ravaud P. Blinding was judged more difficult to achieve and maintain in nonpharmacologic than pharmacologic trials. J Clin Epidemiol. 2004;57: 543-50.
6. Tu JV, Hannan EL, Anderson GM, Iron K, Wu K, Vranizan K, Popp AJ, Grumbach K. The fall and rise of carotid endarterectomy in the United States and Canada. N Engl J Med. 1998;339: 1441-7.
7. Boutron I, Moher D, Tugwell P, Giraudeau B, Poiraudeau S, Nizard R, Ravaud P. A checklist to evaluate a report of a nonpharmacological trial (CLEAR NPT) was developed using consensus. J Clin Epidemiol. 2005;58: 1233-40.
8. Montori VM, Bhandari M, Devereaux PJ, Manns BJ, Ghali WA, Guyatt GH. In the dark: the reporting of blinding status in randomized controlled trials. J Clin Epidemiol. 2002;55: 787-90.
9. Devereaux PJ, Manns BJ, Ghali WA, Quan H, Lacchetti C, Montori VM, Bhandari M, Guyatt GH. Physician interpretations and textbook definitions of blinding terminology in randomized controlled trials. JAMA. 2001;285: 2000-3.
10. Moher D, Jones A, Lepage L; CONSORT Group (Consolidated Standards for Reporting of Trials). Use of the CONSORT statement and quality of reports of randomized trials: a comparative before-and-after evaluation. JAMA. 2001;285: 1992-5.
11. Plint AC, Moher D, Morrison A, Schulz K, Altman DG, Hill C, Gabaury I. Does the CONSORT checklist improve the quality of reports of randomised controlled trials? A systematic review. Med J Aust. 2006;185: 263-7.
12. Foley NC, Bhogal SK, Teasell RW, Bureau Y, Speechley MR. Estimates of quality and reliability with the physiotherapy evidence-based database scale to assess the methodology of randomized controlled trials of pharmacological and nonpharmacological interventions. Phys Ther. 2006;86: 817-24.
13. Jacquier I, Boutron I, Moher D, Roy C, Ravaud P. The reporting of randomized clinical trials using a surgical intervention is in need of immediate improvement: a systematic review. Ann Surg. 2006;244: 677-83.
14. Devereaux PJ, Choi PT, El-Dika S, Bhandari M, Montori VM, Schunemann HJ, Garg AX, Busse JW, Heels-Ansdell D, Ghali WA, Manns BJ, Guyatt GH. An observational study found that authors of randomized controlled trials frequently use concealment of randomization and blinding, despite the failure to report these methods. J Clin Epidemiol. 2004;57: 1232-6.
15. Li P, Mah D, Lim K, Sprague S, Bhandari M. Randomization and concealment in surgical trials: a comparison between orthopaedic and non-orthopaedic randomized trials. Arch Orthop Trauma Surg. 2005;125: 70-2.
16. Bhandari M, Guyatt GH, Lochner H, Sprague S, Tornetta P 3rd. Application of the Consolidated Standards of Reporting Trials (CONSORT) in the Fracture Care Literature. J Bone Joint Surg Am. 2002;84: 485-9.
17. Bhandari M, Richards RR, Sprague S, Schemitsch EH. The quality of reporting of randomized trials in the Journal of Bone and Joint Surgery from 1988 through 2000. J Bone Joint Surg Am. 2002;84: 388-96.
18. Pildal J, Chan AW, Hrobjartsson A, Forfang E, Altman DG, Gotzsche PC. Comparison of descriptions of allocation concealment in trial protocols and the published reports: cohort study. BMJ. 2005;330: 1049.
    Copyright © 2007 by The Journal of Bone and Joint Surgery, Incorporated