The judicious reporting of outcome measures in randomized controlled trials (RCTs) of rehabilitation interventions is necessary to ensure effective transfer and adoption of study results.1–4 Moreover, adequate reporting of outcomes is required to ensure that results of RCTs can be synthesized in systematic reviews and meta-analyses.5 Unlike pharmacological interventions, which primarily target physiological mechanisms, rehabilitation interventions commonly aim to impact functioning at multiple levels through physical therapies, chemical agents, devices, and behavioral modalities.3,4 In addition, rehabilitation interventions are multimodal and need to be tailored to patient needs, and treatment expectations can vary from patient to patient.6 This reality creates significant challenges with the reporting of outcome measures in RCTs. To date, no guideline is available to guide the reporting of outcomes measures specific to rehabilitation interventions in RCTs.
Several conceptual, clinical, and methodological challenges impact the quality and standardization of reporting of outcome measures in RCTs of rehabilitation interventions. As a whole, we refer to these challenges as the “heterogeneity conundrum.” The first level of heterogeneity resides in the fact that patients enrolled in RCTs may have similar diagnoses but different clinical presentations.7–9 This level of heterogeneity likely impacts on patients’ responses to treatment. Second, heterogeneity exists with the timing and number of outcome measurements.6 Some trials measure outcomes proximal to the initiation of the intervention, whereas others measure outcomes at more distal follow-ups. Third, the type of outcomes varies significantly across trials.6 Some RCTs measure “mechanistic” and “intermediary” outcomes related to the process of rehabilitation, whereas others focus on participation as the end point of rehabilitation.3 The fourth level of heterogeneity deals with the need for patient centeredness of interventions.3 Heterogeneous interventions may impact response to treatment and, therefore, the measurement of outcomes. Fifth, heterogeneous methods of outcome measurement complicate the understanding of the effectiveness of rehabilitation interventions.10 Sixth, a clear description of what is a primary outcome versus a secondary outcome is a significant concern, which complicates the interpretation of trial results.6 Finally, heterogeneity exists with the psychometric properties of instruments or tools used to measure outcomes in RCTs. This level of heterogeneity is significant because it may cause similar interventions to have a conflicting demonstration of effectiveness across RCTs of similar conditions. Therefore, “heterogeneity conundrum” impedes the comparison and synthesis of findings from individual rehabilitation RCTs, which further limits the future conduct of high-quality rehabilitation research.
We reviewed the current practices and recommendations on the reporting of outcome in rehabilitation RCTs to inform the development of a checklist of items for the reporting of outcome measures in RCTs of rehabilitation. Our project included four specific objectives and complementary phases. First, we surveyed the author instructions of peer-reviewed rehabilitation journals to determine how authors are required to report outcomes measures in RCTs. The second phase focused on reviewing all consolidated standards of reporting trials (CONSORT) extensions to determine how other specialty groups have addressed the reporting of outcomes specific to their fields. Third, we conducted a focused scoping review of systematic reviews, clinical practice guidelines, and methodological studies on RCTs of rehabilitation interventions to examine the nature and variations of criteria used to evaluate the quality of outcome measures in systematic reviews of RCTs of rehabilitation interventions. Finally, we integrated the findings of phases 1–3 to recommend a list of criteria to be used for the reporting of outcome measures in RCTs of rehabilitation interventions.
OBJECTIVES, METHODS, AND RESULTS
Phase 1 – Survey of Author Instructions for Rehabilitation Journals
We surveyed the requirements of rehabilitation journals regarding the reporting of outcome measures in RCTs. Specifically, we determined whether (1) journals require adherence to the CONSORT statement,11 (2) specific instructions on the reporting of outcomes are provided to authors, and (3) reporting guidelines vary by journal impact factor.
Selection of Journals
We surveyed the author instructions for rehabilitation journals published in English. We identified rehabilitation journals using three strategies. First, we searched for journals containing the word “rehabilitation” listed in the University of Toledo’s Author Instructions in the Health Sciences.12 Second, we reviewed the list of 28 journals that published the 2014 editorial entitled: “Elevating the quality of disability and rehabilitation research: Mandatory use of the reporting guidelines.”13 Finally, we reviewed the journals listed in the rehabilitation section in Journal Citation Reports.14
We reviewed the author instructions and extracted the following information, where available: (1) whether the CONSORT checklist is required for submission (including trial registration) and (2) specific instructions on the reporting of outcomes in RCTs reporting details. We consulted Journal Citation Reports and added impact factors to all journals. One coinvestigator (AT-V) extracted the data and built the tables (Appendix 1A and B, Supplemental Digital Content 1, https://links.lww.com/PHM/B138).
We identified 86 rehabilitation journals. Of those, the most robust author instructions require authors to comply with reporting guidelines (e.g., CONSORT) and provide links to the guidelines for different study designs (Appendix 1A, Supplemental Digital Content 1, https://links.lww.com/PHM/B138). Only one journal, the Journal of Geriatric Physical Therapy, includes a statement referring to a CONSORT extension: “For randomized clinical trials comparing outcomes of intervention, authors should use the CONSORT-NPT 2017 Statement (Consolidated Standards of Reporting Trials, nonpharmacological treatment interventions).” (See http://edmgr.ovid.com/jgpt/accounts/ifauth.htm).
Of the 28 journals included in the editorial by Chan et al.,13 24 require authors to follow the CONSORT statement. Authors submitting to one journal (Occupation, Participation, and Health) are “strongly encouraged” to follow CONSORT, and two more journals (Physical Medicine and Rehabilitation Clinics of North America, Topics in Stroke Rehabilitation) do not mention reporting guidelines but recommended that authors follow the International Committee of Medical Journal Editors recommendations (Table 1).15
TABLE 1 -
Author Instructions Review: editorial by Chan et al.13
republished by 28 journals
|Journals requiring authors to follow CONSORT (n = 24)
Am J Occup Ther, Am J Phys Med Rehabil, Arch Phys Med Rehabi, Can J Occup Ther, Clin Rehabil, Disabil Health J, Eur J Phys Rehabil Med, Int J Osteopath Med, Int J Rehabil Res, J Electromyogr Kinesiol, J Neuroeng Rehabil, J Neurol Phys Ther, J Orthop Sports Phys Ther, J Physiother, J Rehabil Med, J Sport Rehabil, Musculoskelet Sci Pract, Neural Rehabil Neural Repair, Pediatr Phys Ther, Phys Ther, Physiotherapy Can, PM R, Rehabil Nurs, Turk J Phys Med Rehabil
|CONSORT “strongly encouraged” (n = 1)
OTJR: Occupation, Participation and Health
|Only ICMJE stated (n = 2)
Phys Med Rehabil Clin N Am, Top Stroke Rehabil
|Journal phased out (n = 1)
J Rehabil Res Dev
|Journal relaunched (n = 1)
Man Ther relaunched as Musculoskelet Sci Pract
ICMJE, International Committee of Medical Journal Editors.
Most journals with impact factors require the use of CONSORT, but we found exceptions. These include the Neuropsychological Rehabilitation, IEEE Transactions on Neural Systems and Rehabilitation Engineering, Support Care Cancer, Journal of Fluency Disorders, Psychiatric Rehabilitation Journal. (See Appendix 1B, Supplemental Digital Content 1, https://links.lww.com/PHM/B138). Other journals do not mention reporting guidelines but refer authors to Enhancing the Quality And Transparency Of Health Research (EQUATOR).16 Some journals do not refer to reporting guidelines but provide a link to the recommendations of the International Committee of Medical Journal Editors.15 The International Committee of Medical Journal Editors recommendations include a section on reporting guidelines (IV. A. 2. Reporting Guidelines, p. 14).
We found broad variations across rehabilitation journals with the reporting of RCTs. Despite a decade of efforts to standardize the reporting of RCTs, several journals remain noncompliant. Very few journals make specific references in their author instructions to the reporting of outcome measures.
Phase 2 – Review of Official and Unofficial CONSORT Extensions
We reviewed the CONSORT extensions (see Appendix 2A, Supplemental Digital Content 1, https://links.lww.com/PHM/B138) to create an inventory of criteria/items that are specific to the reporting of outcome measures in RCTs.
We reviewed all CONSORT extensions listed on the CONSORT Web site17 (http://www.consort-statement.org/extensions). One author (DW) extracted relevant criteria/items specific to the reporting of outcome measures that are relevant to RCTs of rehabilitation interventions. We aimed to be comprehensive in our data extraction and extracted data for all criteria/items that are relevant to outcomes. We mapped the items listed in the extensions to the current CONSORT classification of reporting items.
We reviewed 23 CONSORT extension statements, including 20 endorsed extensions and 3 unofficial extensions. (See Appendix 2A, Supplemental Digital Content 1, https://links.lww.com/PHM/B138, for the list of CONSORT statements). Of the reviewed 23 CONSORT extension checklists, we identified 13 CONSORT extension checklists with relevant extensions for outcome reporting in RCTs of rehabilitation interventions. Table 2 provides a summary description of relevant criteria items included in the CONSORT extensions for appraising the reporting of outcomes, which we mapped according to the CONSORT 2010 Statement.11,18 (See Appendix 2B for more details, Supplemental Digital Content 1, https://links.lww.com/PHM/B138). Most criteria evaluating the quality of outcome reporting provide additional details for the standardized reporting of outcomes in RCTs.19–22 Specifically, these extension criteria highlight the importance of a clear and complete description of outcome measures, including construct of the selected outcome measure and metrics of measurement conduct (e.g., scale of the measurement tool, when and how was the measurement conducted, as well as who conduct the measurement).19,20 These extension criteria also recommend authors to sufficiently justify the validity and reliability of outcome measurement tools in their specific study settings.19,21
TABLE 2 -
Summary of criteria items for appraising reporting of outcomes in RCTs, mapping relevant items in CONSORT extensions to the CONSORT 2010 Checklist18
|CONSORT 2010 Checklist Item Title
||Consort 2010 Checklist Item No./Extensions
||Item Contents and the Relevant Extensions
||Completely defined prespecified primary and secondary outcome measures, including how and when they were assessed
||- Report the methods of data collection (paper, telephone, electronic, other) and the source of the information.19
||- Report whether an event review committee was used, and how differences of judgment or ambiguities were adjudicated.20
||- Report if any participant training in regards to responding to included patient-reported outcome measures.19
||- A clear justification of the validity and reliability of outcome assessment tools (variable, instrument, index test), evidence should be provided or cited if available.19,21
||- Clarify whether outcome measures pertain to the cluster level, the individual participant level or both.22
| Changes to outcomes
||Any changes to trial outcomes after the trial commenced, with reasons.
| Participant flow
||For each group, the numbers of participants who were randomly assigned, received intended treatment, and were analyzed for the primary outcome
| Losses and exclusions
||For each group, losses and exclusions after randomization, together with reasons
||- The number of participants who completed outcome measurement at baseline and at subsequent time points should be made transparent.19
| Baseline data
||A table showing baseline demographic and clinical characteristics for each group
||- Including baseline outcome measurement when collected.19
| Numbers analyzed
||For each group, number of participants (denominator) included in each analysis and whether the analysis was by original assigned groups
| Outcome and estimation
||For each primary and secondary outcome, results for each group, and the estimated effect size and its precision (such as 95% confidence interval)
| Binary outcomes
||For binary outcomes, presentation of both absolute and relative effect sizes is recommended
||- For multidimensional outcome measures, results from each domain and time point should be reported.19
||- When outcome measures pertain to the cluster level, results at the cluster level and a coefficient of intracluster correlation (ICC or k) for each primary outcome should be reported.22
||Trial limitations, addressing sources of potential bias, imprecision, and, if relevant, multiplicity of analyses
||- Outcome measure-specific limitations and implications for generalizability and clinical practice.19
||Interpretation consistent with results, balancing benefits and harms, and considering other relevant evidence
||- When the outcome measures is patient-reporting based, the result should be interpreted in relation to clinical outcomes including survival data, where relevant.19
From 23 CONSORT extension statements, we identified 13 extension criteria that relate to the reporting of outcomes. These extension criteria highlight the importance of thoroughly describing the nature of the selected outcomes and emphasize the need to demonstrate that the measurement tools are valid and reliable.
Phase 3 – Scoping Review of Systematic Reviews, Clinical Practice Guidelines and Methodological Studies
We aimed to describe the methodological criteria and checklists used to evaluate the reporting and validity of outcome measures in RCTs of rehabilitation interventions.
We conducted a scoping review of systematic reviews, clinical guidelines, and methodological studies according to the methodology developed by Arksey and Levac.23,24 We structured our report according to the Preferred Reporting Items for Systematic Reviews and Meta-Analysis.25
Step 1: Identifying the Research Questions
Our scoping review aimed to answer three questions:
- What methodological criteria and items have been used to evaluate the methodological quality and reporting of outcomes in RCTs of rehabilitation interventions?
- What methodological concerns are raised by authors of systematic reviews, clinical guidelines, and methodological studies regarding the measurement and reporting of outcomes in RCTs of rehabilitation interventions?
- What candidate criteria should be used in evaluating the quality of outcome reporting in RCTs of rehabilitation interventions?
Step 2 and 3: Article Identification and Selection
We developed our search strategy with an experienced health sciences librarian (AT-V). The strategy was developed in MEDLINE and adapted to Embase. We ran the search in both databases for articles published from January 1, 2000, to December 22, 2019 (Appendix 3A, Supplemental Digital Content 1, https://links.lww.com/PHM/B138). Our search strategy combined three concept groups comprised of controlled vocabulary and text words, for outcomes, rehabilitation, and RCTs. (Appendix 3A, Supplemental Digital Content 1, https://links.lww.com/PHM/B138). We limited the search to articles published in English. The eligible study included systematic reviews of RCTs of rehabilitation intervention, with quality appraisal of individual studies, and clinical practice guidelines or methodological studies on the measurement of outcomes for rehabilitation interventions. We excluded surgical studies, letters, narrative reviews, commentaries, and study protocols, as well as studies with eligible study design but having no information on the methodological quality of outcome measurement and reporting.
Starting with the most recently articles published in 2019, one author (DW) screened all citations in reverse-chronological order using the eligibility criteria. We first screened the titles and abstracts but reviewed the full text to determine eligibility if insufficient information was available in the title and abstract. We reviewed the citations in reverse-chronological order because the methodological and reporting criteria for outcome measures in RCTs have evolved significantly over the past 10 yrs. Our screening of articles published in 2018 and 2019 clearly demonstrated a saturation of the appraisal criteria used to evaluate the quality of outcome reporting. This suggested that retrieving new relevant information from older references was unlikely. Nevertheless, we screened articles back to 2000 by randomly sampling (through a computer-generated random algorithm) 10% of retrieved publications from 2000 to 2017.
Step 4: Data Extraction and Compiling
One author (DW) reviewed the full text of every included article and extracted the following information: (1) author and year of publication; (2) health condition under investigation; (3) intervention; and (4) criteria items used to evaluate the quality of outcome reporting. Some articles did not describe specific criteria but referenced a quality assessment tool; in those instances, we consulted the original tool and extracted from it the relevant criteria items. We also extracted investigators’ concerns or recommendations regarding outcome selection, measurement, and reporting from the discussion section of the articles. We extracted the articles or the identified checklist source and compiled them into two separate tables. One table provides a description of the used criteria items for evaluating the quality of outcome reporting, and the other describes the authors’ concerns or recommendations about outcome reporting.
We organized the data following three steps. First, we reported the frequency of use of the different tools and rated their use from high to low. Second, we extracted every criterion from the tools and classified them into three domains: (1) description of outcome measures; (2) measurement process of outcome(s); and (3) and quality of report of outcome data. Third, we identified and grouped unique appraisal metrics within each domain (Fig. 1). The groups of appraisal metrics informed the development of our final criteria list. We applied the same grouping strategy to compile the authors’ concerns regarding outcome reporting (Fig. 1).
Step 5 and 6: Summarizing, Reporting, and Consultation
All authors reviewed the data extracted in steps 3 and 4. The authors then discussed the importance of each item for the evaluation of the quality of outcome reporting in RCTs of rehabilitation interventions. We collated items appraising different quality metrics to construct our final criteria list. Specifically, we excluded items deemed irrelevant or redundant for the evaluation of outcome reporting. Items that targeted the same quality metric were combined into one item. For those one-single items that targeted appraising multiple quality metrics, we split the item into unique parts to enhance clarity and simplicity of use. As a result, each of the unique metrics in our final list may contain more than one item to reflect the detailed focuses under the same quality metric. Our final list of criteria includes a comprehensive range of quality metrics for outcome reporting in rehabilitation RCTs.
We retrieved 1242 records and observed an increase of the number of publications from 11 in 2000 to 105 in 2018 (Fig. 2). We screened the eligibility of 285 articles, of which 181 were published between January 1, 2018, and December 22, 2019. We also screened 104 citations that were randomly sampled from January 1, 2000, to December 31, 2017. A total of 164 articles met the inclusion criteria. The two main reasons for exclusion were study design (n = 45) and type of intervention not related to rehabilitation (n = 38; Fig. 3).
Retrieved Items for Evaluating the Quality of Outcome Reporting
We retrieved 102 items from 21 appraisal tools. (See Appendix 3B, Supplemental Digital Content 1, https://links.lww.com/PHM/B138). The use of most tools (19/21) was supported by literature, but two tools were created for the conduct of specific studies.26,27 The items used to assess the quality reporting of outcomes measures focused on three main constructs: (1) clarity of description of outcome measure (41/102); (2) quality of outcome measurement (34/102); and (3) complete and unselective reporting outcome data (27/102).
Tools Used to Evaluate the Quality of Outcome Reporting
The most frequently used tools for assessing the risk of bias assessment were the Cochrane Collaboration tools (used in 84 of the included studies, 49.7%).5,28–31 Overall, we identified 11 specific items from the Cochrane Collaboration tools that addressed eight constructs relevant to evaluate outcome reporting: (1) clear description of outcome measurement; (2) complete and unselective reporting of outcome data; (3) appropriateness of the selected outcome measurement; (4) comparability of outcome measurement across intervention groups (2 items); (5) blinding (2 items); (7) comparability of outcome measurement across follow-ups (2 items); and (8) description of adverse effects as a secondary outcome. (See Appendix 3B, Supplemental Digital Content 1, https://links.lww.com/PHM/B138).
The second most widely used critical appraisal tool was the PEDro scale (used in 47 of the included studies, 28.7%).32 The PEDro scale consists of four items for the assessment of the outcome reporting: (1) the comparability of outcome data at baseline; (2) blinding of outcome assessors; (3) the amount, nature, and handling of incomplete outcome data; and (4) comparability between intervention groups at baseline. (See Appendix 3B, Supplemental Digital Content 1, https://links.lww.com/PHM/B138).
Other tools used to assess the quality of outcome reporting include the tool by Downs and Black33 (used in 9 studies, 5%), the modified Jadad scale34 (used in 4 studies, 2%), the Scottish Intercollegiate Guidelines Network checklist,35 and the Critical Review Form for quantitative studies36 (used in 2 studies, 1%). These tools include additional items relevant to the reporting of outcomes. Specifically, the Downs and Black33 checklist requires a clear description of outcome measurement in the introduction or methods section. The Jadad scale34 requires the reporting of adverse events, whereas the Scottish Intercollegiate Guidelines Network checklist criteria35 and the Critical Review Form for quantitative studies36 include items about the standard, valid, and reliable measurement of relevant outcomes.
Other less frequently used tools provide seven additional items that were not included in the previously described tools.37–44 (See Appendix 3B, Supplemental Digital Content 1, https://links.lww.com/PHM/B138). Five of these items address the selection of secondary outcomes and preference of objective and quantitative outcome measures, as well as clinically significant outcomes. The other two items target the timeline of outcome measurement, with a focus on the appropriate follow-up times.
Results From Screening Investigators’ Concerns on Outcome Reporting in RCTs of Rehabilitation Interventions
The authors of articles included in our review raised several concerns related to the reporting and measurement of outcomes in RCTs of rehabilitation interventions. (See Appendix 3C, Supplemental Digital Content 1, https://links.lww.com/PHM/B138). The most common concern was the need for standardized outcome measures, which have been explicitly validated to assess the effects of rehabilitation interventions across diverse contexts. Notably, several authors recommended the adoption of a core outcome set of patient-oriented functional outcomes such as the core outcomes set developed from the International Classification of Functioning.27,45–53 Some authors commented on the need for the measurement of long-term outcomes54,55 and the inclusion of outcomes to assess the cost and benefits of rehabilitation interventions.56–58
We identified 12 unique quality metrics to be appraised by 24 specific criteria items (Table 3). Of those, three metrics focus on describing the outcome measures (one item): (1) description of the construct (one item); (2) description of the measurement tool (two items); and (3) justification of the selected outcome measurement tools (five items). Two metrics focus on the process of outcome measurement: (4) blindness in the measurement of outcome (four items) and (5) timelines in the measurement of outcomes (three items). The other seven metrics target the reporting of outcome data, including (6) amount (two items), (7) nature (one item), and (8) handling of missing outcome data (one item); (9) comparability of outcome data at baseline (one item); (10) potential sources of variation in the postintervention outcome measurement (one item); (11) reporting of adverse effects (one item); and (12) selective outcome reporting (one item; Table 3).
TABLE 3 -
Collated list of potential criteria items in appraising outcome
reporting in RCTs of rehabilitation
||Unique Criteria Items
||Specific Criteria Items for Appraising Each of the Quality Metrics of Outcome Reporting in Rehabilitation RCTs
|Description of outcome measures (construct, instruments, scales, etc.)
||Clear definition of the construct(s) used as outcome(s) or end point(s)
||What is (are) the outcome(s) of interest?
For multiple outcomes, clearly specify which outcomes are primary and which are secondary.
||Clear description of the measurement tool (instrument, scales, etc.)
||a. What is the measure tool (e.g., instrument, scales) and which type of data is collected (e.g., qualitative, quantitative, scales, tests, observations). For a scale, name of the scale (e.g., The Hamilton Anxiety Rating Scale), upper and lower limits, and whether a high or low score is favorable, definitions of any thresholds if appropriate.
|b. What is the specific metric of the outcome measures? (e.g., a postintervention measure of outcome, change in outcome from baseline to a postintervention time point, postintervention presence of the observed outcome)
||Justification of the selected outcome measurement tool
||a. Relevance and responsiveness
Was the selected outcome measure relevant and responsive for measuring the intended effects of the intervention? Was a clinically significant, patient-relevant, or function-related outcome considered?
- Is the selected measure assessing what it is intended to measure? A valid measure should include all of the relevant concepts and elements of the outcome
- When the outcome measure was not valid, there should be evidence from a previous or present scientific study that the primary outcome can be considered valid in the context of the present study
- When the outcome measure was newly developed, its content validity should be tested, at least by examining its relationship to other valid measures (criterion validity)
- If the outcome measure is based on an ordinal scale, then justification using IRT is preferred
|c. Reliability: whether a measure is giving the same information over different situations. (e.g., test-retest reliability, interrater reliability)
|d. Standardization: was the outcome measured in a standard way? Was a core outcome set regarding rehabilitation intervention considered? For example, ICF
|e. Comprehensiveness. (optional) Was an appropriate secondary outcome or multidimensional outcome measure considered? For example, cost-effectiveness.
|Process of measurement of outcome(s)
||Blinded measurement of outcome(s)
||a. Who is the outcome assessor(s)?
|b. Any methods used to blind outcome assessor(s)? Consider subjective outcomes and objective outcomes separately in the assessment of blinding. To get credit,
- For objective outcomes, the outcome assessor should be unaware of the participant’s treatment condition;
- If only self-report outcome by the participant is used, the participant should be unaware of his/her assignment to treatment condition.
|c. To which components of the trial are the outcome assessors blinded? For example, study hypothesis, details of the intervention, random assignment, outcome assessment, data analysis
|d. Implications of the intended blinding or lack of blinding in study results.
||Timelines in the measurement of outcome(s)
||a. What is the frequency of outcome measurement?
Can the measurement be administered as often as required by the study design?
|b. What are the timing points (or follow-up period) of outcome measurement (e.g., preintervention or postintervention, specific timing) Whether short-term (up to 3-mo follow-up) or/and intermediate-term (between 3 mos and 1 yr) or/and long-term (1 yr or longer follow-up) outcomes were considered? Does the length of time between assessments match the period over which the selected measure of outcome is likely to show effects?
|c. Was the timing of the outcome assessment comparable for all intervention groups and all primary outcome measures?
|Report of outcome data
||Amount of incomplete outcome data
||a. Was the amount of incomplete outcome data reported?
- Report numbers of attrition and exclusions after allocation in each intervention group (compared with total randomized participants) and at each key stage of the measurement.
- Measures of at least one key outcome were obtained from more than 85% of the subjects initially allocated to groups. In trials in which outcomes are measured at several points in time, a key outcome must have been measured in more than 85% of subjects at one of those points in time. (Optional)
|b. Is simple outcome data at all measurement timing points (including baseline) for all intervention groups reported? For example, numerators and denominators for a categorical outcome variable, mean and standard deviation for a continuous outcome variable. (This question does not cover statistical tests which are considered below.)
||Nature of incomplete outcome data
||Were the reasons for attrition or exclusions in each intervention reported?
This allows readers to justify the potential bias caused by the incomparable completeness of outcome data across different intervention groups.
||Handling of incomplete outcome data
||Adequately addressed the incomplete outcome data by appropriate analysis? For example, “intention to treat” analysis, reinclusion in the analysis?
||Justification of comparability of outcome data at baseline
||At a minimum, the report must describe at least one key outcome measure at baseline. The rater must be satisfied that the groups’ outcomes would not be expected to differ, on the basis of baseline differences in prognostic variables alone, by a clinically significant amount.
(This criterion is satisfied even if only baseline data of study completers are presented.)
||Potential sources of variation in the postintervention outcome
||For the observed variations in postintervention outcomes, were important sources of the variations discussed? Were appropriate subgroup analyses considered?
||Reporting of adverse effects, if any
||Were adverse effects described? Adverse outcomes need special attention depending on whether they are collected systematically or nonsystematically (e.g., by voluntary report)
||Free of selective outcome reporting
||Are reports of the study free of selective outcome reporting? State how the possibility of selective outcome reporting was examined by the review authors and what was done. The selective outcome reporting can be examined by evaluating the discrepancy between protocol and published trial report
ICF, International Classification of Functioning.
Phase 4 – Development of Recommendations
To develop a list of specific items to evaluate the quality of outcome reporting in RCTs of rehabilitation interventions.
The results of phases 1–3 were presented at a consensus meeting of the randomized controlled trials in rehabilitation checklist project in Orlando, Florida, on March 3–4, 2020.59 The presentation included a preliminary list of candidate criteria. The meeting involved a multidisciplinary and international group of clinical and methodological experts who met to develop tools to improve the reporting of research studies in rehabilitation.59 After the presentation of results at the meeting, the experts discussed the recommendations and voted on the preliminary list of criteria.
We recommend that the following four items to be used for the evaluation of outcome reporting in RCTs of rehabilitation interventions:
- The outcome measure (construct, instruments, scales, etc.) is clearly defined (Table 3: Item No. 1-2);
- The selection of the outcome measure is clearly justified according to the International Classification of Functioning framework (Table 3: Item No. 3-d standardization); the outcome measure is valid, reliable, and responsive to change (Table 3: Item No. 3-a, b, c); if the outcome measure is based on an ordinal scale, then justification using Item Response Theory (IRT) is preferred;
- The timing of outcome measurement is clearly described, with consideration of the health condition, the course of disease, and hypothesized effect of the intervention (Table 3: Item No. 5);
- Reporting of outcome data is complete and unselective (Table 3: Item No. 6–8, 12).
We reviewed the author instructions of 86 rehabilitation journals and the CONSORT extensions to determine the need for a standard set of criteria to evaluate the reporting of outcome measures in RCTs of rehabilitation interventions. Overall, our review shows a need for a set of reporting criteria that is specific for RCTs of rehabilitation interventions. Our scoping review informed the development of a list of four items for the reporting of outcome measures in rehabilitation trials. This list of four criteria aims to evaluate the quality of outcome reporting from three main appraisal domains: description of outcome measures, the process of measurement of outcome(s), and report of outcome data. Hereinafter, we discuss the quality metrics addressed by these four criteria.
Description of Outcome Measures (Construct, Instruments, Scales, etc.)
A previous scoping review of methodological issues in reviews of rehabilitation intervention studies found that 23% of the reviews reported that a description of outcome measures was missing in RCTs.60 Similarly, 8% indicated that there was an insufficient correlation between the delivered intervention and the measured outcome.60 Therefore, it is critical to improve the reporting of RCTs by clearly defining the construct(s) used as outcome(s) or end point(s) in RCTs because this information is necessary to select appropriate outcome measures and the timing of outcome measurement.
Furthermore, rehabilitation RCTs need to report the measurement tool used to measure the defined outcome and justify the selection of the tool. Selected tools must be valid, reliable, and be responsive to change. This criterion is not new. In 1992, the American Congress of Rehabilitation Medicine published Measurement Standards for Interdisciplinary Medical Rehabilitation and called for the inclusion of evidence of the reliability and content, criterion, and construct validity to justify the selection and use of an outcome measure.61 However, a following 5-yr survey of the published rehabilitation studies found that the adherence to these standards is very low.62
We observed that confusion may still exist with differentiating methodological concepts such as validity and reliability. To improve this situation, we provide five items to assess the relevance, validity, reliability, standardization, and comprehensiveness of outcome measures. We attach brief explanations of each item to clarify their interpretation. We highlight the need to justify the relevance of the outcome in measuring the effects of the intervention, the validity and reliability of the outcome measures in minimizing measurement error of the outcome, and whether the measurement of the outcome is standardized.
Notably, the standardized measurement of outcomes and lack of international consensus on a core outcome set has been recognized as a critical limitation in the previous rehabilitation trials (see Appendix 3C, Supplemental Digital Content 1, https://links.lww.com/PHM/B138). This not only impedes the comparison of the magnitude of the effect of different rehabilitation interventions but also limit the synthesis of the evidence from studies answering the same research question. Four of the included systematic reviews specifically recommended the use of International Classification of Functioning as the core conceptual framework to assist with the selection of outcome measures.48,50,52,63 This is important because the multidimensional nature of mechanisms underlying the effectiveness of rehabilitation interventions warrants the use of a set of primary and secondary outcomes that evaluate the full impact of rehabilitation interventions and the understanding the mechanism of the observed effect.9,10
If the outcomes are measured on an ordinal scale, then we recommend that investigators and authors use instruments developed after IRT instead of Classical Test Theory. The IRT holds multiple advantages to evaluate self-reported health outcomes.64 In particular, the IRT can facilitate the evaluation of different items related to functioning, inclusion of items with different response formats within the same scale. Finally, IRT is useful to assess outcome change over time.64
Process of Measurement of Outcome(s)
The temporal criteria of outcome measurement is an essential consideration that should be included when reporting on outcome measures. Generally, differences in the timing of outcome measurement should be classified as before, during, and short, medium- or long-term postintervention. Unlike medical interventions, which are often brief and followed by short-term measurable effects, it usually takes time for rehabilitation interventions to produce their effects. Recommendations for measuring the long-term outcome have been emphasized for rehabilitation trials. Our scoping review highlighted that the positive effects of rehabilitation interventions reported in previous publications were primarily limited to short-term outcome measurement, which makes the long-term effect of rehabilitation interventions remain unclear.46,54,55,63,65–69
Rehabilitation interventions are likely to include multimodal interventions to be delivered at different timing points. Trialists, therefore, should carefully evaluate the time course of change in the outcome to determine the frequency of outcome measurement and the specific timing points of the measurement.4 It is also worth considering whether to assess outcomes at multiple intervals to capture the trajectory of change in outcome. Evidence of the most impactful intervention window is critical to evaluate the cost-effectiveness of the studied rehabilitation intervention and further guide the clinical practice and implementation.4 Across different intervention groups, it is vital to ensure that the timing of outcome measurement is identical so that the timing of outcome measurement does not likely affect the effect under observation.
Reporting of Outcome Data
Incomplete and selective reporting of outcome data is an important issue in RCTs of rehabilitation interventions.49,70,71 Trials with multiple outcomes (e.g., rehabilitation trials) widely have the unreported data of predefined outcomes, which would be difficult to identify without access to protocols or detailed information in the methods sections. Such poor reporting of outcome data not only poses a threat to the validity of the effectiveness of an intervention but also prevents its inclusion in meta-analysis in achieving evidence synthesis to guide the clinical practice and future study design.70
It is possible that some of the discrepancies between predefined primary outcome and reported outcome data occurred for valid reasons. For example, the use of predefined primary outcome was proven to be invalidated after the trial commencement. Study investigators then have to make an adaptation where a secondary outcome or a new outcome becomes the primary interest. The data were, therefore, only reported for the new outcomes. However, even with valid reasons to change the reporting focus, it is still essential to justify that the new outcome is free of selective reporting. Supplementary materials are acceptable if there is insufficient space in the main body of text. Journals could also deter such issues by obliging the submission of a research protocol with the study report or the inclusion of a statement if there are any deviations from the protocol. Ideally, after the initial trial registration, study investigators should make the research protocols publicly accessible before study completion. Comparison between the protocol and the report will allow readers to evaluate the existence of unreported outcomes, selective reporting of outcome data, or retrospective identification of outcome construct.
With the complete and unselective reporting of outcome data, readers can estimate the extent of selection bias, critique the appropriateness of data analysis and results interpretation, and determine the trial quality and clinical significance.
Study Strengths and Limitations
Our approach had strengths. First, we used a comprehensive three-phase strategy to develop criteria to evaluate the quality of outcome reporting in RCTs of rehabilitation interventions. Second, we systematically screened the criteria used to assess the quality of outcome reporting in RCTs of rehabilitation interventions and highlighted authors’ concerns with the current reporting of outcome measures in systematic reviews of the literature.
Nevertheless, it is possible that our review missed relevant constructs because we reviewed a random sample (10%) of relevant publications in 2000–2018. However, we assumed that most appraisal tools used in recent systematic reviews built on previous guidelines, thus reducing the risk of missing important concepts. Finally, we only searched the English literature, and it is possible that publication on other languages provided relevant information. Finally, only one author screened the literature and extracted data.
Given the lack of a checklist developed explicitly for quality appraisal of rehabilitation RCTs, we systematically reviewed the criteria items used to appraise the quality of outcome reporting in such trials and screened authors’ concerns regarding outcome reporting issues in RCTs of rehabilitation interventions. As a result, we generated a list of four criteria to evaluate outcome reporting in rehabilitation RCTs. These criteria are important to improve the reporting of rehabilitation RCTs. We recommend that trialists use this criteria list as a reference to facilitate the reporting of outcome construct, measurement, and measured data across follow-ups.
This research was undertaken, in part, thanks to funding from the Canada Research Chairs program to Professor Pierre Côté who holds the Canada Research Chair in Disability Prevention and Rehabilitation at Ontario Tech University. The authors are grateful for the postdoctoral fellowship funded by the Ontario Tech University and the Canadian Memorial Chiropractic College, Ontario, Canada to Dr. Dan Wang. The authors thank the Randomized Controlled Trial Rehabilitation Checklists–Working Groups for their invaluable insights and discussions on our work during the consensus meeting held in Orlando (US) during the 4th Cochrane Rehabilitation Methodology Meeting (CRMM) in March 2020.
1. Haigh R, Tennant A, Biering-Sorensen F, et al.: The use of outcome
measures in physical medicine and rehabilitation
within Europe. J Rehabil Med
2. Gianola S, Castellini G, Corbetta D, et al.: Rehabilitation
interventions in randomized controlled trials for low back pain: proof of statistical significance often is not relevant. Health Qual Life Outcomes
3. Levack WM, Malmivaara A, Meyer T, et al.: Methodological problems in rehabilitation
research. Report from a Cochrane rehabilitation methodology
meeting. Eur J Phys Rehabil Med
4. Wade DT: Outcome
measures for clinical rehabilitation
trials: impairment, function, quality of life, or value?Am J Phys Med Rehabil
5. Higgins JP, Green S: Cochrane Handbook for Systematic Reviews of Interventions
. Vol 4. Chichester, UK: John Wiley & Sons; 2011
6. Mayo NE, Kaur N, Barbic SP, et al.: How have research questions and methods used in clinical trials published in clinical rehabilitation
changed over the last 30 years?Clin Rehabil
7. Vargus-Adams JN, Majnemer A: International Classification Of Functioning, Disability and Health (ICF) as a framework for change: revolutionizing rehabilitation
. J Child Neurol
8. World Health Organization. International Classification of Functioning, Disability, and Health: ICF. 2001. Available at: https://www.who.int/classifications/icf/en/
. Accessed July 11, 2020
9. Engkasan JP, Ahmad-Fauzi A, Sabirin S, et al.: Mapping the primary outcomes reported in Cochrane systematic reviews regarding stroke with the International Classification of Functioning, Disability and Health domains: current trend and future recommendations. Eur J Phys Rehabil Med
10. Stucki G, Pollock A, Engkasan JP, et al.: How to use the International Classification of Functioning, Disability and Health as a reference system for comparative evaluation and standardized reporting of rehabilitation
interventions. Eur J Phys Rehabil Med
11. Schulz KF, Altman DG, Moher D; CONSORT Group: CONSORT 2010 statement: updated guidelines for reporting parallel group randomized trials. Ann Intern Med
12. Mulford Health Sciences Library. University of Toledo. Instructions to Authors in the Health Sciences. Available at: http://mulford.utoledo.edu/instr/
. Accessed January 27, 2020
13. Chan L, Heinemann AW, Roberts J: Elevating the quality of disability and rehabilitation
research: mandatory use of the reporting guidelines. Ann Phys Rehabil Med
14. Journal Citation Reports. Clarivate Web of ScienceTM. Available at: https://clarivate.com/webofsciencegroup/solutions/journal-citation-reports/
. Accessed January 27, 2020
15. International Committee of Medical Journal Editors (ICMJE). Recommendations for the Conduct, Reporting, Editing, and Publication of Scholarly work in Medical Journals. 2019. Available at: http://www.icmje.org/icmje-recommendations.pdf
. Accessed January 27, 2020
16. EQUATOR. EQUATOR network: Enhancing the QUAlity and Transparency Of Health Research. Available at: https://www.equator-network.org/
. Accessed January 27, 2020
17. CONSORT Group. The CONSORT Statement. Consolidated Standards of Reporting Trials. Available at: http://www.consort-statement.org/
. Accessed July 11, 2020
18. Schulz KF, Altman DG, Moher D; CONSORT Group: CONSORT 2010 statement: updated guidelines for reporting parallel group randomised trials. BMC Med
19. Calvert M, Blazeby J, Altman DG, et al.: Reporting of patient-reported outcomes in randomized trials: the CONSORT PRO extension. JAMA
20. Davidson KW, Goldstein M, Kaplan RM, et al.: Evidence-based behavioral medicine: what is it and how do we achieve it?Ann Behav Med
21. Vohra S, Shamseer L, Sampson M, et al.: CONSORT extension for reporting N-of-1 trials (CENT) 2015 statement. BMJ
22. Campbell MK, Piaggio G, Elbourne DR, et al.; CONSORT Group: Consort 2010 statement: extension to cluster randomised trials. BMJ
23. Arksey H, O’Malley L: Scoping studies: towards a methodological framework. Int J Soc Res Method
24. Levac D, Colquhoun H, O’Brien KK: Scoping studies: advancing the methodology
. Implement Sci
25. Tricco AC, Lillie E, Zarin W, et al.: PRISMA Extension for Scoping Reviews (PRISMA-ScR): checklist and explanation. Ann Intern Med
26. Helminski JO, Zee DS, Janssen I, et al.: Effectiveness of particle repositioning maneuvers in the treatment of benign paroxysmal positional vertigo: a systematic review. Phys Ther
27. Rietberg MB, Brooks D, Uitdehaag BM, et al.: Exercise therapy for multiple sclerosis. Cochrane Database Syst Rev
28. Higgins JPT, Thomas J, Chandler J, et al.: Cochrane Handbook for Systematic Reviews of Interventions
, 2nd ed. Chichester, UK, John Wiley & Sons, 2019
29. Sterne JAC, Savović J, Page MJ, et al.: RoB 2: a revised tool for assessing risk of bias in randomised trials. BMJ
30. Van Tulder M, Furlan A, Bombardier C, et al.; Editorial Board of the Cochrane Collaboration Back Review Group: Updated method guidelines for systematic reviews in the cochrane collaboration back review group. Spine (Phila Pa 1976)
31. Furlan AD, Malmivaara A, Chou R, et al.: 2015 updated method guideline for systematic reviews in the Cochrane Back and Neck Group. Spine
32. Maher CG, Sherrington C, Herbert RD, et al.: Reliability of the PEDro scale for rating quality of randomized controlled trials. Phys Ther
33. Downs SH, Black N: The feasibility of creating a checklist for the assessment of the methodological quality both of randomised and non-randomised studies of health care interventions. J Epidemiol Community Health
34. Oremus M, Wolfson C, Perrault A, et al.: Interrater reliability of the modified Jadad quality scale for systematic reviews of Alzheimer’s disease drug trials. Dement Geriatr Cogn Disord
35. Scottish Intercollegiate Guidelines Network. Methodology
checklist 2: Randomised controlled trials. 2012. Available at: https://www.sign.ac.uk/assets/checklist_for_controlled_trials.doc
. Accessed July 11, 2020
36. Law M, Stewart D, Pollock N, et al.: Guidelines for critical review form – quantitative studies. McMaster University Occupational Therapy Evidence-based Practice Research Group
. 1998. Available at: https://srs-mcmaster.ca/wp-content/uploads/2015/05/Guidelines-for-Critical-Review-Form-Quantitative-Studies.pdf
. Accessed July 11, 2020
37. Wykes T, Steel C, Everitt B, et al.: Cognitive behavior therapy for schizophrenia: effect sizes, clinical models, and methodological rigor. Schizophr Bull
38. Suh N, Benson EC, Faber KJ, et al.: Treatment of acute scaphoid fractures: a systematic review and meta-analysis. Hand (N Y)
39. Darrah J, Hickman R, O’Donnell M, et al.: AACPDM Methodology to Develop Systematic Reviews of Treatment Interventions (Revision 1.2)
. Milwaukee, WI, American Academy for Cerebral Palsy and Developmental Medicine, 2008:1–30. Available at: https://www.aacpdm.org/UserFiles/file/systematic-review-methodology.pdf
. Accessed July 11, 2020
40. Jonsson U, Choque Olsson N, Bölte S: Can findings from randomized controlled trials of social skills training in autism spectrum disorder be generalized? The neglected dimension of external validity. Autism
41. Kmet LM, Cook LS, Lee RC: Standard quality assessment
criteria for evaluating primary research papers from a variety of fields. 2004. Available at: https://www.ihe.ca/publications/standard-quality-assessment-criteria-for-evaluating-primary-research-papers-from-a-variety-of-fields
. Accessed July 11, 2020
42. National Heart, Blood and Lung Institute. Quality Assessment
of Controlled Intervention Studies. Bethesda, MD: U.S. Department of Health & Human Services. Available at: https://www.nhlbi.nih.gov/health-topics/study-quality-assessment-tools
. Accessed January 20, 2020
43. Critical Appraisal Skills Programme. CASP checklists. Critical Appraisal Skills Programme (CASP): Making sense of evidence 2014. Available at: https://casp-uk.net/
. Accessed February 20, 2020
44. Sindhu F, Carpenter L, Seers K: Development of a tool to rate the quality assessment
of randomized controlled trials using a Delphi technique. J Adv Nurs
45. Amatya B, Khan F, Galea M: Rehabilitation
for people with multiple sclerosis: an overview of Cochrane reviews. Cochrane Database Syst Rev
46. Barker F, Mackenzie E, Elliott L, et al.: Interventions to improve hearing aid use in adult auditory rehabilitation
. Cochrane Database Syst Rev
47. Campbell E, Coulter EH, Mattison PG, et al.: Physiotherapy rehabilitation
for people with progressive multiple sclerosis: a systematic review. Arch Phys Med Rehabil
48. Fisicaro F, Lanza G, Grasso AA, et al.: Repetitive transcranial magnetic stimulation in stroke rehabilitation
: review of the current evidence and pitfalls. Ther Adv Neurol Disord
49. Jacobsen PB, DeRosa AP, Henderson TO, et al.: Systematic review of the impact of Cancer survivorship care plans on health outcomes and health care delivery. J Clin Oncol
50. Khan F, Amatya B: Rehabilitation
interventions in patients with acute demyelinating inflammatory polyneuropathy: a systematic review. Eur J Phys Rehabil Med
51. Maidment DW, Barker AB, Xia J, et al.: A systematic review and meta-analysis assessing the effectiveness of alternative listening devices to conventional hearing aids in adults with hearing loss. Int J Audiol
52. Mirkowski M, McIntyre A, Faltynek P, et al.: Nonpharmacological rehabilitation
interventions for motor and cognitive outcomes following pediatric stroke: a systematic review. Eur J Pediatr
53. Williamson PR, Altman DG, Blazeby JM, et al.: Developing core outcome
sets for clinical trials: issues to consider. Trials
54. Cano Porras D, Siemonsma P, Inzelberg R, et al.: Advantages of virtual reality in the rehabilitation
of balance and gait: systematic review. Neurology
55. Harvey LA, Brosseau L, Herbert RD: Continuous passive motion following total knee arthroplasty in people with arthritis. Cochrane Database Syst Rev
56. Hwang R, Bruning J, Morris N, et al.: A systematic review of the effects of telerehabilitation in patients with cardiopulmonary diseases. J Cardiopulm Rehabil Prev
57. Legg LA, Lewis SR, Schofield-Robinson OJ, et al.: Occupational therapy for adults with problems in activities of daily living after stroke. Cochrane Database Syst Rev
58. Tomlinson CL, Patel S, Meek C, et al.: Physiotherapy intervention in Parkinson’s disease: systematic review and meta-analysis. BMJ
59. Negrini S, Armijo-Olivo S, Patrini M, et al.; RCTRACK Promoters: The randomized controlled trials rehabilitation
of development of a reporting guideline specific to rehabilitation
. Am J Phys Med Rehabil
60. Cochrane Rehabilitation
. A catalogue of methodological issues in rehabilitation
research. Available at: https://rehabilitation.cochrane.org/resources/methodology/catalogue-methodological-issues-rehabilitation-research
. Accessed February 20, 2020
61. Johnston MV, Keith RA, Hinderer SR: Measurement standards for interdisciplinary medical rehabilitation
. Arch Phys Med Rehabil
62. Dijkers MP, Kropp GC, Esper RM, et al.: Reporting on reliability and validity of outcome
measures in medical rehabilitation
research. Disabil Rehabil
63. Amatya B, Galea MP, Kesselring J, et al.: Effectiveness of telerehabilitation interventions in persons with multiple sclerosis: a systematic review. Mult Scler Relat Disord
64. Hays RD, Morales LS, Reise SP: Item response theory and health outcomes measurement in the 21st century. Med Care
65. Hao Z, Wang D, Zeng Y, et al.: Repetitive transcranial magnetic stimulation for improving function after stroke. Cochrane Database Syst Rev
66. Momosaki R, Yamada N, Ota E, et al.: Repetitive peripheral magnetic stimulation for activities of daily living and functional ability in people after stroke. Cochrane Database Syst Rev
67. Lloyd-Williams F, Mair FS, Leitner M: Exercise training and heart failure: a systematic review of current evidence. Br J Gen Pract
68. Ownsworth T, Arnautovska U, Beadle E, et al.: Efficacy of telerehabilitation for adults with traumatic brain injury: a systematic review. J Head Trauma Rehabil
69. Tsoi WS, Zhang LA, Wang WY, et al.: Improving quality of life of children with cerebral palsy: a systematic review of clinical trials. Child Care Health Dev
70. Chan AW, Hrobjartsson A, Haahr MT, et al.: Empirical evidence for selective reporting of outcomes in randomized trials: comparison of protocols to published articles. JAMA
71. Silva IS, Pedrosa R, Azevedo IG, et al.: Respiratory muscle training in children and adults with neuromuscular disease. Cochrane Database Syst Rev