Criteria to Evaluate the Quality of Outcome Reporting in Randomized Controlled Trials of Rehabilitation Interventions

Abstract

No standardized guideline for the reporting of outcomes measures in randomized controlled trials of rehabilitation interventions is currently available. This study includes four phases to identify, synthesize, and make recommendations for potential attributes of reporting criteria of outcome measures in rehabilitation randomized controlled trials. First, we surveyed the author instructions for rehabilitation journals to determine how journals require authors to report outcomes. Second, we reviewed all consolidated standards of reporting trials extensions to determine how other speciality groups require reporting of outcomes in randomized controlled trials. Third, we conducted a focused scoping review to examine the nature and variations of criteria used to evaluate the quality of outcome measures in randomized controlled trials. Finally, we synthesized the information from phases 1–3 and propose four criteria specific to the reporting of outcomes in randomized controlled trials of rehabilitation interventions: (1) clearly describe the construct to be measured as outcome(s); (2) justify the selection of outcome measures by mapping to World Health Organization International Classification of Function, Disability, and Health (International Classification of Functioning) framework; justify the psychometric properties (relevance, validity, reliability) of the selected measurement tool; (3) clearly describe the timing of outcome measurement, with consideration of the health condition, the course of disease, and hypothesized effect of intervention; and (4) complete and unselective reporting of outcome data.

The judicious reporting of outcome measures in randomized controlled trials (RCTs) of rehabilitation interventions is necessary to ensure effective transfer and adoption of study results.^1–4 Moreover, adequate reporting of outcomes is required to ensure that results of RCTs can be synthesized in systematic reviews and meta-analyses.⁵ Unlike pharmacological interventions, which primarily target physiological mechanisms, rehabilitation interventions commonly aim to impact functioning at multiple levels through physical therapies, chemical agents, devices, and behavioral modalities.^3,4 In addition, rehabilitation interventions are multimodal and need to be tailored to patient needs, and treatment expectations can vary from patient to patient.⁶ This reality creates significant challenges with the reporting of outcome measures in RCTs. To date, no guideline is available to guide the reporting of outcomes measures specific to rehabilitation interventions in RCTs.

Several conceptual, clinical, and methodological challenges impact the quality and standardization of reporting of outcome measures in RCTs of rehabilitation interventions. As a whole, we refer to these challenges as the “heterogeneity conundrum.” The first level of heterogeneity resides in the fact that patients enrolled in RCTs may have similar diagnoses but different clinical presentations.^7–9 This level of heterogeneity likely impacts on patients’ responses to treatment. Second, heterogeneity exists with the timing and number of outcome measurements.⁶ Some trials measure outcomes proximal to the initiation of the intervention, whereas others measure outcomes at more distal follow-ups. Third, the type of outcomes varies significantly across trials.⁶ Some RCTs measure “mechanistic” and “intermediary” outcomes related to the process of rehabilitation, whereas others focus on participation as the end point of rehabilitation.³ The fourth level of heterogeneity deals with the need for patient centeredness of interventions.³ Heterogeneous interventions may impact response to treatment and, therefore, the measurement of outcomes. Fifth, heterogeneous methods of outcome measurement complicate the understanding of the effectiveness of rehabilitation interventions.¹⁰ Sixth, a clear description of what is a primary outcome versus a secondary outcome is a significant concern, which complicates the interpretation of trial results.⁶ Finally, heterogeneity exists with the psychometric properties of instruments or tools used to measure outcomes in RCTs. This level of heterogeneity is significant because it may cause similar interventions to have a conflicting demonstration of effectiveness across RCTs of similar conditions. Therefore, “heterogeneity conundrum” impedes the comparison and synthesis of findings from individual rehabilitation RCTs, which further limits the future conduct of high-quality rehabilitation research.

We reviewed the current practices and recommendations on the reporting of outcome in rehabilitation RCTs to inform the development of a checklist of items for the reporting of outcome measures in RCTs of rehabilitation. Our project included four specific objectives and complementary phases. First, we surveyed the author instructions of peer-reviewed rehabilitation journals to determine how authors are required to report outcomes measures in RCTs. The second phase focused on reviewing all consolidated standards of reporting trials (CONSORT) extensions to determine how other specialty groups have addressed the reporting of outcomes specific to their fields. Third, we conducted a focused scoping review of systematic reviews, clinical practice guidelines, and methodological studies on RCTs of rehabilitation interventions to examine the nature and variations of criteria used to evaluate the quality of outcome measures in systematic reviews of RCTs of rehabilitation interventions. Finally, we integrated the findings of phases 1–3 to recommend a list of criteria to be used for the reporting of outcome measures in RCTs of rehabilitation interventions.

OBJECTIVES, METHODS, AND RESULTS

Phase 1 – Survey of Author Instructions for Rehabilitation Journals

Objectives

We surveyed the requirements of rehabilitation journals regarding the reporting of outcome measures in RCTs. Specifically, we determined whether (1) journals require adherence to the CONSORT statement,¹¹ (2) specific instructions on the reporting of outcomes are provided to authors, and (3) reporting guidelines vary by journal impact factor.

Methods

Selection of Journals

We surveyed the author instructions for rehabilitation journals published in English. We identified rehabilitation journals using three strategies. First, we searched for journals containing the word “rehabilitation” listed in the University of Toledo’s Author Instructions in the Health Sciences.¹² Second, we reviewed the list of 28 journals that published the 2014 editorial entitled: “Elevating the quality of disability and rehabilitation research: Mandatory use of the reporting guidelines.”¹³ Finally, we reviewed the journals listed in the rehabilitation section in Journal Citation Reports.¹⁴

Data Collection

We reviewed the author instructions and extracted the following information, where available: (1) whether the CONSORT checklist is required for submission (including trial registration) and (2) specific instructions on the reporting of outcomes in RCTs reporting details. We consulted Journal Citation Reports and added impact factors to all journals. One coinvestigator (AT-V) extracted the data and built the tables (Appendix 1A and B, Supplemental Digital Content 1, https://links.lww.com/PHM/B138).

Results

We identified 86 rehabilitation journals. Of those, the most robust author instructions require authors to comply with reporting guidelines (e.g., CONSORT) and provide links to the guidelines for different study designs (Appendix 1A, Supplemental Digital Content 1, https://links.lww.com/PHM/B138). Only one journal, the Journal of Geriatric Physical Therapy, includes a statement referring to a CONSORT extension: “For randomized clinical trials comparing outcomes of intervention, authors should use the CONSORT-NPT 2017 Statement (Consolidated Standards of Reporting Trials, nonpharmacological treatment interventions).” (See http://edmgr.ovid.com/jgpt/accounts/ifauth.htm).

Of the 28 journals included in the editorial by Chan et al.,¹³ 24 require authors to follow the CONSORT statement. Authors submitting to one journal (Occupation, Participation, and Health) are “strongly encouraged” to follow CONSORT, and two more journals (Physical Medicine and Rehabilitation Clinics of North America, Topics in Stroke Rehabilitation) do not mention reporting guidelines but recommended that authors follow the International Committee of Medical Journal Editors recommendations (Table 1).¹⁵

TABLE 1 - Author Instructions Review: editorial by Chan et al.¹³ republished by 28 journals

Journals requiring authors to follow CONSORT (n = 24)	Am J Occup Ther, Am J Phys Med Rehabil, Arch Phys Med Rehabi, Can J Occup Ther, Clin Rehabil, Disabil Health J, Eur J Phys Rehabil Med, Int J Osteopath Med, Int J Rehabil Res, J Electromyogr Kinesiol, J Neuroeng Rehabil, J Neurol Phys Ther, J Orthop Sports Phys Ther, J Physiother, J Rehabil Med, J Sport Rehabil, Musculoskelet Sci Pract, Neural Rehabil Neural Repair, Pediatr Phys Ther, Phys Ther, Physiotherapy Can, PM R, Rehabil Nurs, Turk J Phys Med Rehabil
CONSORT “strongly encouraged” (n = 1)	OTJR: Occupation, Participation and Health
Only ICMJE stated (n = 2)	Phys Med Rehabil Clin N Am, Top Stroke Rehabil
Journal phased out (n = 1)	J Rehabil Res Dev
Journal relaunched (n = 1)	Man Ther relaunched as Musculoskelet Sci Pract

ICMJE, International Committee of Medical Journal Editors.

Most journals with impact factors require the use of CONSORT, but we found exceptions. These include the Neuropsychological Rehabilitation, IEEE Transactions on Neural Systems and Rehabilitation Engineering, Support Care Cancer, Journal of Fluency Disorders, Psychiatric Rehabilitation Journal. (See Appendix 1B, Supplemental Digital Content 1, https://links.lww.com/PHM/B138). Other journals do not mention reporting guidelines but refer authors to Enhancing the Quality And Transparency Of Health Research (EQUATOR).¹⁶ Some journals do not refer to reporting guidelines but provide a link to the recommendations of the International Committee of Medical Journal Editors.¹⁵ The International Committee of Medical Journal Editors recommendations include a section on reporting guidelines (IV. A. 2. Reporting Guidelines, p. 14).

Conclusions

We found broad variations across rehabilitation journals with the reporting of RCTs. Despite a decade of efforts to standardize the reporting of RCTs, several journals remain noncompliant. Very few journals make specific references in their author instructions to the reporting of outcome measures.

Phase 2 – Review of Official and Unofficial CONSORT Extensions

Objective

We reviewed the CONSORT extensions (see Appendix 2A, Supplemental Digital Content 1, https://links.lww.com/PHM/B138) to create an inventory of criteria/items that are specific to the reporting of outcome measures in RCTs.

Methods

We reviewed all CONSORT extensions listed on the CONSORT Web site¹⁷ (http://www.consort-statement.org/extensions). One author (DW) extracted relevant criteria/items specific to the reporting of outcome measures that are relevant to RCTs of rehabilitation interventions. We aimed to be comprehensive in our data extraction and extracted data for all criteria/items that are relevant to outcomes. We mapped the items listed in the extensions to the current CONSORT classification of reporting items.

Results

We reviewed 23 CONSORT extension statements, including 20 endorsed extensions and 3 unofficial extensions. (See Appendix 2A, Supplemental Digital Content 1, https://links.lww.com/PHM/B138, for the list of CONSORT statements). Of the reviewed 23 CONSORT extension checklists, we identified 13 CONSORT extension checklists with relevant extensions for outcome reporting in RCTs of rehabilitation interventions. Table 2 provides a summary description of relevant criteria items included in the CONSORT extensions for appraising the reporting of outcomes, which we mapped according to the CONSORT 2010 Statement.^11,18 (See Appendix 2B for more details, Supplemental Digital Content 1, https://links.lww.com/PHM/B138). Most criteria evaluating the quality of outcome reporting provide additional details for the standardized reporting of outcomes in RCTs.^19–22 Specifically, these extension criteria highlight the importance of a clear and complete description of outcome measures, including construct of the selected outcome measure and metrics of measurement conduct (e.g., scale of the measurement tool, when and how was the measurement conducted, as well as who conduct the measurement).^19,20 These extension criteria also recommend authors to sufficiently justify the validity and reliability of outcome measurement tools in their specific study settings.^19,21

TABLE 2 - Summary of criteria items for appraising reporting of outcomes in RCTs, mapping relevant items in CONSORT extensions to the CONSORT 2010 Checklist¹⁸

CONSORT 2010 Checklist Item Title	Consort 2010 Checklist Item No./Extensions	Item Contents and the Relevant Extensions
Methods
Outcomes	Criteria 6a	Completely defined prespecified primary and secondary outcome measures, including how and when they were assessed
	- Extension	- Report the methods of data collection (paper, telephone, electronic, other) and the source of the information.19
	- Extension	- Report whether an event review committee was used, and how differences of judgment or ambiguities were adjudicated.20
	- Extension	- Report if any participant training in regards to responding to included patient-reported outcome measures.19
	- Extension	- A clear justification of the validity and reliability of outcome assessment tools (variable, instrument, index test), evidence should be provided or cited if available.19,21
	- Extension	- Clarify whether outcome measures pertain to the cluster level, the individual participant level or both.22
Changes to outcomes	Criteria 6b	Any changes to trial outcomes after the trial commenced, with reasons.
Results
Participant flow	Criteria 13a	For each group, the numbers of participants who were randomly assigned, received intended treatment, and were analyzed for the primary outcome
Losses and exclusions	Criteria 13b	For each group, losses and exclusions after randomization, together with reasons
	- Extension	- The number of participants who completed outcome measurement at baseline and at subsequent time points should be made transparent.19
Baseline data	Criteria 15	A table showing baseline demographic and clinical characteristics for each group
	- Extension	- Including baseline outcome measurement when collected.19
Numbers analyzed	Criteria 16	For each group, number of participants (denominator) included in each analysis and whether the analysis was by original assigned groups
Outcome and estimation	Criteria 17a	For each primary and secondary outcome, results for each group, and the estimated effect size and its precision (such as 95% confidence interval)
Binary outcomes	Criteria 17b	For binary outcomes, presentation of both absolute and relative effect sizes is recommended
	- Extension	- For multidimensional outcome measures, results from each domain and time point should be reported.19
	- Extension	- When outcome measures pertain to the cluster level, results at the cluster level and a coefficient of intracluster correlation (ICC or k) for each primary outcome should be reported.22
Limitations	Criteria 20	Trial limitations, addressing sources of potential bias, imprecision, and, if relevant, multiplicity of analyses
	- Extension	- Outcome measure-specific limitations and implications for generalizability and clinical practice.19
Interpretation	Criteria 22	Interpretation consistent with results, balancing benefits and harms, and considering other relevant evidence
	- Extension	- When the outcome measures is patient-reporting based, the result should be interpreted in relation to clinical outcomes including survival data, where relevant.19

Conclusion

From 23 CONSORT extension statements, we identified 13 extension criteria that relate to the reporting of outcomes. These extension criteria highlight the importance of thoroughly describing the nature of the selected outcomes and emphasize the need to demonstrate that the measurement tools are valid and reliable.

Phase 3 – Scoping Review of Systematic Reviews, Clinical Practice Guidelines and Methodological Studies

Objective

We aimed to describe the methodological criteria and checklists used to evaluate the reporting and validity of outcome measures in RCTs of rehabilitation interventions.

Methods

We conducted a scoping review of systematic reviews, clinical guidelines, and methodological studies according to the methodology developed by Arksey and Levac.^23,24 We structured our report according to the Preferred Reporting Items for Systematic Reviews and Meta-Analysis.²⁵

Step 1: Identifying the Research Questions

Our scoping review aimed to answer three questions:

1. What methodological criteria and items have been used to evaluate the methodological quality and reporting of outcomes in RCTs of rehabilitation interventions?
2. What methodological concerns are raised by authors of systematic reviews, clinical guidelines, and methodological studies regarding the measurement and reporting of outcomes in RCTs of rehabilitation interventions?
3. What candidate criteria should be used in evaluating the quality of outcome reporting in RCTs of rehabilitation interventions?

Step 2 and 3: Article Identification and Selection

We developed our search strategy with an experienced health sciences librarian (AT-V). The strategy was developed in MEDLINE and adapted to Embase. We ran the search in both databases for articles published from January 1, 2000, to December 22, 2019 (Appendix 3A, Supplemental Digital Content 1, https://links.lww.com/PHM/B138). Our search strategy combined three concept groups comprised of controlled vocabulary and text words, for outcomes, rehabilitation, and RCTs. (Appendix 3A, Supplemental Digital Content 1, https://links.lww.com/PHM/B138). We limited the search to articles published in English. The eligible study included systematic reviews of RCTs of rehabilitation intervention, with quality appraisal of individual studies, and clinical practice guidelines or methodological studies on the measurement of outcomes for rehabilitation interventions. We excluded surgical studies, letters, narrative reviews, commentaries, and study protocols, as well as studies with eligible study design but having no information on the methodological quality of outcome measurement and reporting.

Starting with the most recently articles published in 2019, one author (DW) screened all citations in reverse-chronological order using the eligibility criteria. We first screened the titles and abstracts but reviewed the full text to determine eligibility if insufficient information was available in the title and abstract. We reviewed the citations in reverse-chronological order because the methodological and reporting criteria for outcome measures in RCTs have evolved significantly over the past 10 yrs. Our screening of articles published in 2018 and 2019 clearly demonstrated a saturation of the appraisal criteria used to evaluate the quality of outcome reporting. This suggested that retrieving new relevant information from older references was unlikely. Nevertheless, we screened articles back to 2000 by randomly sampling (through a computer-generated random algorithm) 10% of retrieved publications from 2000 to 2017.

Step 4: Data Extraction and Compiling

One author (DW) reviewed the full text of every included article and extracted the following information: (1) author and year of publication; (2) health condition under investigation; (3) intervention; and (4) criteria items used to evaluate the quality of outcome reporting. Some articles did not describe specific criteria but referenced a quality assessment tool; in those instances, we consulted the original tool and extracted from it the relevant criteria items. We also extracted investigators’ concerns or recommendations regarding outcome selection, measurement, and reporting from the discussion section of the articles. We extracted the articles or the identified checklist source and compiled them into two separate tables. One table provides a description of the used criteria items for evaluating the quality of outcome reporting, and the other describes the authors’ concerns or recommendations about outcome reporting.

We organized the data following three steps. First, we reported the frequency of use of the different tools and rated their use from high to low. Second, we extracted every criterion from the tools and classified them into three domains: (1) description of outcome measures; (2) measurement process of outcome(s); and (3) and quality of report of outcome data. Third, we identified and grouped unique appraisal metrics within each domain (Fig. 1). The groups of appraisal metrics informed the development of our final criteria list. We applied the same grouping strategy to compile the authors’ concerns regarding outcome reporting (Fig. 1).

FIGURE 1: Quality metrics of outcome reporting in RCTs of rehabilitation interventions.

Step 5 and 6: Summarizing, Reporting, and Consultation

All authors reviewed the data extracted in steps 3 and 4. The authors then discussed the importance of each item for the evaluation of the quality of outcome reporting in RCTs of rehabilitation interventions. We collated items appraising different quality metrics to construct our final criteria list. Specifically, we excluded items deemed irrelevant or redundant for the evaluation of outcome reporting. Items that targeted the same quality metric were combined into one item. For those one-single items that targeted appraising multiple quality metrics, we split the item into unique parts to enhance clarity and simplicity of use. As a result, each of the unique metrics in our final list may contain more than one item to reflect the detailed focuses under the same quality metric. Our final list of criteria includes a comprehensive range of quality metrics for outcome reporting in rehabilitation RCTs.

Results

We retrieved 1242 records and observed an increase of the number of publications from 11 in 2000 to 105 in 2018 (Fig. 2). We screened the eligibility of 285 articles, of which 181 were published between January 1, 2018, and December 22, 2019. We also screened 104 citations that were randomly sampled from January 1, 2000, to December 31, 2017. A total of 164 articles met the inclusion criteria. The two main reasons for exclusion were study design (n = 45) and type of intervention not related to rehabilitation (n = 38; Fig. 3).

FIGURE 2: Number of retrieved publications (systematic reviews, guidelines, and methodological studies of RCTs of rehabilitation interventions) per year from 2000 to 2019.

FIGURE 3: Flow diagram of study selection for the scoping review.

Retrieved Items for Evaluating the Quality of Outcome Reporting

We retrieved 102 items from 21 appraisal tools. (See Appendix 3B, Supplemental Digital Content 1, https://links.lww.com/PHM/B138). The use of most tools (19/21) was supported by literature, but two tools were created for the conduct of specific studies.^26,27 The items used to assess the quality reporting of outcomes measures focused on three main constructs: (1) clarity of description of outcome measure (41/102); (2) quality of outcome measurement (34/102); and (3) complete and unselective reporting outcome data (27/102).

Tools Used to Evaluate the Quality of Outcome Reporting

The most frequently used tools for assessing the risk of bias assessment were the Cochrane Collaboration tools (used in 84 of the included studies, 49.7%).^5,28–31 Overall, we identified 11 specific items from the Cochrane Collaboration tools that addressed eight constructs relevant to evaluate outcome reporting: (1) clear description of outcome measurement; (2) complete and unselective reporting of outcome data; (3) appropriateness of the selected outcome measurement; (4) comparability of outcome measurement across intervention groups (2 items); (5) blinding (2 items); (7) comparability of outcome measurement across follow-ups (2 items); and (8) description of adverse effects as a secondary outcome. (See Appendix 3B, Supplemental Digital Content 1, https://links.lww.com/PHM/B138).

The second most widely used critical appraisal tool was the PEDro scale (used in 47 of the included studies, 28.7%).³² The PEDro scale consists of four items for the assessment of the outcome reporting: (1) the comparability of outcome data at baseline; (2) blinding of outcome assessors; (3) the amount, nature, and handling of incomplete outcome data; and (4) comparability between intervention groups at baseline. (See Appendix 3B, Supplemental Digital Content 1, https://links.lww.com/PHM/B138).

Other tools used to assess the quality of outcome reporting include the tool by Downs and Black³³ (used in 9 studies, 5%), the modified Jadad scale³⁴ (used in 4 studies, 2%), the Scottish Intercollegiate Guidelines Network checklist,³⁵ and the Critical Review Form for quantitative studies³⁶ (used in 2 studies, 1%). These tools include additional items relevant to the reporting of outcomes. Specifically, the Downs and Black³³ checklist requires a clear description of outcome measurement in the introduction or methods section. The Jadad scale³⁴ requires the reporting of adverse events, whereas the Scottish Intercollegiate Guidelines Network checklist criteria³⁵ and the Critical Review Form for quantitative studies³⁶ include items about the standard, valid, and reliable measurement of relevant outcomes.

Other less frequently used tools provide seven additional items that were not included in the previously described tools.^37–44 (See Appendix 3B, Supplemental Digital Content 1, https://links.lww.com/PHM/B138). Five of these items address the selection of secondary outcomes and preference of objective and quantitative outcome measures, as well as clinically significant outcomes. The other two items target the timeline of outcome measurement, with a focus on the appropriate follow-up times.

Results From Screening Investigators’ Concerns on Outcome Reporting in RCTs of Rehabilitation Interventions

The authors of articles included in our review raised several concerns related to the reporting and measurement of outcomes in RCTs of rehabilitation interventions. (See Appendix 3C, Supplemental Digital Content 1, https://links.lww.com/PHM/B138). The most common concern was the need for standardized outcome measures, which have been explicitly validated to assess the effects of rehabilitation interventions across diverse contexts. Notably, several authors recommended the adoption of a core outcome set of patient-oriented functional outcomes such as the core outcomes set developed from the International Classification of Functioning.^27,45–53 Some authors commented on the need for the measurement of long-term outcomes^54,55 and the inclusion of outcomes to assess the cost and benefits of rehabilitation interventions.^56–58

Conclusions

We identified 12 unique quality metrics to be appraised by 24 specific criteria items (Table 3). Of those, three metrics focus on describing the outcome measures (one item): (1) description of the construct (one item); (2) description of the measurement tool (two items); and (3) justification of the selected outcome measurement tools (five items). Two metrics focus on the process of outcome measurement: (4) blindness in the measurement of outcome (four items) and (5) timelines in the measurement of outcomes (three items). The other seven metrics target the reporting of outcome data, including (6) amount (two items), (7) nature (one item), and (8) handling of missing outcome data (one item); (9) comparability of outcome data at baseline (one item); (10) potential sources of variation in the postintervention outcome measurement (one item); (11) reporting of adverse effects (one item); and (12) selective outcome reporting (one item; Table 3).

TABLE 3 - Collated list of potential criteria items in appraising outcome reporting in RCTs of rehabilitation interventions

Metric No.	Unique Criteria Items	Specific Criteria Items for Appraising Each of the Quality Metrics of Outcome Reporting in Rehabilitation RCTs
Description of outcome measures (construct, instruments, scales, etc.)
1.	Clear definition of the construct(s) used as outcome(s) or end point(s)	What is (are) the outcome(s) of interest? For multiple outcomes, clearly specify which outcomes are primary and which are secondary.
2.	Clear description of the measurement tool (instrument, scales, etc.)	a. What is the measure tool (e.g., instrument, scales) and which type of data is collected (e.g., qualitative, quantitative, scales, tests, observations). For a scale, name of the scale (e.g., The Hamilton Anxiety Rating Scale), upper and lower limits, and whether a high or low score is favorable, definitions of any thresholds if appropriate.
		b. What is the specific metric of the outcome measures? (e.g., a postintervention measure of outcome, change in outcome from baseline to a postintervention time point, postintervention presence of the observed outcome)
3.	Justification of the selected outcome measurement tool	a. Relevance and responsiveness Was the selected outcome measure relevant and responsive for measuring the intended effects of the intervention? Was a clinically significant, patient-relevant, or function-related outcome considered?
		b. Validity - Is the selected measure assessing what it is intended to measure? A valid measure should include all of the relevant concepts and elements of the outcome - When the outcome measure was not valid, there should be evidence from a previous or present scientific study that the primary outcome can be considered valid in the context of the present study - When the outcome measure was newly developed, its content validity should be tested, at least by examining its relationship to other valid measures (criterion validity) - If the outcome measure is based on an ordinal scale, then justification using IRT is preferred
		c. Reliability: whether a measure is giving the same information over different situations. (e.g., test-retest reliability, interrater reliability)
		d. Standardization: was the outcome measured in a standard way? Was a core outcome set regarding rehabilitation intervention considered? For example, ICF
		e. Comprehensiveness. (optional) Was an appropriate secondary outcome or multidimensional outcome measure considered? For example, cost-effectiveness.
Process of measurement of outcome(s)
4.	Blinded measurement of outcome(s)	a. Who is the outcome assessor(s)?
		b. Any methods used to blind outcome assessor(s)? Consider subjective outcomes and objective outcomes separately in the assessment of blinding. To get credit, - For objective outcomes, the outcome assessor should be unaware of the participant’s treatment condition; - If only self-report outcome by the participant is used, the participant should be unaware of his/her assignment to treatment condition.
		c. To which components of the trial are the outcome assessors blinded? For example, study hypothesis, details of the intervention, random assignment, outcome assessment, data analysis
		d. Implications of the intended blinding or lack of blinding in study results.
5.	Timelines in the measurement of outcome(s)	a. What is the frequency of outcome measurement? Can the measurement be administered as often as required by the study design?
		b. What are the timing points (or follow-up period) of outcome measurement (e.g., preintervention or postintervention, specific timing) Whether short-term (up to 3-mo follow-up) or/and intermediate-term (between 3 mos and 1 yr) or/and long-term (1 yr or longer follow-up) outcomes were considered? Does the length of time between assessments match the period over which the selected measure of outcome is likely to show effects?
		c. Was the timing of the outcome assessment comparable for all intervention groups and all primary outcome measures?
Report of outcome data
6.	Amount of incomplete outcome data	a. Was the amount of incomplete outcome data reported? - Report numbers of attrition and exclusions after allocation in each intervention group (compared with total randomized participants) and at each key stage of the measurement. - Measures of at least one key outcome were obtained from more than 85% of the subjects initially allocated to groups. In trials in which outcomes are measured at several points in time, a key outcome must have been measured in more than 85% of subjects at one of those points in time. (Optional)
		b. Is simple outcome data at all measurement timing points (including baseline) for all intervention groups reported? For example, numerators and denominators for a categorical outcome variable, mean and standard deviation for a continuous outcome variable. (This question does not cover statistical tests which are considered below.)
7.	Nature of incomplete outcome data	Were the reasons for attrition or exclusions in each intervention reported? This allows readers to justify the potential bias caused by the incomparable completeness of outcome data across different intervention groups.
8.	Handling of incomplete outcome data	Adequately addressed the incomplete outcome data by appropriate analysis? For example, “intention to treat” analysis, reinclusion in the analysis?
9.	Justification of comparability of outcome data at baseline	At a minimum, the report must describe at least one key outcome measure at baseline. The rater must be satisfied that the groups’ outcomes would not be expected to differ, on the basis of baseline differences in prognostic variables alone, by a clinically significant amount. (This criterion is satisfied even if only baseline data of study completers are presented.)
10.	Potential sources of variation in the postintervention outcome	For the observed variations in postintervention outcomes, were important sources of the variations discussed? Were appropriate subgroup analyses considered?
11.	Reporting of adverse effects, if any	Were adverse effects described? Adverse outcomes need special attention depending on whether they are collected systematically or nonsystematically (e.g., by voluntary report)
12.	Free of selective outcome reporting	Are reports of the study free of selective outcome reporting? State how the possibility of selective outcome reporting was examined by the review authors and what was done. The selective outcome reporting can be examined by evaluating the discrepancy between protocol and published trial report

ICF, International Classification of Functioning.

Phase 4 – Development of Recommendations

Objective

To develop a list of specific items to evaluate the quality of outcome reporting in RCTs of rehabilitation interventions.

Methods

The results of phases 1–3 were presented at a consensus meeting of the randomized controlled trials in rehabilitation checklist project in Orlando, Florida, on March 3–4, 2020.⁵⁹ The presentation included a preliminary list of candidate criteria. The meeting involved a multidisciplinary and international group of clinical and methodological experts who met to develop tools to improve the reporting of research studies in rehabilitation.⁵⁹ After the presentation of results at the meeting, the experts discussed the recommendations and voted on the preliminary list of criteria.

Results

We recommend that the following four items to be used for the evaluation of outcome reporting in RCTs of rehabilitation interventions:

1. The outcome measure (construct, instruments, scales, etc.) is clearly defined (Table 3: Item No. 1-2);
2. The selection of the outcome measure is clearly justified according to the International Classification of Functioning framework (Table 3: Item No. 3-d standardization); the outcome measure is valid, reliable, and responsive to change (Table 3: Item No. 3-a, b, c); if the outcome measure is based on an ordinal scale, then justification using Item Response Theory (IRT) is preferred;
3. The timing of outcome measurement is clearly described, with consideration of the health condition, the course of disease, and hypothesized effect of the intervention (Table 3: Item No. 5);
4. Reporting of outcome data is complete and unselective (Table 3: Item No. 6–8, 12).

DISCUSSION

We reviewed the author instructions of 86 rehabilitation journals and the CONSORT extensions to determine the need for a standard set of criteria to evaluate the reporting of outcome measures in RCTs of rehabilitation interventions. Overall, our review shows a need for a set of reporting criteria that is specific for RCTs of rehabilitation interventions. Our scoping review informed the development of a list of four items for the reporting of outcome measures in rehabilitation trials. This list of four criteria aims to evaluate the quality of outcome reporting from three main appraisal domains: description of outcome measures, the process of measurement of outcome(s), and report of outcome data. Hereinafter, we discuss the quality metrics addressed by these four criteria.

Description of Outcome Measures (Construct, Instruments, Scales, etc.)

A previous scoping review of methodological issues in reviews of rehabilitation intervention studies found that 23% of the reviews reported that a description of outcome measures was missing in RCTs.⁶⁰ Similarly, 8% indicated that there was an insufficient correlation between the delivered intervention and the measured outcome.⁶⁰ Therefore, it is critical to improve the reporting of RCTs by clearly defining the construct(s) used as outcome(s) or end point(s) in RCTs because this information is necessary to select appropriate outcome measures and the timing of outcome measurement.

Furthermore, rehabilitation RCTs need to report the measurement tool used to measure the defined outcome and justify the selection of the tool. Selected tools must be valid, reliable, and be responsive to change. This criterion is not new. In 1992, the American Congress of Rehabilitation Medicine published Measurement Standards for Interdisciplinary Medical Rehabilitation and called for the inclusion of evidence of the reliability and content, criterion, and construct validity to justify the selection and use of an outcome measure.⁶¹ However, a following 5-yr survey of the published rehabilitation studies found that the adherence to these standards is very low.⁶²

We observed that confusion may still exist with differentiating methodological concepts such as validity and reliability. To improve this situation, we provide five items to assess the relevance, validity, reliability, standardization, and comprehensiveness of outcome measures. We attach brief explanations of each item to clarify their interpretation. We highlight the need to justify the relevance of the outcome in measuring the effects of the intervention, the validity and reliability of the outcome measures in minimizing measurement error of the outcome, and whether the measurement of the outcome is standardized.

Notably, the standardized measurement of outcomes and lack of international consensus on a core outcome set has been recognized as a critical limitation in the previous rehabilitation trials (see Appendix 3C, Supplemental Digital Content 1, https://links.lww.com/PHM/B138). This not only impedes the comparison of the magnitude of the effect of different rehabilitation interventions but also limit the synthesis of the evidence from studies answering the same research question. Four of the included systematic reviews specifically recommended the use of International Classification of Functioning as the core conceptual framework to assist with the selection of outcome measures.^48,50,52,63 This is important because the multidimensional nature of mechanisms underlying the effectiveness of rehabilitation interventions warrants the use of a set of primary and secondary outcomes that evaluate the full impact of rehabilitation interventions and the understanding the mechanism of the observed effect.^9,10

If the outcomes are measured on an ordinal scale, then we recommend that investigators and authors use instruments developed after IRT instead of Classical Test Theory. The IRT holds multiple advantages to evaluate self-reported health outcomes.⁶⁴ In particular, the IRT can facilitate the evaluation of different items related to functioning, inclusion of items with different response formats within the same scale. Finally, IRT is useful to assess outcome change over time.⁶⁴

Process of Measurement of Outcome(s)

The temporal criteria of outcome measurement is an essential consideration that should be included when reporting on outcome measures. Generally, differences in the timing of outcome measurement should be classified as before, during, and short, medium- or long-term postintervention. Unlike medical interventions, which are often brief and followed by short-term measurable effects, it usually takes time for rehabilitation interventions to produce their effects. Recommendations for measuring the long-term outcome have been emphasized for rehabilitation trials. Our scoping review highlighted that the positive effects of rehabilitation interventions reported in previous publications were primarily limited to short-term outcome measurement, which makes the long-term effect of rehabilitation interventions remain unclear.^{46,54,55,63,65–69}

Rehabilitation interventions are likely to include multimodal interventions to be delivered at different timing points. Trialists, therefore, should carefully evaluate the time course of change in the outcome to determine the frequency of outcome measurement and the specific timing points of the measurement.⁴ It is also worth considering whether to assess outcomes at multiple intervals to capture the trajectory of change in outcome. Evidence of the most impactful intervention window is critical to evaluate the cost-effectiveness of the studied rehabilitation intervention and further guide the clinical practice and implementation.⁴ Across different intervention groups, it is vital to ensure that the timing of outcome measurement is identical so that the timing of outcome measurement does not likely affect the effect under observation.

Reporting of Outcome Data

Incomplete and selective reporting of outcome data is an important issue in RCTs of rehabilitation interventions.^49,70,71 Trials with multiple outcomes (e.g., rehabilitation trials) widely have the unreported data of predefined outcomes, which would be difficult to identify without access to protocols or detailed information in the methods sections. Such poor reporting of outcome data not only poses a threat to the validity of the effectiveness of an intervention but also prevents its inclusion in meta-analysis in achieving evidence synthesis to guide the clinical practice and future study design.⁷⁰

It is possible that some of the discrepancies between predefined primary outcome and reported outcome data occurred for valid reasons. For example, the use of predefined primary outcome was proven to be invalidated after the trial commencement. Study investigators then have to make an adaptation where a secondary outcome or a new outcome becomes the primary interest. The data were, therefore, only reported for the new outcomes. However, even with valid reasons to change the reporting focus, it is still essential to justify that the new outcome is free of selective reporting. Supplementary materials are acceptable if there is insufficient space in the main body of text. Journals could also deter such issues by obliging the submission of a research protocol with the study report or the inclusion of a statement if there are any deviations from the protocol. Ideally, after the initial trial registration, study investigators should make the research protocols publicly accessible before study completion. Comparison between the protocol and the report will allow readers to evaluate the existence of unreported outcomes, selective reporting of outcome data, or retrospective identification of outcome construct.

With the complete and unselective reporting of outcome data, readers can estimate the extent of selection bias, critique the appropriateness of data analysis and results interpretation, and determine the trial quality and clinical significance.

Study Strengths and Limitations

Our approach had strengths. First, we used a comprehensive three-phase strategy to develop criteria to evaluate the quality of outcome reporting in RCTs of rehabilitation interventions. Second, we systematically screened the criteria used to assess the quality of outcome reporting in RCTs of rehabilitation interventions and highlighted authors’ concerns with the current reporting of outcome measures in systematic reviews of the literature.

Nevertheless, it is possible that our review missed relevant constructs because we reviewed a random sample (10%) of relevant publications in 2000–2018. However, we assumed that most appraisal tools used in recent systematic reviews built on previous guidelines, thus reducing the risk of missing important concepts. Finally, we only searched the English literature, and it is possible that publication on other languages provided relevant information. Finally, only one author screened the literature and extracted data.

CONCLUSIONS

Given the lack of a checklist developed explicitly for quality appraisal of rehabilitation RCTs, we systematically reviewed the criteria items used to appraise the quality of outcome reporting in such trials and screened authors’ concerns regarding outcome reporting issues in RCTs of rehabilitation interventions. As a result, we generated a list of four criteria to evaluate outcome reporting in rehabilitation RCTs. These criteria are important to improve the reporting of rehabilitation RCTs. We recommend that trialists use this criteria list as a reference to facilitate the reporting of outcome construct, measurement, and measured data across follow-ups.