The Structure of Research Questions in Randomized Controlled Trials in the Rehabilitation Field

A Methodological Study

Abstract

Objective 

The aim of this study is to assess whether and how the PICO (Population, Intervention, Comparator, and Outcomes) format is described to frame research questions in randomized controlled trials looking at effectiveness of rehabilitation interventions.

Design 

A methodological study was conducted. Randomized controlled trials in the rehabilitation field, published between July 1, 2019, and December 31, 2019, were included. The framing of the primary research question from each trial was evaluated.

Results 

A total of 97 randomized controlled trials were included in the analysis. The most frequent framing of the primary research question was as an “objective” statement (55%), and in 33% of the articles, this was stated as an “objective” together with a “hypothesis” description. All PICO elements were present in 55% of research questions, but only 49% have used the statement suggested by Cochrane.

Conclusion 

The findings of this study suggest that a specific item about the “research question” and the rationale that drove the proposed design following the form suggested by Cochrane should be included in the RCT Rehabilitation Checklist.

What Is Known

• A structured research question could be associated with better methodological quality.
• A structured research question could guide the development of a research study to evaluate the effectiveness/efficacy of an intervention.

What Is New

• Currently, the primary research question of published randomized controlled trials is most often framed as an “objective.”
• Currently, few published randomized controlled trials describe the PICO elements as recommended by Cochrane.
• The findings suggest that a specific item about the “research question” and the rationale that drove the proposed design following the form suggested by Cochrane should be included in the RCT Rehabilitation Checklist.

Randomized controlled clinical trials (RCTs) are considered the gold standard study design to evaluate the effectiveness/efficacy of interventions in biomedical research.¹ The choice of an appropriate study design is informed by a clear research question (RQ).^2–5 The RQ represents the starting point for research studies to evaluate the effectiveness/efficacy of interventions because it guides the definition of the population, interventions and outcomes; consequently, this influences the development of the right study design to answer the question of interest.

Clinical epidemiologists have proposed the use of a structured RQ to guide the development of a research study that evaluates the effectiveness/efficacy of interventions.⁶ The RQ should contain the following four elements: Population, Intervention, Comparator, and Outcomes. These elements are commonly referred to by the acronym PICO. A well-structured RQ increases the likelihood of finding a solution to the problem, informs selection of the study design, and guides analysis decisions and the interpretation of results.² The explicit statement of the four PICO elements prompts the researcher to think about the design to use and to consider the balance between RQ and the feasibility to answer it. Some studies show that a structured RQ could be associated with better methodological quality, but more research is needed to confirm this finding.^2–5

In rehabilitation research, a scoping review by Arienti et al.⁷ reported a lack of clarity in RQs and that RCTs in the rehabilitation field rarely use the PICO format to define key terms. Several authors have argued that RCTs in rehabilitation frequently use inadequate designs for answering RQs related to rehabilitation and this could depend on how the RQ is formulated. The detailed specification of the RQ requires consideration of several key components that can be encapsulated by the PICO element that practicing clinicians, healthcare professionals, researchers, policy makers, and patients deal with.⁸ The accuracy of RQ framing is one of the main methodological issues described in rehabilitation research.

In recognition of this problem, Cochrane Rehabilitation highlighted the need to develop a specific checklist to guide the design, conducting, and reporting of trials in the rehabilitation field.⁹ During the second Cochrane Rehabilitation Methodological Meeting held in Kobe, Japan, in 2019, the RCT Rehabilitation Checklist (RCTRACK) project was launched to produce a reporting guideline for rehabilitation RCTs. During the kick-off meeting, eight topics were identified for the RCTRACK Technical Working Groups: one of these was the “research question.”

Therefore, the aim of this study is to assess whether and how the PICO format is described to frame RQs in RCTs about the efficacy/effectiveness of rehabilitation interventions and whether it is an important element that should be put and described in RCTRACK checklist.

METHODS

Study Design and Search Strategy

A methodological study, described as a study for assessing research methods and summarizing methodological issues in the conduct, analysis, and reporting of health research,^10,11 was conducted by the “Research Question” Technical Working Group on RCTs in the rehabilitation field published between July 1, 2019, and December 31, 2019, in journals suggested by the European Society of Physical Rehabilitation and Medicine. This study conforms to all PRISMA guidelines and reports the required information accordingly (see Supplemental Checklist, Supplemental Digital Content 1, https://links.lww.com/PHM/B146). Specific criteria^12–14 for inclusion of these journals were as follows: (1) belongs to the first quartile (Q1) according to the Journal Impact Factor (JIF) from the Web of Science Journal Citation Reports and (2) they were journals dealing with “rehabilitation” medicine and related disciplines specifically. The eligible journals were the following: Annals of Physical and Rehabilitation Medicine (JIF = 4.196), Archives of Physical Medicine and Rehabilitation (JIF = 2.697), Clinical Rehabilitation (JIF = 2.738), Disability and Rehabilitation (JIF = 2.054), European Journal of Cancer Care (JIF = 2.421), IEEE Transactions on Neural Systems and Rehabilitation Engineering (JIF = 3.478), Journal of Fluency Disorders (JIF = 2.349), Journal of Head Trauma Rehabilitation (JIF = 2.667), Journal of NeuroEngineering and Rehabilitation (JIF = 3.582), Journal of Neurologic Physical Therapy (JIF = 2.614), Journal of Orthopedic & Sports Physical Therapy (JIF = 3.058), Journal of Physiotherapy (JIF = 5.551), Manual Therapy (JIF = 2.622), Neurorehabilitation and Neural Repair (JIF = 3.757), Physical Therapy (JIF = 3.043), Physiotherapy (JIF = 2.534), PM&R—The Journal of Injury, Function and Rehabilitation (JIF = 1.902), Supportive Care in Cancer (JIF = 2.754), the American Journal of Physical Medicine & Rehabilitation (JIF = 1.908), the European Journal of Physical and Rehabilitation Medicine (JIF = 2.101), the International Journal of Rehabilitation Research (JIF = 1.378), and the Journal of Rehabilitation Medicine (JIF = 1.907). The search for eligible RCTs published in those journals was conducted on PubMed on May 12, 2020, and performed by an author (S.G. Lazzarini) as general search. See Supplementary Table 1 (Supplemental Digital Content 2, https://links.lww.com/PHM/B147) for the full search strategy.

Eligibility and Screening

All RCTs addressing an RQ regarding the effectiveness/efficacy of interventions in the field of rehabilitation published in print or ahead of print in the targeted period were included. Crossover and nonrandomized clinical trials, secondary analysis of RCT data, preliminary results, pilot studies, protocols, RCTs in which the randomization process was stratified by any factor, and articles addressing non–rehabilitation interventions were excluded.

The selection process was performed in duplicate by two independent authors (S.G. Lazzarini and M. Patrini) during (1) title and abstract and (2) full-text screening phases. A third author (C. Arienti) resolved the discrepancies.

Rating the Framing of the RQ

The methodology proposed by Rios et al.² was used to analyze how the RQ was described. In brief, the framing of the primary RQ of each study was evaluated firstly based on the introduction and secondly from the title and methodology sections. This evaluation was performed regardless of whether the RQ was formulated as a question, objective, or hypothesis. Each reviewer identified a paragraph or sections where the RQ was discussed and then identified whether the four elements of PICO were present in those sections. A “PICO score” was used, with a possible score between 0 and 4, as a measure of the completeness of the description of the primary RQ, study objective, or research hypothesis. A score of 4 confirmed that all PICO elements were described (complete PICO). Reports that did not describe these 4 elements (incomplete PICO) did not qualify as providing a structured RQ. Next, the adequacy of question formulation was evaluated based on the structure recommended by Cochrane. Cochrane proposes that the statement of an RQ should begin with a precise statement of the primary objective, ideally in a single sentence. The recommended sentence style and order is as follows: “to assess the effects of [intervention or comparison] for [health problem] in [types of people, disease or problem and setting if specified].”¹⁵ This specific order helps to clarify the aim of RCTs, enhancing a reader’s understanding of the goal of a study of the effectiveness/efficacy of an intervention. For the purpose of this study, this specific statement was defined as the “PICO structure” and a score of 1 was assigned if it was used, and a score of 0, if it was not used.

Assessment of the Quality of Reporting in Included Studies

The included studies were assessed for reporting using the Consolidated Standards of Reporting Trials (CONSORT) Statement for Randomized Trials of Nonpharmacologic Treatments checklist (CONSORT-NPTs checklist) to assess the quality of reporting in nonpharmacologic trials. This is an extension of the CONSORT checklist, developed to improve the reporting of RCTs investigating nonpharmacologic treatments.^16,17

The CONSORT-NPT checklist includes 45 items and each of them was scored 1 if it was reported and 0 if it was not clearly stated or definitely not stated. Item 4a, “Eligibility criteria for participants; When applicable, eligibility criteria for centers and for care providers,” has been split to address both topics independently. Therefore, an overall quality score (OQS) was defined with a possible value between 0 (no adherence) and 45 (complete adherence) points to measure the completeness of the reporting, that is, adherence with the CONSORT-NPT checklist. A pretraining quality of reporting assessment was performed by the reviewers (S.G. Lazzarini and M. Patrini) to define the evaluation criteria for the reporting quality. After the reporting evaluation, any disagreements were resolved involving a third reviewer (C. Arienti).

Data Extraction

A standardized data abstraction form was used to extract data from each article. The following article characteristics were collected: first author, year, title, the RQ description and type (question, objective or hypothesis format) firstly described in introduction section and secondly in title and methods section, outcome measures from each trial, rehabilitation interventions, and PICO format. Two reviewers blinded to each other’s ratings extracted data independently and rated the framing of the RQ; they resolved any disagreement through consensus.

Statistical Analysis

The percentage of trials that clearly stated each PICO element and the associated 95% confidence interval were calculated. Descriptive statistics on categorical data are reported as frequencies and percentages. Scores (i.e., PICO score and OQS) are reported as median and interquartile range. Considering the not normally distribution of the data (Shapiro Wilk test), whether a high PICO score was associated with high reporting quality was evaluated by conducting linear regression analysis with PICO score and OQS as variables using Spearman correlation coefficient (Spearman rho, r_s). Variables were considered to be statistically significant at α = 0.05. All analyses were conducted using STATA V.14.0 (StataCorp LP, College Station, TX).

RESULTS

After removal of duplicates, 227 records were screened; of these, 97 RCTs met the inclusion criteria and were included in the analysis. The characteristics of the included studies are reported in Supplementary Table 2 (Supplemental Digital Content 3, https://links.lww.com/PHM/B148). The reasons for exclusion and the number of articles excluded, at title-abstract and at full-text screening stage, are listed in the PRISMA flow diagram (Fig. 1). Twenty seven percent (n = 26) of articles were published in the journal Clinical Rehabilitation, 14% (n = 14) in Archives of Physical Medicine & Rehabilitation, 13% (n = 13) in the American Journal of Physical Medicine & Rehabilitation, and 10% (n = 10) in European Journal of Physical and Rehabilitation Medicine (see Table 1 for details of the search strategy). The most frequent framing of the primary RQ was as an “objective” statement (55%), and in 33% of the articles, this was as “objective” together with a “hypothesis” description. The frequency of each PICO element reported in all included articles is provided in Table 2. Patients, interventions, and outcomes were often adequately described, whereas in 36% of the articles, the comparison interventions were not described. All PICO elements were present in 55% of RQ, but only 49% have used the statement suggested by Cochrane. Of these, 85% had the completeness of PICO (PICO score median of 4 [2–4]).

FIGURE 1: PRISMA flow diagram.

TABLE 1 - Distribution of included studies among journals

Journals	All Articles (97)
	n	%
Clinical Rehabilitation	26	27
Archives of Physical Medicine & Rehabilitation	14	14
American Journal of Physical Medicine & Rehabilitation	13	13
European Journal of Physical and Rehabilitation Medicine	10	10
Disability and Rehabilitation	6	6
Journal of Rehabilitation Medicine	6	6
Supportive Care in Cancer	6	6
Neurorehabilitation and Neural Repair	4	4
Annals of Physical and Rehabilitation Medicine	2	2
European Journal of Cancer Care	2	2
International Journal of Rehabilitation Research	2	2
Journal of Physiotherapy	2	2
Physiotherapy	2	2
IEEE Transactions on Neural Systems and Rehabilitation Engineering	1	1
Physical Therapy	1	1

TABLE 2 - Frequency of each PICO element

Research Question	All Articles (97)
	n	%
P	95	98
I	97	100
C	62	64
O	81	84
Complete PICO	53	55
Structured PICO (Cochrane suggestion)	48	49

The CONSORT-NPT Checklist assessment revealed that the articles described 80% (36) of all checklist items, with median OQS of 36 (26–41). Items for which the lowest adherence to the checklist was found were the adherence of care providers (2%) and participants (35%) to interventions, blinding description (16%) and the description of any attempts to limit the blinding as bias (5%), the period of recruitment and follow-up description (23%), the presentation of both absolute and relative effect size in binary outcomes (15%), and the description of generalizability of the trial findings according to the intervention, comparators, patients, care providers and centers involved in the trial (44%). The highest adherence to the checklist was observed in the description of title and abstract (91%), background and aim (100%), trial design (93%), participants (100%), interventions (86%), outcomes (100%), sample size (80%), statistical methods (99%), and interpretation of results. Eighty six percent of the RCTs were registered in a trial registration database (see Supplementary Table 3, Supplemental Digital Content 4, https://links.lww.com/PHM/B149). The Spearman correlation coefficient between the completeness of PICO and the overall reporting quality was r_s = −0.051.

DISCUSSION

This study evaluated whether and how the PICO format is described to frame RQs in RCTs addressing the effectiveness/efficacy of rehabilitation interventions published in the highest-ranking rehabilitation journals during the second half of 2019.

The results showed that the most frequent framing of primary RQs was in a form of a statement about study objective using all PICO elements, but few articles followed the statement suggested by the Cochrane (PICO structure) to describe them. The comparison intervention was the least frequently described element when compared with the other elements (ie, population, intervention, and outcome). The lack of comparison intervention description is quite frequent in rehabilitation context in which establishing the control treatment is difficult because (1) this type of intervention is rarely a single specific item with a high level of heterogeneity in terms of name used for defined it and of protocol ingredients, leading to a nonlinear causal-effect relationships,¹⁸ and (2) the rehabilitation setting where the control intervention is delivered usually represents a complex clinical situation that could affect the clinical replicability of interventions.¹⁴ Therefore, this study showed that the overall reporting quality, evaluated with CONSORT-NPT checklist, was satisfactory, with 80% of reporting completeness and the best items described were those more related to PICO elements, but it was not directly related to the completeness of PICO. This could be explained by the characteristics of the CONSORT-NPT checklist that is an extension of CONSORT and includes 20 more items regarding more details on the description of the experimental treatment, comparator, care providers expertise, centers, blinding status, adherence to the protocol and the treatment, statistical methods, and the generalizability of the trial findings according to the intervention, comparators, patients, care providers, and centers involved in the trial.^16,17 These specific items represent the main methodological issues found in rehabilitation research.⁷ Most of them are related to the methodological quality rather than reporting quality, in particular to the conduct of the study that include elements such as allocation concealment (selection bias), method of blinding (performance and detection bias), incomplete outcome data (attrition bias), protocol availability (reporting bias), and compliance related biases.¹⁹ All these biases could affect the treatment estimates of RCTs and, consequently, the effectiveness/efficacy of rehabilitation interventions.²⁰ Therefore, a structured RQ could be associated with better methodological quality and could facilitate and make the RQ more understandable to guide clinicians and researchers in the literature search, in the protocol development, and in the conduct of a study² in rehabilitation research. The incompleteness and unclearness of RQs have been also found in other fields in biomedical research, such as endocrinology,²¹ urology,²² venous ulcer disease,³ surgery,²³ and anesthesia⁵ literature. These studies highlighted a significant association between the completeness of the RQ description and the quality of reporting, and this could involve the overall quality of methodology of the studies.⁶ Because the risk of bias assessment, one of the methodology quality elements, is closely linked to the quality of reporting, further research should include the evaluation of both reporting and methodological quality.²⁴ A structured RQ might be considered as a systematic way to construct the RQ and to conduct a study with the aim of giving information for clinical decision making⁶ in rehabilitation research. These considerations highlight the need to develop a specific checklist for the rehabilitation field, like RCTRACK, which includes a specific item on the framing of RQ to guide the development of future RCT studies.

The limitations were as follows: first, the PICO score and OQS are not validated and have not been rigorously tested for validity and reliability. Second, the interrater agreements were not calculated; however, the reviewers performed a pretraining reporting quality assessment to define the evaluation criteria and the disagreements were always resolved by consensus with the third reviewer.

CONCLUSION

The lack of well-designed and reported clinical trials reduces confidence in RCT results. Asking a clearly defined RQ is the first step in conducting a well-designed study. Consequently, the key implication of this study is that trialists in the rehabilitation field should pay attention to the proper framing of the RQ using a structured approach, such as the PICO format. This should comprise a precise statement of the primary objective, ideally in a single sentence as suggested by Cochrane. This clearly defined RQ should inform how the study is designed, conducted, and reported. Consequently, the “Research Question” Technical Working Group recommends that the RCTRACK checklist includes the following specific item about the “research question” “definition of the research question and rationale of the chosen design to answer to the research question described according to the PICO format.”