Journal Logo

Editorials

The Right Study Design for the Right Study Question

Gershman, Evgeni MD; Shojaee, Samira MD, MPH

Author Information
Journal of Bronchology & Interventional Pulmonology: April 2020 - Volume 27 - Issue 2 - p 75-76
doi: 10.1097/LBR.0000000000000661
  • Free

In 2000, Concato and colleagues challenged the established consensus on the hierarchy of clinical research design by comparing the results of well-designed observational studies with randomized controlled trials (RCT) of the same topic. In their study, they reported no overestimation of the magnitude of the association between exposure and outcomes of RCTs and well-designed observational studies.1 Other investigators have shown that effect estimates in observational studies are most similar to RCTs when similar exclusion criteria are used and prognostic factors accounted for.2 Despite these reports, RCTs hold the foremost position in the pyramid of evidence-based medicine.

Meanwhile, the ultimate reproach of observational studies is their vulnerability to the effect of unrecognized confounding factors which may distort the results. Irrespective of the perceived hierarchy of evidence, the appropriate choice of study design should integrate consideration of research question, cost, access to data, required epidemiologic measures, and the level of existing evidence on the exposure-outcome relationship of interest. Although RCTs are considered as evidence of highest grade, they can be difficult to conduct or even impossible and unethical in some instances such as for various surgical interventions. Other shortcomings exist such as prohibitive cost when attempting to identify risk factors, treatment safety, and prognostic indicators, where observational studies are best suited. In addition, rigorously selected and monitored inclusion and exclusion criteria may limit external validity and make RCTs an inferior choice for prognostic model development.

Moreover, RCTs are not an appropriate choice of study for specific research questions such as evaluating the causes of rare diseases which may be far more efficiently and effectively conducted through a retrospective case-control design, while cross-sectional studies may be the most suitable design for ecological, diagnostic, and screening studies. The field of interventional pulmonology (IP), a primarily procedural-based field, also faces some challenges in research design. The comparison of procedural interventions is sometimes unethical, blinding occasionally impossible and study outcomes may suffer from short follow-up periods because of limited patient survival.

In this issue of the journal, Porcel et al3 present their experience over 5 and a half years of using indwelling tunneled pleural catheters (IPC) to treat recurrent pleural effusions. The authors attempt to sort out factors that may predict IPC removal, and IPC-related infection, using a single-center retrospective observational study design. A total of 308 patients receiving 336 IPCs were included in the study, majority of whom suffered from malignant pleural effusion (MPE) (83%). Pleurodesis occurred in 51% of the patients at a median time of 52 days. Eastern Cooperative Oncology Group (ECOG) grade of 0 to 2, expandable lung physiology, and the presence of pleural space septation were associated with higher rates of IPC removal. Furthermore, the presence of hepatic hydrothorax and pleural fluid C-reactive protein of <15 mg/dL were noted to be the strongest predictors of pleural space infection development.

Despite numerous studies into the utility, effect, and safety of IPCs for recurrent pleural effusion, questions remain regarding the role of IPCs in pleurodesis and the factors that influence successful pleurodesis post-IPC placement. In addition, the subject of IPC-related infection, contributing factors, and treatment strategies have been and still remain a topic of debate. Acknowledging the existing gaps in the literature and challenging the disadvantages of IPC use such as infection and long-term IPC drainage regimen, Porcel and colleagues attempt to investigate the predictors of IPC-related infection and pleurodesis.

The authors should be congratulated as this study enjoys numerous strengths in its design and concept. The investigators attempt to answer a question that is highly significant, where the prediction of desirable and palliative outcomes, a primarily patient-centric question, is especially important in the era of personalized medicine. The authors define variables and outcomes a priori, which is important in every research design and especially crucial when data are being collected retrospectively. Their use of a competing risk model and a relatively large sample size are among other strengths in design. The study, however, also suffers from a list of limitations in its quest to explore and ultimately uncover the predictors of outcome. Although using an observational study, which is a suitable method for the study question, there is potential bias and diminished generalizability of data because of its single-center and retrospective design. In addition, the study fails to include infection-related IPC removal as a competing risk in its competing risk analysis and finally, the inclusion of a heterogenous population of MPE and non-MPE and a low event (infections) rate further impedes the accuracy of effect estimates.

Despite its limitations, the study by Porcel and colleagues adds to the existing knowledge on IPCs by defining the gaps in evidence, clarifying its aim, and designing a study that leads to additional hypotheses to be investigated. It is advantageous that the selected variables in this study are routinely measured in clinical practice and can be used in larger studies to develop prognostic models. Although such models can sometimes be developed from RCTs with liberal inclusion and exclusion criteria,4 most prognostic models are developed from retrospective or prospective multicenter well-designed observational studies. The development of such prognostic models are made easier through collaborative research that includes efficient extraction of data from multiple research institutes5,6 and designated IP collaborative research networks such as IP outcomes group.7 Finally, regardless of the research quality, reporting of observational study results is equally important and should be handled with utmost care. Published guidelines exist that can help investigators approach their observational research design and study reporting in a thoughtful manner. The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) group provides a 22-item checklist to optimize the reporting of observational studies, adopted by many journals across disciplines.8

Given the prevalent use of IPCs for symptom management, it is necessary to develop prognostic models in well-designed observational studies.

REFERENCES

1. Concato J, Shah N, Horwitz RI. Randomized, controlled trials, observational studies, and the hierarchy of research designs. N Engl J Med. 2000;342:1887–1892.
2. McMcKee M, Britton A, Black N, et al. Methods in health services research: interpreting the evidence: choosing between randomised and non-randomised studies. BMJ. 1999;319:312–315.
3. Porcel J, Torres M, Pardina M, et al. Predictors of indwelling pleural catheter removal and infection: a single-center experience with 336 procedures J Bronchol Interv Pulmonol. 2019. Doi: 10.1097/LBR.0000000000000632.
4. Rahman N, Kahan B, Miller R, et al. A clinical score (RAPID) to identify those at risk for poor outcome at presentation in patients with pleural infection. Chest. 2014;145:848–855.
5. O’Connell OJ, Almeida FA, Simoff MJ, et al. A prediction model to help with the assessment of adenopathy in lung cancer: HAL. Am J Respir Crit Care Med. 2017;195:1651–1660.
6. Martinez-Zayas G, Almeida FA, Simoff MJ, et al. A prediction model to Help with Oncologic Mediastinal Evaluation for Radiation: HOMER. Am J Respir Crit Care Med. 2020;201:212–223.
7. Maldonado F, Yarmus L. Pragmatic studies in interventional pulmonology: two steps forward, one step back, but an imminent leap forward. Introducing IPOG, the Interventional Pulmonary Outcome Group. J Bronchol Interv Pulmonol. 2019;26:150–152.
8. Von Elm E, Altman DG, Egger M, et al. The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) statement: guidelines for reporting observational studies. J Clin Epidemiol. 2008;61:344–349.
Copyright © 2020 Wolters Kluwer Health, Inc. All rights reserved.