Share this article on:

What Constitutes a Well-Designed Pilot Study?

McGrath, Jacqueline M., PhD, RN, FNAP, FAAN; Brandon, Debra, PhD, RN, CCNS, FAAN

doi: 10.1097/ANC.0000000000000535
Letter From the Editors

Coeditor; Advances in Neonatal Care Jacqueline.mcgrath@uconn.edu

Coeditor; Advances in Neonatal Care debra.brandon@duke.edu

The editors declare no conflict of interest.

Dear NANN Colleagues,

A well-designed pilot study is an important contribution to the literature. Unlike some journals, Advances in Neonatal Care publishes pilot studies. As such increasing understanding about what constitutes a well-designed pilot study is important to both authors (researchers) and our readership. Thus, this editorial provides an overview of the components of well-designed pilot research.

The specific purpose of a pilot study is to provide data to inform future work. Therefore, a pilot study is typically designed to pilot one or more of the following: an idea, innovation, or concept, or more precisely to test the safety, feasibility, acceptability, or efficacy of study designs or methods.1–4 Simply put, pilot studies are about examining new concepts or novel ideas. This could possibly be a study testing a new intervention, or new piece of equipment or a strategy that has worked in one population that is now being tested in another population (first tested with late preterm infants and now being tested with extremely early born infants). In the laboratory, pilot studies are often referred to as “proof of concept” and are considered a routine aspect of laboratory science. In the laboratory, the researchers may do the experiment with a very small sample just to see what happens; they conduct the experiment based on an educated “hunch” about what relationships might exist.5 , 6 Sometimes effects are discovered accidently during these experiments. As you might guess, in the clinical setting experiments based on educated hunches might not be justified or ethical, given the potential risks to our patients and families.

It is important to note that these investigations are not just studies executed with small samples because the available convenience sample was small. By definition, pilot studies have small samples on purpose, such that once the concept is found to be safe or feasible, the plan is to continue to do further testing of the novel concept in larger samples.3–5 For (nurse) scientists, pilot studies constitute the first steps in developing an idea. These preliminary steps (pilot studies) are an important aspect of a planned research trajectory and are meant (designed intentionally) to build knowledge in a particular area of science.7 Thus, the discussion of future research (next steps) is an important aspect of the pilot study report.6 In particular, the discussion needs to emphasize the limitations of the current study, particularly what can and cannot be generalized from the results as well as the next studies that need to occur.6 Pilot studies are generally underpowered (sample not large enough to reach an appropriate level of significance). And thus, because of the use of a small sample size, the results are less significant and can only be used to generate beliefs that they will provide a trend toward significance guiding in the development of larger studies. Even if there are positive findings, the researchers cannot recommend the intervention or idea being tested for integration into practice. What might be said is “this is a promising concept” that with further research could be integrated into practice in the future. In other words, “stay tuned for further details.” Pilot studies are often the preliminary studies (small samples) used to build the science to support the need for further testing of an intervention in full-scale randomized control trial.1 , 8 One caveat that needs to be considered is that “if” the concept under study is of low risk to the population of interest than possibly cautionary recommendations for practice can be implied but this is not typical in neonatal research since our population is generally considered a high-risk population.

We are not trying to diminish the importance of pilot studies with these cautionary statements. Evidence-based practice guidelines place pilot studies, even when well-designed, at one of the lowest levels of evidence. As such further scrutiny of the results is required, and the level of risk must be considered if integration of findings into practice is done without further research testing the concept. Yet, pilot studies are important to neonatal care for several reasons.9 Research involving neonates and their families is considered high risk and as such the pretesting of novel concepts with smaller samples is quite appropriate since we want to guard against risk until a concept, idea, or bundle (group of interventions) is ready for full-scale testing.9 Furthermore, choosing the sample size is important even in pilot studies and must be done thoughtfully with intention. The sample needs be large enough to provide direction for refinement of the intervention in next studies (for example, establishing effect size) and yet, small enough to not expose unnecessarily at-risk populations to greater risks.9 , 10

In the clinical setting, there are basically 4 types of pilot studies: (1) safety (is it safe; no change in infant stability and does not increase other risks); (2) feasibility (will it work); (3) acceptability (will clinicians, participants, and families find the intervention reasonable); and (4) efficacy (does implementation make a difference).8 In the literature the definitions of these different types of pilot studies are overlapping. Not all types of pilot studies are appropriate for testing all concepts or designs; however, safety, feasibility, acceptability, and efficacy studies do build upon each other and do build the science that is often necessary when the risks are high or the population of interest is already considered high risk, as is the case when conducting research with neonates.7 To increase understanding please find next an example of each type of pilot study.

Back to Top | Article Outline

SAFETY EXAMPLE

A new intervention is proposed for preterm infants with feeding problems, but the feeding intervention is first tested with well newborns who would have greater tolerance for the intervention and would be thus at less risk for adverse effects. Once tested (and refined as needed given infant behaviors and physiology stability) with the more stable population, the inclusion/exclusion criteria could be altered in the next study design to the true population of interest (preterm infants). Designing studies in sequence in this way allows researchers to test and refine the intervention while controlling for extraneous variables and yet also decrease risk for the study participants.11

Back to Top | Article Outline

FEASIBILITY EXAMPLE

An instrument to measure infant behaviors related to drug exposure was initially tested in full-term infants. Now the researchers want to see whether the same instrument could be used with preterm infants who have also been drug exposed. Feasibility studies examine study designs, methods, instruments, or data collection methods for effects on the results. Are the data collected appropriate or not to achieve the intended results, given the strategies that were employed in the study design?11 , 12

Back to Top | Article Outline

ACCEPTABILITY EXAMPLE

An intervention is proposed to increase family engagement, and during this trial the focus would be on whether families find out whether families believe the timing and their involvement works for them or not. Issues that might be addressed are whether the measures used are burdensome to families to complete (too long to complete or difficult to understand). Can they get to the neonatal intensive care unit at the time the intervention is planned or not? As you can see, the focus is not particularly on the outcome of the intervention but more on the process for how the intervention is delivered to families.11 , 12 It is important to note that sometimes feasibility and acceptability data can be collected concurrently in the same pilot study; however, this can sometimes be cumbersome and burdensome to participants and may not provide clarity about next steps that are important to why the researcher is testing the intervention in a pilot trial.13

Back to Top | Article Outline

EFFICACY EXAMPLE

In a small trial (no randomization), a new intervention for supporting an infant during a retinopathy examination is tested to see whether the intervention is efficacious—makes a difference in short- and long-term behavioral and physiologic outcomes for the infant. The sample is small, and the plan is with refinement of the intervention strategy (as needed) from this pilot study to test in a full-scale trial with a larger sample.14 , 15

What is not a pilot study is any research with a small sample size. Within Advances in Neonatal Care, we publish studies with small samples. However, when considering the overall significance of any study (implications for practice and future research), there is always the concern that a study with a small sample will not generate statistically or clinically significant results. Consequently, researchers might, after the fact, call their studies with small samples, pilot studies to justify sample size, study design and/or methods and to make the results appear more publishable.16 , 17 During the manuscript review process, we might ask authors to retitle their manuscripts when the research presented is not truly a pilot study. Even if the results are meaningful, generalizing results from studies with small samples can only be done with extreme caution. In addition, making unwarranted claims about potential generalizability of the results from studies with small samples can be misleading to readers that may have limited knowledge of research design and methods. These studies with small samples are often the result of recruiting from a population of convenience (one location) with limited time to conduct the work (ie, study must be completed in 6-12 months) and possibly the result of not enough funding to complete the work.5 , 7 Given this weakness (the chance that results presented could be misleading), some journals will not publish pilot studies. However, studies with small samples may provide direction (positive or negative outcomes) and guidance about effect sizes that can be useful for future research.

In conclusion, pilot studies are important to building neonatal science. However, pilot studies must be executed with intent and reported clearly. Study aims must be specific to study purpose (safety, feasibility, acceptability, and/or efficacy) and appropriate for the level of evidence that exists about the topic or concept.9 , 10 The presentation of pilot studies needs to be succinct and direct, so readers are able to interpret results appropriately. This means authors need to provide clear information about what is known and what is not known about a phenomenon and what research designs, methodologies, and protocol processes are being tested so that readers do not draw inappropriate conclusions.14 , 15 Reports need to include information about how the results should or should not be used in the clinical setting and about what the next steps in the research trajectory need to be. Clearly delineating these items in the research report is important to building the science, as well as protecting our patients and families. Given the novelty of neonatal science, we believe in publishing pilot studies, they are important to growing neonatal science yet at the same time we need to clearly state the facts and findings, so all who read and consider using the results for further research or to guide practice understand the true level of the evidence.

Thank you to our readership and for your submissions to Advances in Neonatal Care,

Jacqueline M. McGrath, PhD, RN, FNAP, FAAN

Coeditor; Advances in Neonatal Care

Jacqueline.mcgrath@uconn.edu

Debra Brandon, PhD, RN, CCNS, FAAN

Coeditor; Advances in Neonatal Care

debra.brandon@duke.edu

Back to Top | Article Outline

References

1. Altman D, Burton N, Cuthill I, et al Why Do a Pilot Study? London, England: National Centre for the Replacement, Refinement and Reduction in Animals in Research; 2006;1–2.
2. Arian M, Campbell MJ, Cooper CL, Lancaster GA. What is a pilot or feasibility study? A review of current practice and editorial policy. BMC Med Res Methodol. 2010;10(67):1–7.
3. Chi KR. Conducting a trial. Nature. 2013;493:565–567.
4. Moore CG, Carter RE, Nietert PJ, Stewart PW. Recommendations for planning pilot studies in clinical and translational research. Clin Transl Sci. 2011;4(5):332–337. doi:10.1111/j.1752-8062.2011.00347.x.
5. Leon AC, Davis LL, Kraemer HC. The role and interpretation of pilot studies in clinical research. J Psychiatr Res. 2011;45:626–629.
6. Eldridge SM, Lancaster GA, Campbell MJ, et al Defining feasibility and pilot studies in preparation for randomised controlled trials: development of a conceptual framework. PLoS One. 2016;11(3):e0150205. doi:10.1371/journal.pone.0150205.
7. Conn VS. Building a trajectory of research: editorial. West J Nurs Res. 2004;26(6):592–594.
8. van Teijlingen ER, Rennie AM, Hundley V, et al The importance of conducting and reporting pilot studies: the example of the Scottish Births Survey. J Adv Nurs. 2001;34(3):289–295.
9. McGrath JM. Not all studies with small samples are pilot studies. J Perinat Neonatal Nurs. 2013;27(4):281–283. doi:10.1097/01.JPN.0000437186.01731.bc.
10. Hertzog MA. Considerations in determining sample size for pilot studies. Res Nurs Health. 2008;31:180–191.
11. Thabane L, Ma J, Chu R, Cheng J, et al A tutorial on pilot studies: the what, why, and how. BMC Med Res Methodol. 2010;10(1):1–10.
12. Bowen DJ, Kreuter M, Spring B, et al How we design feasibility studies. Am J Prev Med. 2009;36(5):452–457.
13. Horner SD. Best practices for improving intervention fidelity that every nurse should know. Pediatr Nurs. 2012;17:171–174.
14. Arnold DM, Burns KEA, Adhikari NKJ, et al The design and interpretation of pilot studies in clinical research in critical care. Crit Care Med. 2009;37(1):S69–S74.
15. Lancaster GA, Dodd S, Williamson PR. Design and analysis of pilot studies: recommendations for good practice. J Eval Clin Pract. 2004;10(2):307–312.
16. Becker PT. Publishing pilot intervention studies. Res Nurs Health. 2008;31(1):1–3.
17. Fain JA. Should we publish pilot feasibility studies? Diabetes Educ. 2010;36(4):521.
© 2018 by The National Association of Neonatal Nurses