Journal Logo

Systematic Reviews, Step by Step

Developing the Review Question and Inclusion Criteria

Stern, Cindy PhD; Jordan, Zoe PhD; McArthur, Alexa MPHC, MClinSc

Author Information
AJN, American Journal of Nursing: April 2014 - Volume 114 - Issue 4 - p 53-56
doi: 10.1097/01.NAJ.0000445689.67800.86

What constitutes appropriate “evidence” for evidence-based practice? This question has been the subject of considerable discussion for many years. It's also of critical importance when conducting a systematic review. The first article in this series on systematic reviews from the Joanna Briggs Institute (JBI), published last month, presented an overview and definitions of the systematic review. Briefly, a systematic review is research undertaken to identify, evaluate, and synthesize the results of individual studies on a particular topic, making reliable data available in a usable form.1 Alan Pearson and colleagues at the JBI have written that when making decisions “clinicians (often quite subconsciously) are frequently trying to select an appropriate activity or intervention and to assess the degree to which the decision will meet the four practice interests of health professionals”—namely, whether it's feasible, appropriate, meaningful, and effective (FAME).2

But the evidence-based practice movement has focused largely on just one of these interests, effectiveness. Pearson and colleagues have argued for a pluralistic approach when considering what counts as evidence in health care; they write that not all questions can be answered from studies measuring effectiveness alone.3 To meet the wide array of problems health care professionals encounter, a wide range of research methodologies and a broad definition of evidence are warranted.

Constructing a well-built clinical question for a systematic review is a skill that can be learned. Reviewers can ask themselves a number of questions, such as: Is the information we seek analytical? Is it focused on a particular therapy? If so, will we examine its preventive effects in terms of quality of life? Or, conversely, will we look at its economic viability? Will the outcomes measured be meaningful enough to justify the high costs of conducting the review? Remember, the question puts the review process in motion and forms the basis for the inclusion and exclusion criteria. It therefore merits careful consideration.


A clear question will not only guide researchers in conducting a review, it will also help readers to discern whether or not they should read it. The question also facilitates indexing in online databases such as PubMed or the Cumulative Index to Nursing and Allied Health Literature (CINAHL) and will show a clear relationship to the inclusion criteria.

If we compare the two following questions, we can see that the first is clearer in its intentions and contains more information for both reviewer and reader than the second.

  • What are the effects of turning long-term care residents every two hours compared with every four hours in preventing pressure ulcers?
  • What is the best way to prevent pressure ulcers?

Determining the question is one of the first steps in planning a systematic review because it largely establishes the conduct of the review; for example, inclusion criteria are developed as a result of the question. A good question should incorporate the four elements included in the PICO mnemonic:

  • Population
  • Intervention
  • Comparison intervention
  • Outcome measures

A variety of mnemonics exists to help reviewers structure the review question. PICO is most frequently used in quantitative reviews (those incorporating research based on traditional scientific methods that generate numerical data).4 Its variants PICOS and PICOT, where S stands for study designs (indicating which study designs, such as randomized controlled trial [RCT] or diagnostic study, are eligible to answer your question) and T stands for time frame (a period over which outcomes are assessed, such as 24 hours after surgery), can also be used. Such mnemonics aid in the clarification of the structure of the review and its question. At the JBI4 and the Cochrane Collaboration,5 PICO is the preferred choice for question development. Furthermore, PICO may be used in the systematic review process to guide concept mapping when designing the search strategy. (Concept mapping is used to help identify relevant keywords and search terms for your review.)

PICo (with a lowercase o)can be equally useful for qualitative reviews (those seeking to analyze human experience and social phenomena).4 With qualitative evidence there is no outcome or comparator to be considered. The core elements of PICo are:

  • Population
  • phenomenon of Interest
  • Context

The phenomenon of interest differs from an intervention in its focus. Quantitative reviews are concerned with an intervention and seek to isolate it from the happenings and influences of study participants. Reviews containing qualitative evidence focus on the engagement between the participant and the intervention. A qualitative review may describe an intervention, but its question focuses on the perspective of the individuals experiencing it as part of a larger phenomenon.

Other mnemonics useful for qualitative reviews include SPICE (Setting, Perspective, Intervention, Comparison, Evaluation) and SPIDER (Sample, Phenomenon of Interest, Design, Evaluation, Research type).6

Perhaps you want to ask both a quantitative and qualitative question on the same topic. For example, if you were interested in the effectiveness of compression stockings in preventing deep vein thrombosis (DVT), you might want to find studies that compare the stockings to placebo (a quantitative review), as well as explore the experiences of those who use them (a qualitative review). Although compression stockings may be effective in preventing DVT, they may be uncomfortable to use, and compliance rates may be low. Reviews that incorporate more than one type of data are called “comprehensive” or “mixed methods” systematic reviews.

Let's look at both quantitative and qualitative review questions in more detail.


A solid objective will inform the identification and subsequent inclusion of studies, the data extraction, and the data synthesis. The objective can also help readers in a preliminary assessment of the review's relevance to them. Quantitative reviews conducted by JBI researchers will specify the population, the intervention, and the outcomes of interest.

When developing the review question, reviewers should consider how general the review will be with regard to the characteristics of the population (for example, “nurses” versus “female RNs with a minimum of five years’ experience”), the type of intervention (such as any drug therapy used for depression of any dosage for any duration), and the outcomes of interest (such as adverse effects or depression measured by any validated scale at any time). These details can then be added when completing the inclusion criteria. The JBI's reviewers’ manual and the Cochrane Collaboration's reviewer's handbook recommend that the following features be considered when developing a question for a quantitative review4, 5:

  • the most significant features of the population under investigation (such as age or illness)
  • the experimental and control interventions
  • any variations in the intervention (such as administration method or dosage) and whether studies involving such variations will be included
  • whether RCTs addressing only part of the intervention or combined with another intervention will be included

Regarding outcomes, the Cochrane Collaboration makes the following suggestions5:

“Outcomes may include survival (mortality), clinical events (e.g. strokes or myocardial infarction), patient-reported outcomes (e.g. symptoms, quality of life), adverse events, burdens (e.g. demands on caregivers, frequency of tests, restrictions on lifestyle) and economic outcomes (e.g. cost and resource use). It is critical that outcomes used to assess adverse effects as well as outcomes used to assess beneficial effects are among those addressed by a review…. If combinations of outcomes will be considered, these need to be specified. For example, if a study fails to make a distinction between non-fatal and fatal strokes, will these data be included in a meta-analysis if the question specifically relates to stroke death?”

“Review authors should consider how outcomes may be measured, both in terms of the type of scale likely to be used and the timing of measurement. Outcomes may be measured objectively (e.g. blood pressure, number of strokes) or subjectively as rated by a clinician, patient, or carer (e.g. disability scales). It may be important to specify whether measurement scales have been published or validated. When defining the timing of outcome measurement, authors may consider whether all time frames or only selected time-points will be included in the review.”

An example of a quantitative review question is this from the JBI database: “What is the effect of an individualized survivorship care plan as compared to usual care on quality of life on the adult female breast cancer survivor?”7 The review question clearly satisfies all four of the PICO elements: population (adult female breast cancer survivors), intervention (individualized survivorship care plan), comparison intervention (usual care), and outcome measure (quality of life).


Qualitative reviews seek “to understand the meaning of phenomena and their relationships”8 and use the PICo mnemonic. Specifications on the population (either for inclusion or exclusion criteria) must be delineated. Although the term population is also used in qualitative reviews, its use doesn't imply that all of the features relevant to quantitative reviews such as sampling methods or homogeneity (which refers to similarity among included studies’ results) are appropriate here. Rather, population characteristics in a qualitative review relate to peoples’ subjective experience or the meaning that a disease or an intervention holds for them.

A phenomenon of interest is the experience, event, or process under study. Examples might include patients’ responses to pain or how they cope with breast cancer. The level of detail ascribed to the phenomenon will differ depending on the nature or intricacy of the subject. A question on the experience of older adults exercising may be rather straightforward, for example, whereas a question on the experiences of women who were sexually abused as children may lend itself to a more complex kind of detail. Regardless, the question may be clarified, expanded, or revised as the protocol develops.

In reviews containing qualitative evidence, context will also vary; it will depend on the review's objective and questions. When determining context, reviewers may consider factors such as geographic location, interests based on race or gender, and clinical setting (such as long-term care). Remember that in qualitative reviews there is no need to list outcomes; the focus is on the experiences of the participants.

An example of a qualitative review question from the JBI database is: “What is the experience of the adult neutropenic patient with cancer being nursed in the isolation room?”9 The review question identifies the population (adult neutropenic patients with cancer), the phenomenon of interest (the patients’ experiences while being cared for by nurses), and the context (in isolation).


A good review question lays the foundation for the development of a robust protocol—that is, where you flesh out the elements of PICO or PICo in making a plan that ensures scientific rigor and minimizes bias. Regardless of whether the review involves quantitative or qualitative research (or both), criteria exist that must be addressed in the protocol (such as inclusion criteria and methods).

Inclusion criteria determine which research articles will be selected. In order for the reader to understand the focus of the review (and its limitations), the reviewers need to be precise in outlining the inclusion criteria. The following aspects should be addressed2:

  • the types of studies to be included (such as cohort or ethnographic studies)
  • the intervention, activity, or phenomenon under investigation (such as drug therapy for smoking cessation or the experience of smokers undertaking hypnotherapy)
  • the outcome (for quantitative questions; for example, the effectiveness of drug therapy for smoking cessation)
  • the population (such as females ages 16 years or older who have smoked for at least three years)
  • publication language (such as English only or English, simplified Chinese, and Japanese)
  • the time period (such as studies published between 1999 and 2013)

The clarity of the inclusion criteria also ensures the replicability of the review.

Methods. It is important to clarify the methods you will use to search the literature, appraise the studies retrieved, and extract and synthesize the data. (These steps will be discussed in later articles in this series.)

Conclusion. While health care workers frequently want to answer very general questions, it is often easier to conduct a systematic review on a narrow, more focused question. In doing so, the final product is also more likely to present useful results that can be applied when making clinical decisions. If a reviewer is interested in a broad topic such as managing heart disease, which covers several factors (pharmacologic or surgical treatment and lifestyle modifications, for example), it is better to establish a series of questions related to that topic and conduct a series of reviews than try to cover all of them in a single review.


1. Korhonen A, et al. Meta-synthesis and evidence-based health care—a method for systematic review. Scand J Caring Sci 2013;27(4):1027–34
2. Pearson A, et al. Evidence-based clinical practice in nursing and health care: assimilating research, experience, and expertise. Oxford, UK; Malden, MA: Blackwell Publishing; 2007..
3. Pearson A, et al. A re-consideration of what constitutes “evidence” in the healthcare professions. Nurs Sci Q . 2007;20(1):85–8
4. Joanna Briggs Institute. Joanna Briggs Institute reviewers’ manual: 2011 edition Adelaide, South Australia: University of Adelaide; 2011.
5. Higgins JPT, Green S, eds. Cochrane handbook for systematic reviews of interventions version 5.1.0. [updated Mar 2011] Chichester, West Sussex: Cochrane Collaboration; Wiley-Blackwell; 2011. Cochrane book series.
6. Cooke A, et al. Beyond PICO: the SPIDER tool for qualitative evidence synthesis. Qual Health Res. 2012;22(10):1435–43
7. Martin TA, et al. Effectiveness of individualized survivorship care plans on quality of life of adult female breast cancer survivors: a systematic review. JBI Database of Systematic Reviews and Implementation Reports . 2013;11(9):258–309
8. Gough D, et al. An introduction to systematic reviews. Thousand Oaks, CA: Sage Publications; 2010..
9. Lee YM, et al. The experience of being a neutropenic cancer patient in an acute care isolation room: a systematic review of qualitative evidence. JBI Library of Systematic Reviews . 2011;9(12):400–16

clinical question; inclusion criteria; qualitative review; quantitative review; review question; systematic review

© 2014 Lippincott Williams & Wilkins. All rights reserved.