The systematic literature review, widely regarded as the gold standard for determining evidence-based practice, is increasingly used to guide policy decisions and the direction of future research. The findings of systematic reviews have greater validity than those of other types of reviews because the systematic methods used seek to minimize bias and increase rigor in identifying and synthesizing the best available evidence on a particular question. It's therefore important that when you search for evidence, you attempt to find all eligible studies and consider them for inclusion in your review.1
One rule of thumb we use when beginning a search for evidence to support a systematic review: if you don't find the evidence, it can't be reviewed! Unfortunately, there is no prescriptive approach to conducting a comprehensive search. But searching is an art that can be cultivated and practiced. It involves several standard processes, such as developing search strings, searching across bibliographic citation databases that index health care research, looking for “gray,” or unpublished, literature, and hand searching.
GETTING STARTED
Developing a search strategy is an iterative process—that is, it involves continual assessment and refinement. As keywords or key terms are used in a search, their usefulness will be determined by the search results. Consequently, searching for evidence is sometimes considered more of an art than a science. It's therefore unlikely that two people, whether they are clinicians or librarians, will develop an identical search strategy or yield identical results from a search on the same review question.
The time required to conduct a search for a systematic review will also vary. It's dependent on the review question, the breadth of the evidence base, and the scope of the proposed search as stated in the review protocol. Narrow searches will often be adequate when investigating a topic requiring a few specific keywords, such as when you're searching only for randomized controlled trials (RCTs) conducted in a single population with a rare disorder. A narrow search will be less resource intensive than a search conducted when the review question is broader or the search relies on general keywords (such as education, prevention, or experience). And while it may seem important conceptually to use a general keyword (such as safety in a search for articles on medical errors, for example), in practice it will add few relevant studies beyond those identified using more specific terms (such as error or harm).
When beginning the search for evidence, you should conduct a few small searches as a test of various search terms and combinations of terms. An ideal search strategy is both sensitive and specific: a sensitive search will recall relevant studies, while a specific search will exclude irrelevant studies. A search that is overly sensitive may capture all the necessary studies but may require a labor-intensive vetting of unnecessary studies at the stage of study selection. A search that is overly specific will yield fewer results but is always subject to the risk that important studies may have been omitted.
Finding help. Given the complexity of the many indexing languages and rules governing the various databases, we recommend that early in the process you make use of an experienced research librarian who can examine your search strategy and help you choose citation databases relevant to your review question. If you can't easily access the services of a research librarian, there are many online tutorials that can help. A Google search—for example, “How do I search using PubMed?”—will reveal sites containing helpful hints and training developed by the U.S. National Library of Medicine (NLM) and librarians from across the globe.
DEVELOPING THE SEARCH STRATEGY
A review protocol with a clearly defined review question and inclusion criteria will provide the foundation for your search strategy. Before embarking on the search, you will need to understand the review question and what information you'll need to address it. For example, it's important to consider the type of data being sought (quantitative, qualitative, economic), the types of studies that report the data (RCTs, cohort studies, ethnographic studies), and the limits or restrictions you'll apply (publication date or language). This will shorten the time required to search and help to ensure that the information retrieved is both relevant and valid.
Once you've determined the review question, you'll need to identify the key terms articulated in the question and the protocol and create a logic grid or concept map. In a logic grid for a review on the effectiveness of an intervention, for example, each column represents a discrete concept that is generally aligned with each element of the PICO mnemonic—Population, Intervention, Comparison intervention, and Outcome measures.
;)
Table 1: Initial Logic Grid Aligned with the PICO Elements of the Review Question
Consider an example using the following review question: “Is animal-assisted therapy more effective than music therapy in managing aggressive behavior in elderly people with dementia?” Within this question are the four PICO concepts: elderly patients with dementia (population), animal-assisted therapy (intervention), music therapy (comparison intervention), and aggressive behavior (outcome measures) (see Table 1 for an example of a logic grid).
Table 2: Logic Grid with Identified Keywords Added
Keywords or free-text words. The first formal step in all searches is to determine any alternative terms or synonyms for the identified concepts in the logic grid. Normally, you'll identify these terms—often referred to as keywords or free-text words—within the literature itself. Perhaps you'll start with a simple search using the terms dementia and animal-assisted therapy or music therapy and aggressive behavior. By looking at the titles and abstracts of the retrieved articles, you can find key terms used in the literature, as well as key concepts that are important to your question. For instance, is the term animal-assisted therapy used synonymously with the term pet therapy? Furthermore, retrieving and reading a few relevant studies of any design—such as an experimental study or a traditional literature review on the topic—will further aid in identifying any commonly used terms.
When developing your search strategy, note that most search platforms (such as Ovid or EBSCOhost) used to access databases (such as MEDLINE) search for the exact terms entered in the database, including any misspellings. This means that to conduct a comprehensive search, you should enter as many relevant key terms as possible. Important articles may be overlooked if all relevant synonyms for a concept aren't included, as some authors may refer to the same concept using a different term (such as heart attack instead of myocardial infarction). Such differences notwithstanding, you may find that including a relevant but broad term may retrieve many irrelevant studies.
Expanding on the logic grid shown in Table 1, Table 2 now contains the keywords chosen from scanning the titles and abstracts of retrieved articles in your initial search. Column one contains terms relating to dementia, the defining feature of the population of interest; columns two and three contain terms relating to animal-assisted therapy and music therapy, the intervention and comparator of interest; and column four contains terms relating to aggressive behavior, the outcome of interest. Placing the terms into a logic grid illustrates how the related concepts or synonyms will combine to construct the final search string.
Index terms or subject headings. Comprehensive search strategies should consist of both keywords or free-text words and index terms, which are used by some major bibliographic databases to describe the content of each published article using a “controlled vocabulary”—that is, a list of standard terms that categorize articles based on their content (such terms will vary from database to database). For example, PubMed uses medical subject heading (MeSH) terms, the controlled vocabulary of MEDLINE.2 MeSH terms are categorized within 16 main “trees” (such as anatomy, organisms, diseases, drugs, and chemicals), each of which branches from the broadest to the most specific terms.
To determine whether index terms exist for the concepts you've identified in your review question, you can search for each term in the MeSH database (selected from the drop-down list on the PubMed home page). For example, by entering dementia, PubMed will identify relevant MeSH terms that include Dementia and Alzheimer Disease. By selecting Dementia, you'll see the term's tree, including the subcategories listed below it, such as Lewy Body Disease.
As was the case when identifying key terms to use in the search strategy, it is also recommended that an initial, simple search using a few key concepts (dementia AND animal-assisted therapy or dementia AND music therapy AND aggressive behavior) be performed in PubMed to identify index terms. The aim is to retrieve a few relevant articles to see how they were indexed using the controlled vocabulary. Once the results are displayed, you can scroll through the citations and click on the title of any eligible article to view its details. From here, follow the link to the article's MeSH terms and examine which ones were used to describe the article's content. Repeat this process with a number of different articles to determine whether similar indexing terms have been used.
Table 3: Logic Grid with Keywords and Index Terms or Subject Headings
The terms in the logic grid can now be updated with the MeSH terms you have chosen from those listed with each retrieved article (see Table 3). The [mh] that appears next to these terms in the grid is the search-field descriptor that stands for “MeSH headings.” It's worth noting that “Entry Terms” under each search term's MeSH listing (if one is available) can also be examined for suggestions of alternative terms that can be searched in titles and abstracts.
Because new articles in PubMed are not indexed immediately, and because indexing is a manual, subjective process susceptible to human variation, it's important to also search for the key terms in the titles and abstracts of articles—in other words, for free-text or keywords—to capture any articles that could be missed by using index terms (such as MeSH headings) alone. For example, if we did not search for free-text words and did not include the index term Bonding, Human Pet (a MeSH term), we might miss an important article that wasn't indexed under the MeSH term Animal-Assisted Therapy.
Table 4: Logic Grid with Keywords and Index Terms Qualified with Field Codes and Wildcard Characters
By adding the search-field descriptor [tiab] (meaning “title/abstract”) to a search term, you can direct PubMed to search the title and abstract field code for these terms. A number of other search-field descriptors can be used as well, such as [au] for “author” and [pt] for “publication type.”2 Using a search-field descriptor such as [tw] (“text word”) is often preferred over [tiab] for systematic reviews because the former searches in the title and abstract of articles as well as across a greater number of fields and will return a greater number of results for the same search query. Shortcuts or “wildcard” characters can also be used to account for different terminology or spelling. For example, PubMed allows truncation searching, in which an asterisk can substitute for any word's beginning or ending (for instance, a search for therap* will retrieve articles with the words therapy and therapeutic). Search-field descriptors and wildcard characters should be applied to any newly identified keywords and index terms in the logic grid (see Table 4).
Once all search terms, including both free-text words and indexing terms, have been collected and finalized, a second search can then be undertaken across all selected citation databases. Initially, the key terms and synonyms within each column in the logic grid are combined using “OR.” (Most databases use some form of Boolean logic—search terms connected by the Boolean operators “OR” and “AND,” among others.) This will direct the database to find articles containing any of the search terms within the indicated fields. To do this in PubMed, select the “Advanced” search box and clear the search history. Copy and paste the first set of terms into PubMed and run the search.
For example, an initial search for articles related to different types of dementia might look like this:
“Dementia [tw] OR Alzheimer [tw] OR Huntington* [tw] OR Kluver [tw] OR Lewy [tw] OR Dementia [mh] OR Alzheimer disease [mh]"
This search could yield more than 100,000 citations. Following this, clear the search box and repeat the process with search terms from the second column in Table 4. It is easier to search each column of the logic grid individually—particularly if each column contains an extensive list of search terms—rather than combining all the search sets in one go. Furthermore, by running each search successively you can determine if a component of the search string is producing many irrelevant results and easily adjust the search strategy. In our example, if you add the term aggress* [tw] to capture aggressive and aggression in the title or abstract, you will get an overwhelming number of irrelevant results because these terms are also used to describe the spread of certain cancers.
Once you complete the searches aligned to each concept, click on the “Advanced” option again. This allows for display of the “search history” and for a ready combination of the individual searches using the Boolean operators “AND” and “OR.” Using this method, parentheses are automatically placed around each set of terms to maintain the logical structure of the search. For example, the search for articles on animal-assisted therapy versus music therapy to treat aggression in patients with dementia might look like this:
“(Dementia [tw] OR Alzheimer [tw] OR Huntington* [tw] OR Kluver [tw] OR Lewy [tw] OR Dementia [mh] OR Alzheimer disease [mh]) AND (Animal assisted therapy [tw] OR Animal assisted activit* [tiab] OR Animal assisted intervention* [tiab] OR Animal therapy [tw] OR Pet therapy [tw] OR Dog therapy [tw] OR Dog assisted therapy [tw] OR Canine assisted therapy [tw] OR Aquarium [tiab] OR Animal Assisted Therapy [mh] OR Pets [mh] OR Dogs [mh] OR Cats [mh] OR Birds [mh] OR Bonding, Human-Pet [mh] OR Animals, Domestic [mh]) OR (Music* [tw] OR Music therapy [tw] OR Singing [tw] OR Sing [tw] OR Auditory stimulat* [tw] OR Music [mh] OR Music Therapy [mh] OR Acoustic Stimulation [mh] OR Singing [mh]) AND (Aggression [tw] OR Neuropsychiatric [tiab] OR Apathy inventory [tiab] OR Cornell scale [tiab] OR Cohen Mansfield [tiab] OR BEHAVE-AD [tiab] OR CERAD-BRSD [tiab] OR Behavior* [tiab] OR Behaviour* [tiab] OR Aggression [mh] OR Personality inventory [mh] OR Psychomotor agitation [mh])"
Once the final search has been conducted, you can further refine search results by publication date, study groups, language, or any other limits appropriate to the review topic by selecting the relevant filter (left-hand side of the screen in PubMed) from the range available. PubMed also provides predefined search filters that restrict search results to specific clinical study categories or subject matters (such as clinical queries). You will have determined the date range for the search at the protocol development stage. Given that your aim is to summarize the evidence surrounding a particular question, you should justify any limits to the publication date of included studies in the background section of the protocol. The chosen time frame will vary depending on the review question. For example, reviewers may impose a start date for a search that coincides with the introduction of a new intervention and the advent of the preceding clinical research on it.
The structure of the search strategy will remain the same regardless of the search platform used to search a database. But since most major databases use a unique controlled vocabulary to index their articles, the indexing terms will need to be adapted to each database; in most cases the key terms remain the same across different databases. These differences in indexing terms are the main reason it is not recommended to search bibliographic citation databases for a systematic review using a federated search engine or platform—that is, one that searches multiple databases and sources at once.
You should also be aware that the platforms used to search citation databases often use different wildcard characters or commands. For this reason, beginning searchers should use the online tutorials and help pages of the various platforms and databases. For example, while Ovid's search platform can also be used to search the MEDLINE database, the terms used for truncation searching are quite different: an asterisk (*) is used for unlimited truncation within PubMed and a dollar symbol ($) in Ovid. Moreover, in Ovid the question mark (?) wildcard can be used within or at the end of a word to substitute for one character or no characters (behavio?r will retrieve articles with the words behaviour and behavior); the number sign (#) wildcard can substitute for a single character (wom#n will retrieve articles with both woman and women). The use of wildcards for substitution of characters is not supported in PubMed.
Because searching is an iterative process, you won't want to predetermine when it will end. Consequently, it is important to look at the results of the search continually as you develop the search strategy to determine whether the results are relevant. One way to do this is to check if already identified relevant articles are being captured by the search. If not, the search strategy will need to be modified accordingly.
Once the search is complete, the results can be exported to bibliographic management software such as EndNote or Reference Manager. These tools are useful for organizing the search results, removing duplicate citations, and selecting studies (the next step of the systematic review process, to be discussed in the next article in this series).
WHERE TO SEARCH?
Developing the search strategy and search filters for use within each database is an important and time-consuming part of the search process, often more so than the search itself! Another important consideration is where to search. A search for a systematic review should be comprehensive and attempt to identify all of the available evidence. This can be an enormous undertaking.
Generally, a systematic review to inform health care practice and policy should search the major medical databases including MEDLINE from the NLM in North America and searchable through PubMed, and Embase, a product of Elsevier that indexes many European biomedical journals; the controlled vocabulary for Embase is searchable through Emtree, which also contains all MeSH terms (www.elsevier.com/online-tools/embase/emtree). Nurses undertaking systematic reviews will find that much literature relevant to nursing practice is also available in the Cumulative Index to Nursing and Allied Health Literature (CINAHL) database by EBSCO. Beyond these, there are many others: Web of Science, PsycINFO, Scopus, JSTOR, Academic Search Premier, Academic Onefile, the Cochrane Nursing Care Field trials register, and the list goes on.
You should establish which databases index articles relevant to the topic at hand. Some databases have a specific topic focus, such as PsycINFO, which should be searched for a question related to mental health. The JBI Database of Systematic Reviews and Implementation Reports is, as the name suggests, a repository for systematic reviews and would be unnecessary for most review searches (systematic reviews rarely include other systematic reviews among their inclusion criteria). Similarly, a quick Google search (“What information is in… ?”) to establish the content and coverage of other databases is worthwhile and will help in identifying unnecessary overlap in the search strategy.
Hand searching. You may also wish to consider more traditional means of locating evidence. Screening the reference lists of studies already selected for inclusion in the review is often a valuable means of identifying other pertinent studies. Similarly, hand searching specific journals is often used by systematic review authors to locate studies. Journals selected for hand searching should be identified as relevant from database or preliminary searching; the likelihood is that these journals may contain relevant studies. Because hand searching can be an onerous task, it's recommended that no more than two or three relevant journals should be hand searched for a review.
Finding experts is another method of locating evidence. While contacting authors to clarify details of studies and to request data are relatively common pursuits for the systematic reviewer during the appraisal and extraction processes, doing so to identify relevant studies can also be useful. Such experts can often provide papers that even a comprehensive search may have failed to identify.
SHADES OF GRAY
Systematic reviews that purport to have conducted a comprehensive search should have made some attempt to search for gray literature. The International Conference on Grey Literature in Luxembourg defined it in 1997 (and expanded on it in 2004) as literature “produced at all levels of government, academic, business and industry in electronic and print formats not controlled by commercial publishing.”3 However, this definition is often broadened to include any study or paper that has not been formally published or peer reviewed. Gray literature often appears in the form of government or institution reports and newsletters and even in blogs, conference proceedings, census reports, or nonindependent research papers. As a result, these reports or manuscripts are often not as widely available and are generally more difficult to locate.
Nonetheless, the inclusion of gray literature in systematic reviews has emerged as an important adjunct to commercially published research, as it often reflects a source of timely or alternative information that can help to minimize publication bias and provide a more accurate and thorough account of the evidence.4, 5
There are three common ways to search for gray literature. The first involves searching or browsing the Web sites of organizations relevant to the review question (such as the World Health Organization or the National Institute for Health and Care Excellence). The second involves searching databases that collate and index gray literature. Although gray literature is rarely indexed, two commonly used sources are OpenGrey (www.opengrey.eu), an open access database to gray literature from Europe, and the Grey Literature Report (www.greylit.org), a bimonthly report from the New York Academy of Medicine. Reviewers will find that such databases do not have an extensive or advanced search capability, and therefore searching them is often limited to the use of a few critical keywords. Furthermore, they lack indexing or subject headings; without this feature a search can be quite time consuming. The third approach is to use online search engines. Search engines such as Google do not use a controlled vocabulary and so performing a simple search of a few select keywords is best. Such sites will yield a large number of results. To make results more manageable, you can try limiting the search to terms that appear on the title page of an article only6 or by using keywords that limit the results to specific documents (such as guidelines). Searches can also be limited by language or sources (for example, adding site:gov to a Google search will limit results to government Web sites). An example of a tool that can also help is the federated search engine MedNar (http://mednar.com/mednar) that searches across a range of government and organizational sites, as well as commercial databases.
Other sources of gray literature can be found in numerous guides developed to assist researchers. For example, the Canadian Agency for Drugs and Technologies in Health's Grey Matters provides an extensive list of gray literature sources that can be searched.7 Developed with the systematic reviewer in mind, the tool kit provides a checklist that aids users in documenting the search process and in ensuring it has been conducted in a standardized way.
REPORTING THE SEARCH STRATEGY
The final consideration is reporting the details of the search strategy, including the filters (such as language, date limits) and databases and other sources used. A hallmark of a systematic review is its reproducibility; another researcher should be able to review the same question and arrive at similar conclusions. Without a transparent reporting of the search strategy—one that allows readers to assess the quality of the search and its sources, and in turn, make a judgment on the likely credibility of the review8, 9—this would not be possible.
Most journals that publish systematic reviews now espouse the PRISMA guidelines (Preferred Reporting Items for Systematic Reviews and Meta-Analyses; online at www.prisma-statement.org), which dictate that the full search strategy for at least one major database should be reported in an appendix and published along with the review.10 Online repositories of systematic reviews, such as the JBI Database of Systematic Reviews and Implementation Reports and the Cochrane Database of Systematic Reviews, allow for publication of all the search filters and strategies across the databases and sites used. A systematic reviewer will appreciate that reporting only the search filters used is inadequate. The methods section of a review should list all of the bibliographic citation databases searched, ideally with the platform used to search them, as well as the dates they were searched and any limits used. The results of the search should be adequately reported, as well; this is often quite simple to convey in a flow diagram, which is also detailed in the PRISMA guidelines.10
Once the search is complete and the results from each source have been exported, the next step, study selection, can begin. This is where titles, abstracts, and sometimes the full text of studies found are screened against the inclusion and exclusion criteria. This step of the process will be the focus of the next article in this series.
