Systematic reviews are fundamental to evidence-based healthcare and provide the highest level of evidence to inform decision-making. The role played by systematic reviews in healthcare has evolved in part out of recognition of the overwhelming amount of research evidence available. Systematic reviews offer a ready and rigorous solution for decision makers to navigate their way to the best available evidence to inform their decision-making. The number of systematic reviews published to accommodate the demands of evidence informed decision-making has increased markedly over the past two decades. One recent estimate suggests that 11 systematic reviews are published every day.1
With the increased number of systematic reviews available, a logical and appropriate next step has been the conduct of reviews of existing systematic reviews, allowing the findings of separate reviews to be compared and contrasted, thereby providing decision makers in healthcare with the evidence they need. Reviews of systematic reviews are referred to by several different names in the scientific literature, including umbrella reviews, overviews of reviews, reviews of reviews, a summary of systematic reviews and also a synthesis of reviews. Irrespective of their name, all of these types of reviews have a defining feature in common: a systematic review is the principal and often sole ‘study type’ that is considered for inclusion.2–4 In this study, the review of existing systematic reviews and meta-analyses will be referred to as an umbrella review. The term research synthesis is also used to encompass study types, systematic reviews and meta-analyses, throughout this study.
The principle reason for the conduct of an umbrella review is to summarize the evidence from multiple research syntheses.3 Conduct of an umbrella review may also offer a means for a rapid review of the evidence to address a broad and high-quality evidence base in relation to a topic.5 Umbrella reviews are conducted to provide an overall examination of the body of information that is available for a given topic, and to compare and contrast the results of published systematic reviews.2 The wide picture obtainable from the conduct of an umbrella review is ideal to highlight whether the evidence base around a topic is consistent or contradictory, and to explore the reasons for the findings. Furthermore, an umbrella review allows ready assessment of whether review authors addressing similar review questions independently observe similar results and arrive at generally similar conclusions.
The aim of an umbrella review is not to repeat the searches, assessment of study eligibility, assessment of risk of bias or meta-analyses from the included reviews, but rather to provide an overall picture of findings for particular questions or phenomenon. For example, compared with a systematic review or meta-analysis limited to one treatment comparison, an umbrella review can provide a broader picture of many treatments. This is more useful to inform guidelines and clinical practice when all of the management options need to be considered.6
The objective of the umbrella review will determine whether the review will include analyses of different interventions for the same condition or investigation of the same intervention and condition, but where different syntheses address and report on different outcomes. Furthermore, a reviewer may wish to summarize more than one research synthesis for different conditions or populations.3 The growth in the number of systematic reviews has been in part fueled by the development of review methodologies to answer questions of ‘how’ and ‘why’ interventions do or do not work, and how recipients of the intervention may experience them rather than being focused on the effectiveness of interventions.7 As a result, many syntheses published today are of original qualitative research.7 Similarly, we may find that umbrella reviews inevitably ask questions that direct the reviewer predominantly to existing qualitative reviews.
In 2013, a methodology working group was formed by the Joanna Briggs Institute to evaluate umbrella reviews and develop guidance for the conduct of an umbrella review. The aim of this study is to describe the development of guidance for the conduct and reporting of an umbrella review including diverse types of evidence. The guidance presented here also appears in the Joanna Briggs Institute Reviewer's Manual.8
The working group was composed of six participants including systematic review authors, researchers, clinicians and journal editors. Participants were from Australia, the USA, Canada and Thailand. The working group met regularly via teleconference and also via e-mail correspondence from May 2013 to November 2013, to discuss, define and develop methods for conduct and reporting of an umbrella review. A 3-day meeting of the working group was held in Adelaide, Australia, in October 2013. Consensus amongst members of the group regarding the methodology was reached through continual discussion and testing elements of the methods proposed. Throughout the development period, details of the methodology were pre-circulated and presented to the International Scientific Committee of the Joanna Briggs Institute for further consideration and discussion every 6 weeks. In October 2013, the methodology was presented in a workshop during the Joanna Briggs Institute Convention to over 30 participants. International participants of the workshop included systematic review authors, methodologists, researchers, clinicians and consumers. The workshop allowed participants the opportunity to provide testing, critique and feedback on the proposed methodology. In addition, the methodology was presented to the Joanna Briggs Institute Committee of Directors, comprising over 90 international experts in research synthesis from over 20 countries for further discussion and feedback. A cyclic process of feedback and review was used at all stages of the development process. Following this, the methodology was ratified at a meeting of the International Scientific Committee of the Joanna Briggs Institute in December 2013.
Umbrella review methodology and summary of evidence from research syntheses
The umbrella review should be preceded by an a priori, peer-reviewed protocol, and should include a clearly articulated question(s), detailed inclusion criteria, a structured search process to locate and select relevant existing reviews, method for critical appraisal of the included reviews and a formal process of data extraction followed by means to summarize and present the data. It is important to note that the principal aim of an umbrella review is to provide a summary of existing research syntheses related to a given topic or question, not to re-synthesize, for example, with meta-analysis or meta-synthesis, the results of existing reviews or syntheses.
Objectives and inclusion criteria
The umbrella review objective(s) and specific umbrella review question(s) should be clearly stated. The stated objectives will commonly be broad and should logically inform the specific umbrella review question(s).9
The inclusion criteria indicate the basis on which available reviews will be considered for inclusion into the umbrella review.9 These criteria provide a guide for the reader to clearly understand what is proposed by the umbrella review authors and, more importantly, a guide for the reviewers themselves to base decisions about the reviews to be selected during the study selection phase of the umbrella review. One of the unique features of an umbrella review, when considering other types of systematic reviews, is that the analytical units, or studies that will be included, are exclusively systematic reviews and meta-analyses, and not primary or original research. Research syntheses included in an umbrella review should represent syntheses of empirical research evidence. Due to the range of review types available in the literature,10 authors of umbrella reviews will have to stipulate clearly which review types will be included a priori in the protocol. Reviews that incorporate theoretical studies or published opinion as their primary source of evidence should not be included in an umbrella review and should be listed as an explicit exclusion criterion in the a priori protocol.
Presentation of the inclusion criteria for an umbrella review assessing the effectiveness of an intervention(s) should stipulate a Population, Intervention, Comparator, Outcome (PICO) element.9 Similarly, an umbrella review that addresses a question that would lend itself to inclusion of qualitative systematic reviews would include a Population, Phenomena of interest and Context (PICo) element.9 Umbrella reviews that include both quantitative and qualitative evidence should stipulate both PICO and PICo elements. All of the inclusion criteria presented should accurately reflect and be clearly congruent with the review objective and question(s).
Types of participants
Important characteristics of participants should be detailed, including age and other qualifying criteria. Umbrella reviews that encompass multiple participant or demographic groups should define each group clearly.
Interventions/phenomena of interest
Interventions may be focused, for example, only considering pharmacological management of a patient group, or broad, including both pharmacological and other interventions (e.g. diet, exercise, surgery). Definitions should be used, when appropriate, to provide clarity. Umbrella reviews that address multiple interventions and treatments should define each potential intervention of interest clearly.
The context should be clearly defined and may include, but is not limited to, consideration of geographic location or other cultural factors or racial or sex-based characteristics. In some cases, context may also encompass detail about the specific healthcare setting (such as acute care, primary healthcare or the community).
Outcomes of interest should be predefined in umbrella reviews that lend themselves to quantitative evidence. Outcomes should be relevant to the question posed by the umbrella review. Outcome measures should be included in the description [e.g. measurement of quality of life using the 36-item Short Form Health Survey]. Surrogate outcomes should be explained and presented when there is a clear association with patient relevant outcomes. To provide a balanced overview of the evidence base related to a particular topic and fully inform decision-making, an umbrella review should attempt to report both beneficial and adverse outcomes.
Types of studies
While it is clear that an umbrella review will include only existing systematic reviews and meta-analyses, there should be a match in this section between the methodology of the research synthesis to be considered for inclusion in the umbrella review and its primary objective. For example, an umbrella review assessing the effectiveness of a range of interventions for aggressive behaviors in elderly dementia patients may limit itself to including systematic reviews that assessed effectiveness by including only randomized controlled trials (RCTs) and other experimental study designs.
The search strategy for an umbrella review should aim to identify all research syntheses relevant to the review question.11 The search strategy should be comprehensively reported, and the detailed search filters employed should be presented sequentially in the single appendix for all the databases that were searched listed along with the search dates. As well as biomedical citation databases such as Medline or PubMed, Embase and CINAHL, other sources to search include the major repositories of systematic reviews such as the JBI Database of Systematic Reviews and Implementation Reports, the Cochrane Database of Systematic Reviews, Database of Abstracts of Reviews of Effects and the PROSPERO register. Predefined search filters designed to identify review articles for various databases already exist and they are worthwhile investigating when developing the search strategy; an example is the ‘systematic[sb]’ search filter for PubMed. As many databases do not have a predefined search filter for review articles, in these cases, it is preferable to search with key terms such as ‘systematic’ or ‘meta-analysis’ in the title or abstract fields. Most authors will use these terms in the title of their publications to clearly identify the type of publication which facilitates the searching process. A comprehensive search for an umbrella review should also encompass a search for grey literature, reports from government and non-government organizations, and reports that are not commercially published.11 The search for systematic reviews and meta-analyses rarely needs to extend prior to 1990 as there were very few research syntheses published prior to that time.4 Including research syntheses conducted within the past 5–10 years will reflect original/primary research conducted over 30 years prior that has been included in the located research syntheses.
Presentation of the results of the umbrella review
An introductory section to the results of the umbrella review should begin with a narrative description of the umbrella review process accompanied by a flowchart as stipulated in the Preferred Reporting Items for Systematic Reviews and Meta-analyses guidelines.12 The introductory section of the results should include an overall description of the included reviews with clear reference to the detailed ‘Table of Included Review Characteristics’ (see below under the ‘Findings of the umbrella review’ section for recommended review details to be presented). This section should provide context to the results and sufficient descriptive detail for the reader to support the inclusion of the research syntheses into the umbrella review, the relevance of included research syntheses to the umbrella review question and the evidence base they offer to the question. Specific items or points of interest from individual reviews may also be highlighted here.
Systematic reviews and meta-analyses that are eligible for inclusion in an umbrella review must be assessed for methodological quality (critical appraisal). There are checklists and tools available to assess research syntheses, including AMSTAR13 and, most recently, ROBIS.14 A checklist has been developed by the umbrella review methodology working group and is presented in Table 1. Each of the questions posed in the checklist can be scored as being ‘met’, ‘not met’, ‘unclear’ or ‘not applicable’. The checklist was piloted by members of the methodology working group and by workshop participants. The tool to assist in the assessment of methodological quality should be used by two independent reviewers conducting the appraisal independently. The decision as to whether or not to include a review can be made based on meeting a pre-determined proportion of all criteria, or on certain criteria being met. It is also possible to weight certain criteria differently. Decisions about a scoring system or any cut-off for exclusion should be made in advance and agreed upon by all reviewers before critical appraisal commences, and detailed in the umbrella review protocol.
The presentation of the results of quality appraisal should include a narrative summary of the overall methodological quality of the included reviews. In an umbrella review, it is also important to present to the reader with clear indication of the quality of the included primary/original research studies in each of the research syntheses that have been included in the umbrella review. This will have an impact on the interpretation and implications for practice and research, and must be noted with clarity to the reader of the review in the body of the umbrella review report.
Findings of the umbrella review
To minimize risk of bias in the umbrella review process, a standardized data extraction tool should be employed by two independent reviewers to extract data from each included review. Guided by the data extraction tool, information extracted from each included review should include the following:
- citation details
- objectives of the included review
- type of review
- participant details
- setting and context
- number of databases sourced and searched
- date range of database searching
- publication date range of studies included in the review that inform each outcome of interest
- number of studies, types of studies and country of origin of studies included in each review
- instrument used to appraise the primary studies and the rating of their quality
- outcomes reported that are relevant to the umbrella review question
- method of synthesis/analysis employed to synthesize the evidence and
- comments or notes the umbrella review authors may have regarding any included study
The majority of this information will appear in the ‘Table of Included Review Characteristics’ that appears in the umbrella review report.
Extraction and presentation of findings and results for an umbrella review should be limited to those presented by the included systematic reviews and meta-analyses; primary research study level data should not be reported in an umbrella review (except when an outcome is only informed by one included study). Details of the issue of interest to the umbrella review; for example, the range of interventions, phenomena of interest, population details or outcome differences, should be extracted in detail. Clear indication of overlap of original/primary research studies in each of the included studies should be presented in the umbrella review. For example, one study may be included in multiple syntheses, and to avoid double counting, this should be indicated clearly to the reader.
The presentation of the findings and results should align to the umbrella review question. When overall effect estimates extracted from systematic reviews or other similar numerical data are presented, tabular presentation of findings is recommended (see Table 2). Alongside the presentation of quantitative data, the number of reviews that inform the outcome, number of participants (from included reviews) and statistical heterogeneity should also be reported.4 An example of the table of findings for an umbrella review titled ‘ Effectiveness of non-pharmacological strategies to manage aggressive behavior in patients with dementia ‘ is presented below in Table 2 for one outcome; in this example, the outcome is ‘aggressive behaviors’, if other outcomes were included, the final three columns of the table would be repeated for each. Tabular presentation should be accompanied by a clear and detailed description of the interventions addressed.
When the evidence from qualitative systematic reviews is included in the umbrella review, the final or overall synthesized findings from each review should be presented in tabular format (see Table 3). To facilitate interpretability, qualitative synthesized findings should be presented with enough relevant contextual information alongside each synthesized finding. A description of the phenomenon of interest should also be presented alongside the synthesized findings reported. An example of the tabular presentation of qualitative findings for an umbrella review titled ‘Experiences of dementia patients and their caregivers with the use of non-pharmacological interventions to manage aggressive behavior’ is presented in Table 3. The synthesized finding presented should be an accurate, verbatim replication of the synthesized finding from the source review. The descriptive information in the final column (Table 3 ‘Details of strategies’) may constitute the umbrella review authors’ own words to provide the necessary detail for interpretability.
Summary of evidence
The umbrella review methodology presented here concludes with a simple means to facilitate the summary presentation of the results to the reader of the umbrella review. For quantitative findings, a final ‘Summary of evidence’ table should be presented that names the intervention, identifies the included research synthesis and provides a clear and simple indication of the results for the reader. For example, presentation of the results could follow a simple, visual ‘stop-light’ indicator, where green indicates the intervention is beneficial (effective), amber that there is no difference in the investigated comparison and red that the results suggest the intervention is detrimental or less effective than the comparator. Actual details and effect estimates are presented in the findings of the umbrella review (see Table 2). Table 4 presents an example for the outcome ‘aggressive behavior’. Further outcomes reported in an umbrella review could be added in columns to the right. When a review does not report on an outcome, the indicator square should be left blank.
Similarly, in the final summary table presented in relation to a qualitative umbrella review question, the key synthesized findings should be presented for the reader; for other contextual details, the main findings can be referred to (see Table 3). As with summary presentation of quantitative findings, visual indicators of the finding may also be included when possible. In the example provided in Table 5, those perspectives (see Table 5 ‘Phenomenon of interest’) that are beneficial or facilitative are highlighted in green, while those that are inhibitory are highlighted in red.
Umbrella reviews provide a ready means for decision makers in healthcare to gain a clear understanding of a broad topic area. Reviewers should consider the conduct of an umbrella review when questions are posed that have a wide scope, for example, requiring investigation of a range of interventions for a particular problem,3 and when evidence is required rapidly to inform a new policy or procedure,5 and it is clear that existing research syntheses are available. Whilst methods for the conduct of umbrella reviews of quantitative evidence are available,3 the methodology for an umbrella review described here is the first that considers the synthesis of qualitative syntheses. Reviewers must endeavor to provide the contextual detail of the results and findings presented in the umbrella review to create an accurate and complete picture for the reader.18 The umbrella review methodology presented here is different from that of Cochrane.3 A Cochrane Overview of Reviews typically includes only relevant Cochrane intervention reviews produced by individual Cochrane review groups and would not consider for inclusion non-Cochrane reviews or qualitative evidence. Our methodology is more inclusive of other published reviews and meta-analyses.
Most of the features described here are not unique for the operational conduct of an umbrella review, and researchers familiar with the conduct of a systematic review will immediately identify the similarities in process and methods used. Despite these similarities, there are several important features for researchers undertaking an umbrella review worth noting. Firstly, in an umbrella review, the knowledge base at the primary research level is not being examined. The umbrella review author must remain at the level of the research syntheses and discuss the results and knowledge relevant to the review question from this higher level. The umbrella review is, in essence, a summary of the syntheses that exist of the evidence. As a summary of the review evidence, the umbrella review should not revert to reporting of the evidence from included reviews in turn (just as in the systematic review, single study reporting is not an acceptable method of synthesis). Hence, when reporting the findings, the umbrella review authors should look to present outcomes or concepts across the included reviews in order to summarize the evidence in a user-friendly manner for the reader. These combinations can be based upon research methodology (e.g. all evidence gained from systematic reviews of randomized controlled trial data) or can be based on outcomes examined (e.g. all reviews measuring pain). Reporting of findings could also be done by groupings of participants (e.g. outcomes for children). The working group favored the presentation of summary of evidence tables using a clear visual and coloured indication system aligned to the results to facilitate ease of use of the umbrella review report.
With the ever increasing number of systematic reviews published daily, umbrella reviews have a clear role in evidence-based healthcare and evidence-informed decision-making.
The authors wish to acknowledge the contribution to this work of attendees of the Umbrella Review Methodology workshop held at the Joanna Briggs Convention in October 2013 and also the Joanna Briggs Institute Scientific Committee.
Conflicts of interest
E.A., C.G. and H.K. were all members of the JBI Scientific Committee when the JBI Umbrella Review Methodology was considered. All three abstained from ratification of the methodology.
1. Bastian H, Glasziou P, Chalmers I. Seventy-five trials and eleven systematic reviews a day: how will we ever keep up? PLoS Med
2. Hartling L, Chisholm A, Thomson D, Dryden DM. A descriptive analysis of overviews of reviews published between 2000 and 2011. PLoS One
3. Becker LA, Oxman AD. Chapter 22: overviews of reviews. In Higgins JPT, Green S editors. Cochrane handbook for systematic reviews of interventions, Version 5.1.0 (updated March 2011). The Cochrane Collaboration; 2011. www.cochrane-handbook.org
4. Smith V, Devane D, Begley C, Clarke M. Methodology in conducting a systematic review of systematic reviews of healthcare interventions. BMC Med Res Methodol
5. Khangura S, Konnyu K, Cushman R, Grimshaw J, Moher M. Evidence summaries: the evolution of a rapid review approach. Syst Rev
6. Worswick J, Wayne SC, Bennett R, et al. Improving quality of care for persons with diabetes: an overview of systematic reviews: what does the evidence tell us? Syst Rev
7. Aromataris E, Pearson A. The systematic review: an overview. Am J Nurs
8. Aromataris E, Fernandez R, Godfrey C, Holly C, Kahlil H, Tungpunkom P. Methodology for JBI umbrella reviews. In Joanna Briggs Institute reviewer's manual 2014. The Joanna Briggs Institute, Adelaide. http://joannabriggs.org/assets/docs/sumari/ReviewersManual-Methodology-JBI_Umbrella%20Reviews-2014.pdf
9. Stern C, Jordan Z, McArthur A. Developing the review question and inclusion criteria. Am J Nurs
10. Grant M, Booth A. A typology of reviews: an analysis of 14 review types and associated methodologies. Health Inform Libraries J
11. Aromataris E, Riitano D. Systematic reviews: constructing a search strategy and searching for evidence. Am J Nurs
12. Moher D, Liberati A, Tetzlaff J, Altman DG The PRISMA Group (2009). Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. PLoS Med 6(7): e1000097. doi:10.1371/journal.pmed.1000097.
13. Shea BJ, Grimshaw JM, Wells GA, et al. Development of AMSTAR: a measurement tool to assess the methodological quality of systematic reviews. BMC Med Res Methodol
14. Whiting P, Savovic J, Higgins JPT, et al. and the ROBIS groupROBIS: a new tool to assess risk of bias in systematic reviews was developed. J Clin Epidemiology
2015; doi: 10.1016/j.jclinepi.2015.06.005.
15. Kynoch K, Wu CJ, Chang AM. The effectiveness of interventions in the prevention and management of aggressive behaviours in patients admitted to an acute hospital setting. JBI Library Syst Rev
16. Forbes D, Forbes S, Morgan DG, Markle-Reid M, Wood J, Culum I. Physical activity programs for people with dementia. Cochrane Database Syst Rev
2008; (3): CD006489. doi: 10.1002/14651858.CD006489.pub2.
17. Finfgeld-Connett D. Management of aggression among demented or brain-injured patients. Clin Nurs Res
18. Munn Z, Tufanaru C, Aromataris E. JBI's systematic reviews: data extraction and synthesis. Am J Nurs