Qualitative research synthesis

methodological guidance for systematic reviewers utilizing meta-aggregation

Abstract

Qualitative synthesis informs important aspects of evidence-based healthcare, particularly within the practical decision-making contexts that health professionals work in. Of the qualitative methodologies available for synthesis, meta-aggregation is most transparently aligned with accepted conventions for the conduct of high-quality systematic reviews. Meta-aggregation is philosophically grounded in pragmatism and transcendental phenomenology. The essential characteristics of a meta-aggregative review are that the reviewer avoids re-interpretation of included studies, but instead accurately and reliably presents the findings of the included studies as intended by the original authors. This study reports on the methodology and methods of meta-aggregation within the structure of an a priori protocol and standardized frameworks for reporting of results by over-viewing the essential components of a systematic review report.

Introduction

Qualitative research methods facilitate the analysis of human experience, and cultural and social phenomena.¹ Qualitative research has its origins in the humanities and social sciences, and seeks to analyse the complexity of human phenomena in naturalistic settings and from a holistic perspective.² The term ‘qualitative’ applies to various research methodologies including, but not limited to, ethnography, phenomenology, qualitative inquiry, action research, discourse analysis and grounded theory. Each methodology is associated with specific understandings of the nature of knowledge (epistemology), and given the subjectivity inherent in qualitative methodologies there is also an ontological link. Ontological and epistemological understanding contributes to the paradigm within which each methodology is understood based upon the language, assumptions and the rules that guide how the methodology is applied. Qualitative methodologies are distinct from, yet closely aligned to, research methods (data-collection techniques). The methodology informs choices of method, and choice of the method leads to the conduct of data collection, including interviews (whether group or individual) and observation (either direct or indirect). Researchers who use qualitative methodologies seek a deeper understanding, aiming to ‘study things in their natural setting, attempting to make sense of, or interpret, phenomena in terms of the meanings people bring to them’.³

Qualitative evidence comprises data that are expressed in terms of the meaning or experiences of acts or events rather than in terms of a quantitative measurement.^4–6 Arguably, one of the features of its contribution to research inquiry lies in the stories and accounts of living and the richness of meanings conveyed through the words of participants.⁵ It is the words people express in communicating subjective meanings and experiences that form much of the data of interest to qualitative researchers. In the healthcare context, qualitative research ‘...seeks to understand and interpret personal experiences, behaviours, interactions, and social contexts to explain the phenomena of interest, such as the attitudes, beliefs, and perspectives of patients and clinicians; the interpersonal nature of caregiver and patient relationships; the illness experience; or the impact of human suffering’.⁷

Acknowledgement of the contribution that qualitative research findings make in improving the quality and relevance of healthcare practice is increasing. Porritt and Pearson⁸ assert this is, in part, led by key qualitative researchers and key organizations striving to improve health practitioners’, policy makers’, administrators’ and researchers’ understanding of qualitative research and how it informs understandings of human experiences and behaviours. For example, Hannes and Lockwood⁹ describe how the ‘richness of qualitative evidence and process-related information provides credible, complementary material to address practice and policy-related questions’ (p. 1633).

The synthesis of qualitative data

Systematic reviews of qualitative research have an important role in informing the delivery of evidence-based healthcare. Qualitative systematic reviews have investigated the culture of communities, exploring how consumers experience, perceive and manage their health and journey through the health system, and can evaluate components and activities of health services such as health promotion and community development.¹⁰

The paradigms and methodologies associated with primary qualitative research have had a significant impact on the development of qualitative synthesis. It has been proposed that this may, in part, be due to the fact that paradigms are emblematic of not only how knowledge arising from but also how the quality of qualitative research is understood. As Chin and Jacobs¹¹ assert, knowledge as subjective truth requires a researcher or author to explicitly state the chosen paradigm as it has implications for how a reader will understand the written word and how the methodology and methods will be read and understood.

One of the motivators for synthesis is, as Porritt and Pearson⁸ describe, the growing need to consolidate bodies of knowledge that have been subject to critique and evaluation for reliability and dependability. This utilitarian view needs to be contextualized with the understanding of qualitative research and how it has influenced qualitative synthesis. Qualitative synthesis is an interpretive process, and therefore understanding what is meant by interpretation (i.e. what ‘rules’ of interpretation are to be applied) is fundamental to selection of a synthesis methodology.¹² As has been observed by Hannes and Lockwood⁹, most published qualitative synthesis studies do not detail what factors facilitate the choice of methodology beyond familiarity and best fit with the reviewers’ preferred school of thought. While not optimal (as it has methodology driving the question instead of the question driving the choice of methodology), it does allow expert reviewers to proceed within a perspective they are familiar with, and therefore limit reliance upon external rule sets. Indeed, in 2007, Sandelowski and Barroso,¹³ although reluctant to create or promulgate rules for qualitative synthesis, posit that the first rule (if any should exist) is that the methods of synthesis should not violate the philosophic foundations (i.e. paradigm) of the approach used. It is evident then that while synthesis is a different process to primary research, the principles and processes of qualitative synthesis must be sensitive to the core assumptions of the critical and interpretive paradigms.

Meta-aggregation: the Joanna Briggs Institute approach to qualitative systematic review

The Joanna Briggs Institute (JBI) was established in 1996, with a focus on bringing high-quality summarized evidence (all forms of evidence, including qualitative, economic, textual and effectiveness) to clinicians at their place of practice. Recognizing the value of qualitative research for health service delivery, JBI conceptualized a role for qualitative systematic reviews. The meta-aggregative approach was developed in the early 2000 by an expert group of international qualitative researchers, and there are now significant numbers of qualitative systematic reviews published following this approach. Meta-aggregation is sensitive to the nature and traditions of qualitative research while being predicated on the process of systematic review.⁸ It is one of the many available methods of synthesis, with at least 15 available approaches published in the literature to date.

Meta-aggregation is grounded in the philosophic traditions of pragmatism and Husserlian transcendental phenomenology. The roots within pragmatism are reflected in a process-driven approach to the method. The overall emphasis is on producing findings that inform practice-level theory or, as has been described elsewhere, lines of action that have applicability to healthcare policy or practice.⁹ It embodies the complex nature of interpretive and critical understanding, while ensuring the synthesized findings are meaningful and practical. Transcendental phenomenology is a significant influence in meta-aggregation, the purpose of which is to develop knowledge in an unbiased way, not prejudiced nor influenced by self or outside factors, but instead is illuminated by the particulars of intentional consciousness of the phenomena of interest.¹⁴

Meta-aggregation seeks to enable generalizable statements in the form of recommendations to guide practitioners and policy makers.⁹ As such, it contrasts with meta-ethnography or realist synthesis, which has a focus on re-interpretation and mid-level theory generation. Lockwood and Pearson¹⁴ posit that when deciding upon a methodology for qualitative synthesis, that consideration of whether the primary question is suited to an integrative approach such as meta-aggregation or whether the reviewers prefer to act as interpretive agents from limited published data such as in meta-ethnography.

Planning and preparation for a meta-aggregative systematic review

The phases involved in conducting a meta-aggregative review are parallel to the well defined, internationally accepted characteristics of systematic reviews. As outlined by Aromataris and Pearson¹⁵, the features of a systematic review include the following:

1. a clearly defined objective and question
2. detailed inclusion and exclusion criteria
3. a comprehensive search strategy
4. quality appraisal of the included studies (although this is a contentious issue in qualitative systematic reviews and will be discussed later in the study)
5. analysis of the data extracted
6. presentation and synthesis of the findings and,
7. transparent reporting of the approach undertaken

A meta-aggregative review begins, as with all systematic reviews, with a review protocol. The protocol must be subject to peer review prior to undertaking the review itself. The review protocol pre-defines the objectives and methods, detailing the proposed plan for the review. A protocol also provides background information to the topic being investigated. The protocol background situates the phenomena of interest and the context within which it is understood. At least two reviewers are required to ensure the reliability of the meta-synthesis. Meta-aggregative reviews should be reported in line with current standards for reporting qualitative research synthesis, appropriate to the methodology.

The purpose of this study is to overview the rationale, methodology and methods for meta-aggregation as an approach to qualitative synthesis and to discuss its significance for evidence-based healthcare.

Forming a review question and inclusion criteria

The question(s) guides and directs the development of the specific review criteria. Clarity in the review question(s) assists both the researcher and the reader.¹⁶ For the researcher, it provides a framework for the development of the protocol, facilitates an effective search strategy and provides a structure for the development of the full systematic review report. For the reader, it helps clarify whether the review is relevant to their information needs.¹⁶

The traditional mnemonic for systematic reviews of effectiveness is the population, interventions, comparators and outcomes (PICO) approach. Qualitative reviews seek to understand the meaning of phenomena and their relationships; therefore, a different approach is required in the synthesis of qualitative research.³ In meta-aggregation, a revised PICO mnemonic is used to guide the development of a clear and meaningful question. In meta-aggregation, PICo stands for the population, the phenomena of interest and the context; however, the specific structure and meaning of a question can vary considerably within this framework (Figs. 1–3 illustrate each element of the PICo and how they are structured).

Figure 1: PICo mnemonic of the experience of a primary health care intervention. PICo, population, the phenomena of interest and the context.

Figure 2: PICo mnemomic of a professional phenomenon for review. PICo, population, the phenomena of interest and the context.

Figure 3: PICo mnemomic of experiences arising from decisions made following a diagnosis. PICo, population, the phenomena of interest and the context.

A qualitative review will have a primary question. If that question sufficiently addresses the PICo, there is no need for sub-questions. However, as per the illustrations below, some questions benefit from one or more sub-questions that delve into particular attributes of context, population or phenomena of interest.

For example, the primary question below relates to the nursing profession; however, the sub-questions delve into the particular issues related to professionally trained nurses and student nurses as distinct sub-populations:

1. What are the experiences of lateral or horizontal violence in the profession of nursing?
2. What is the experience of lateral or horizontal violence for professional nurses?
3. What is the experience of lateral or horizontal violence for student nurses?

Inclusion criteria detail the basis on which studies are considered for inclusion into the systematic review. The criteria must be clearly outlined addressing the stated aspects below.

Types of participants

There needs to be a clear and direct link between the review title and the participant characteristics in the inclusion criteria. For example, the population characteristics for conservative treatment for men may consider the following:

1. Age ranges (29–75 years)
2. Sex (male)
3. A diagnosis of prostate cancer (diagnosed within the past 6 months, either new, or recurrent disease)
4. Staging of severity of prostate cancer (I–IV)

The population should be clearly described and avoid ambiguity that may confound study selection.

The reasons for the inclusion or exclusion of participants should be justified (with supporting citations) in the background.

Phenomena of interest

Congruence between the review question and the phenomena of interest must exist. In the example question of men diagnosed with prostate cancer, the phenomena of interest is their experiences having chosen conservative treatment. This may be presented in the form of a statement. How the phenomenon relates to the topic under review should be further detailed in the background.

Context

The context informs aspects of the setting, culture or perspective of the review. Context therefore may include, but is not limited to, the description of the following:

1. cultural or sub-cultural factors,
2. geographic location,
3. specific racial or sex-based interests, or
4. detail about the specific setting (such as acute care, primary health care or the community).

Types of studies

Historically, limiting reviews to a single methodology was considered important (meta-ethnography was originally conceptualized solely for ethnographic studies). Meta-aggregation is more suited to inclusive reviews of all forms of qualitative evidence across paradigms. However, it can be applied in a focused way to either critical or interpretive literature.

A narrow approach in terms of focusing solely on either interpretive or critical designs alone is not recommended unless there is a clear, theoretically informed rationale and a requirement to do so. The international consensus is heavily in favour of inclusive reviews of literature across both the critical and interpretive paradigm.

Searching in meta-aggregation reviews

Searching in a JBI qualitative review is both comprehensive and exhaustive. This includes black and grey (unpublished) literature. Searching has been addressed in previous publications.^17,18

Assessment of methodological quality

Debate continues regarding the virtue of critical appraisal of qualitative studies during the systematic review process. Some have argued for rejection of appraisal as it impinges on interpretation and creativity, whereas others argue appraisal is central to ongoing credibility and transferability of qualitative evidence.¹⁹ In the meta-aggregative approach, critical appraisal assists to inform reviewers on which studies to eventually include in their review. This relates to the purpose of meta-aggregation, which is to develop practice-level theory to inform healthcare policy and practice. In contrast, methodologies such as realist and critical interpretive synthesis, and meta-ethnography are intended to develop mid-level theory to inform new conceptualizations of a phenomenon. These theories then require testing and validation before implementation into policy or practice; whereas the results of a meta-aggregative review are explicitly linked to the quality of the included studies as the results are intended to have immediate applicability to practice, and must meet the rigorous expectations of clinicians for high-quality evidence with appropriate transparent reporting of quality. Thus, critical appraisal is the basis by which the quality and congruency of findings in included studies is established in a meta-aggregative review. A detailed discussion on the critical appraisal instrument advocated for use in meta-aggregative reviews is published elsewhere.²⁰ The critical appraisal questions common to meta-aggregation are outlined in Fig. 4.

Figure 4: Joanna Briggs Institute Critical Appraisal Checklist for critical and interpretive research.

Data extraction

Data extraction in meta-aggregation is a multi-phase process. The general details of studies are extracted first and include citations details, the population, phenomena of interest, context, methodology, methods, settings and cultural information. The meta-aggregative approach can incorporate any number of studies, and there is no ‘ideal’ number nor cut-off. The methods are robust to inclusion of large numbers of studies and complex phenomena, and conversely, to very limited numbers of studies.

Extracting findings is both the second phase of data extraction and the first step in data synthesis. In meta-aggregation, a finding is defined as a verbatim extract of the author's analytical interpretation of the results or data.¹⁸ Each extracted finding is to be accompanied by an illustration from the same study that informs the finding. An illustration may be either a direct quotation of the participant voice, field-work observations or other supporting data.¹⁸

As a finding is extracted, a level of ‘plausibility’ is allocated based on the reviewers’ assessment of the degree of fit, or congruency between the data and the accompanying illustration. There are three levels of plausibility, given as follows:

1. Unequivocal (findings accompanied by an illustration that is beyond reasonable doubt and; therefore not open to challenge)
2. Equivocal (findings accompanied by an illustration lacking clear association with it and therefore open to challenge)
3. Unsupported (findings are not supported by the data)

Reviewers should document in their review report how the decision was made to allocate these levels, what (if any) issues arose during the process or whether there was good agreement between the review team members. Unsupported findings do not appear in the synthesis; this filtering is built into the software Qualitative Analysis and Review Instrument (QARI), which is most commonly used to facilitate conduct and reporting of meta-aggregative reviews.¹⁸

Findings that are ranked unequivocal and findings that are ranked equivocal share equal recognition in the synthesis. The rating is an important indicator of the reliability of the interpretation of the primary study authors compared with the voices of the study participants. This provides transparency for readers and potential users of the evidence. However, for reviewers undertaking further analysis of the confidence associated with their final synthesis and developing a summary of findings table via CONqual, these levels of plausibility become important.¹¹

Data synthesis

The methods of synthesis in meta-aggregation have been extensively described elsewhere.^{8,9,12,14,18,21} Meta-aggregation uses a three-step approach to thematic analysis. A software package (JBI QARI) has been designed to assist and facilitate researchers in this process.²²

In overview, data synthesis in a meta-aggregative review consists of the following:

1. Extraction of all findings from all included studies with an accompanying illustration and allocated level of plausibility for each finding.
2. Developing categories for findings with at least two findings per category.
3. Developing one or more synthesized findings of at least two categories.

Categorization involves repeated, detailed examination of the assembled findings. The reviewer identifies groups of findings on the basis of similarity in meaning to create categories.

A category is the combination of a brief description of a key concept arising from the aggregation of two or more like findings. This description is accompanied by an explanatory statement that conveys the whole inclusive meaning of a group of similar findings.

In meta-aggregation, a synthesized finding is an over-arching description of a group of categories. Synthesized findings are expressed as ‘indicatory’ statements. These form the basis of recommendations for policy or practice crafted by the reviewers as guidance arising from their findings. ‘Indicatory’ is a subjective term; reviewers variously craft their synthesized findings as being suggestive, characteristic, representative, symbolic or emblematic of the evidence being brought together. This open definition facilitates development of synthesized findings that are nuanced to the perspectives of the authors and evidence, but are still able to be used as a basis for informing policy or practice.

In overview, in reporting the data synthesis reviewers describe the following:

1. what data were considered ‘findings’ in their review (i.e. was it limited to themes and metaphors, or did it include other analytic data from the studies that might have been an author observation – as is often the case in ethnographic research – rather than the results of thematic analysis);
2. the process by which findings were identified (i.e. repeated reading of text, or selection of themes from the results section only);
3. how findings were grouped in order to develop categories (i.e. was it based on similarity in wording, or concept);
4. how category descriptions were created (i.e. by single reviewer, or by consensus process between reviewers/review group members);
5. how synthesized findings and their accompanying descriptions were created and finalized.

Meta-aggregation does not involve a re-interpretation nor re-conceptualization of the evidence as is central to meta-ethnography, realist synthesis and critical interpretive synthesis. Instead, it focuses on the combination of findings from across the included studies.^8,20

Visually, the synthesis in meta-aggregation is presented as a progression of data concentration from a larger number of findings through to decreasing numbers of categories and synthesized findings, as illustrated in Fig. 5.

Figure 5: Example of expression of the synthesis in meta-aggregation.

Implications for practice

Meta-aggregative reviews are intended to inform practice. The link between the original data and synthesized findings, critical appraisal to establish the quality of included studies, and the basis in both pragmatism and transcendental phenomenology lend to the development of recommendations for practice that are transparently linked to the data, and not a re-interpretation based on the review authors’ interests.

Therefore, recommendations arising from a meta-aggregative review are practical, specific, detailed and measurable. Review authors draw upon knowledge of the included studies and the participants those studies represent to create context rich recommendations relevant and applicable to practice.

Implications for research

This section should include clear, specific recommendations for future research based on gaps in knowledge identified from the results of the review. Generalized statements regarding repeating the research in other settings are not sufficiently informative. Research recommendations should be developed based on the knowledge gained through the comprehensive searching, review and analysis of the literature on the topic.

Confidence in the evidence from qualitative research synthesis

Meta-aggregation was developed in the health sciences for the evidence needs of healthcare practitioners and policy makers. Confidence in ‘how’ evidence has been developed and reported is as important as the specific recommendations arising from a review, and therefore a high level of transparency in reporting was considered a non-negotiable requirement. To assist with, and maintain transparency in, reporting the findings of meta-aggregative reviews ‘CONqual’ has been adopted as an approach to grading individual findings.

Integrating CONqual with meta-aggregation at the synthesized finding development stage identifies the level of confidence or certainty that reviewers have in the results of a meta-aggregative synthesis.¹⁰ This approach, detailed elsewhere, involves ranking synthesized findings based on their creditability and dependability.¹⁰ CONqual facilitates users of meta-aggregative reviews being able to establish confidence in the findings and serves as a practical tool to assist in evidence-based decision-making.

Conclusion

Evidence-based practice incorporates and utilizes the best available information to assist decision-making related to patients and patient care. Qualitative studies provide important information around the experiences and expectations of healthcare service delivery and, similar to quantitative studies, requires critical evaluation and synthesis. Meta-aggregation is one such approach to assist this process. Meta-synthesis using meta-aggregation ‘produces systematically complied, assessed and concentrated information for clinical administrative decision-making’ (p. 1032) and, as such, the findings resulting from a meta-aggregative review have significance to healthcare.²³

The field of systematic review methodology is an ever changing one. The meta-aggregative approach is now over 10 years old, and there are many examples of this method guiding the conduct of qualitative systematic reviews. As the field evolves, the meta-aggregative approach will continue to do so as well. There is planned work on further evaluating how CONqual and other developments in qualitative evidence synthesis may impact on the meta-aggregative approach.

There currently exist a number of methodologies for conducting the synthesis of qualitative research. The meta-aggregative approach is a widely used method with the pragmatic aim to systematically review qualitative research to generate synthesized findings that can be used to inform healthcare practice or policy.

Acknowledgements

The authors would like to thank the following colleagues for their comments and contribution to the methodology that assisted in informing this study: Dr Leslie Rittenmyer, Professor of Nursing, Purdue University, Calumet; Dr Anthony Tuckett, Senior Lecturer, School of Nursing, The University of Queensland.