Introduction
Qualitative and quantitative systematic reviews each contribute to our understanding of the best available evidence on a topic, yet increasingly, both perspectives are required to inform clinical policy or organizational decisions. Decision-makers who use systematic reviews have argued for a more complete synthesis of the evidence than that currently offered by single method reviews.1 Mixed methods systematic reviews (MMSRs) have become an important development in evidence-based health care as they maximize the ability of review findings to assist in clinical and policy decision-making. This type of review is also referred to as mixed methods research syntheses2 or mixed research syntheses.3
The conceptual foundation of MMSRs is informed by two research paradigms, namely positivism and constructivism. Positivism is associated with quantitative studies, such as prevalence/incidence or descriptive studies, or an analytical study that examines associations between variables or a cause-and-effect relationship.4 Conversely, constructivism is commonly associated with qualitative studies that explore a complex phenomenon of interest.4 Through the development of well-structured MMSRs, the objective numerical data inherent in the logical positivist paradigm combines with the equally important subjective opinions and perspectives presented in the constructivist paradigm. For example, Classen and Lopez5 used a mixed methods review approach to achieve a better understanding of safety issues among older drivers. An initial quantitative synthesis identified risk and protective factors of older driver safety (ie, etiologic studies), followed by a synthesis of qualitative studies that captured the perspectives of older adults relating to their driving ability and safety.5 Without the integration of quantitative results and qualitative results, a complete overarching picture of the inherent complexities associated with older driver safety could not be obtained.
More commonly, MMSRs bring together the findings of effectiveness (quantitative evidence) and patient experiences (qualitative evidence) to allow better understanding of whether and how an intervention works and inform subsequent clinical decision making. For example, although quantitative evidence suggests that the use of larval therapy is clinically and financially effective in the debridement of wounds,6-10 evidence from qualitative studies indicates that negative patient experiences and perceptions impact the acceptability of the therapy.11,12 Much like the previous example, without “combining the power of stories and the power of numbers,”4(p.29) the understanding about the treatment of wounds using larval therapy is incomplete, which can preclude the development of best-practice recommendations.
Depending on the review question posed, MMSRs can examine the degree of concordance between quantitative and qualitative data to validate or triangulate results and findings, identify discrepancies within the available evidence, and determine whether the quantitative and qualitative data address different aspects of a phenomenon of interest (which can subsequently assist in highlighting gaps in research). Mixed methods systematic reviews also allow one type of data to explore, contextualize, or explain the findings of the other type of data. The methodology for conducting MMSRs is an emerging field of enquiry. Although there is a degree of complexity in conducting MMSRs, the core intention is to combine quantitative and qualitative data (from primary studies) or integrate quantitative and qualitative evidence to create a breadth and depth of understanding that can confirm or dispute evidence and ultimately answer the review question posed. Although MMSRs are gaining traction among health care professionals due to their usefulness and practicality, guidance regarding the methodology of combining quantitative and qualitative data is limited and largely at the theoretical stage.13-21
In 2014, the JBI Mixed Methods Review Methodology Group developed guidance for MMSRs based on the segregated approach to mixed methods synthesis as described by Sandelowski et al.,3 which consists of separate syntheses of the quantitative and qualitative components of the systematic review.14,22 A Bayesian approach was then recommended to pool the findings from the individual syntheses. Since the introduction of this guidance, there have been significant developments in the area of mixed methods synthesis.13,15,17,23-25 As such, the methodology group recognized the need to revise the guidance to ensure it was accurate and aligned with the current evidence base.
This article describes the methods utilized to revise the guidance and presents the updated methodological approach for undertaking such reviews. It focuses on the conduct of MMSRs as opposed to the reporting of MMSRs; the full official guidance (including reporting requirements) is available in the JBI Manual for Evidence Synthesis.26 Mixed methods systematic reviews share features that apply to all types of reviews, including formulation of review questions, establishment of eligibility criteria, development of a search strategy, searching and retrieval of relevant studies, assessment of methodological quality, and data extraction. Therefore, the focus of this paper is on illustrating the distinct features of MMSRs as they relate to data synthesis, specifically, methods related to how data are combined and the overall integration of the quantitative and qualitative evidence.
Methods
In 2015, it became apparent to the JBI Mixed Methods Review Methodology Group that revision of the guidance was required. In the following year, the group convened to revisit the existing guidance and update the MMSR methodology. The group was composed of a chair (responsible for chairing the meetings and providing feedback on written work), two convenors (responsible for drafting and coordinating of written work, organizing meetings, and reporting progress to the JBI Scientific Committee), and six members (responsible for attending meetings regularly and providing feedback on written work). All members were academics and experienced in conducting different types of systematic reviews. Group members were from Australia, Canada, Portugal, the United Kingdom, and United States of America. An extensive review of the literature was undertaken, which focused on locating all available methodological guidance in the area of MMSRs as well as published examples of MMSRs. Where required, other experts in the field of mixed methods synthesis were contacted for support and clarification. A series of teleconferences and annual face-to-face meetings were also held between 2016 and 2018, and were supplemented by regular email correspondence. Half-day face-to-face meetings were held on the following dates: November 10, 2016 (Adelaide, South Australia), September 15, 2017 (Cape Town, South Africa), and May 1, 2018 (Antwerp, Belgium). Minutes were recorded to ensure a formal approach to tracking progress, allocating work and responsibilities, and completing milestones was maintained. The proposed guidance was presented at scientific conferences in South Africa (2017 Global Evidence Summit) and Belgium (2018 10th Biennial JBI Colloquium), during which international researchers provided comments that were valuable in informing the methodology.
The final draft of the updated guidance (as a chapter in the JBI Manual for Evidence Synthesis) was completed following a consensus among members, and on August 6, 2018, was submitted to the JBI International Scientific Committee for consideration, discussion, and approval. Following initial submission, the committee approved the guidance pending minor revisions. Comments and feedback were formally addressed by the methodology group, and a revised version was resubmitted on January 31, 2019. On February 13, 2019, the JBI MMSR methodological guidance was ratified at a meeting of the Scientific Committee and thus supersedes all previous MMSR guidance produced by JBI.14,22
Results: JBI methodological approach for conducting an MMSR
To avoid confusion in describing this approach, it is important to outline the core concepts related to MMSRs to fully inform this approach (Table 1).
;)
Table 1: Summary of core concepts related to MMSRs
The JBI approach to MMSRs is based on the typology developed by Hong et al.17 in a review of systematic reviews, which examined the different methods used to synthesize quantitative and qualitative data or integrate quantitative and qualitative evidence. Following the inclusion of 459 reviews, Hong and colleagues17 identified a number of frameworks used for integration. However, in their work, it became evident there were two frameworks that were predominant: the convergent approach (where the synthesis occurs simultaneously) and the sequential approach (where the synthesis occurs consecutively).17 Based on minimal usage of the sequential approach by systematic reviewers (approximately 5%),17 the JBI MMSR methodology currently focuses exclusively on the convergent approach. The convergent design can be broken down into a series of methods that have been simplified into two groups: convergent integrated (which involves data transformation and allows reviewers to combine quantitative and qualitative data), and convergent segregated (which involves independent synthesis of quantitative and qualitative data leading to the generation of quantitative and qualitative evidence, which are then integrated together). The decision as to which approach to use is dependent on the nature/type of questions that are posed in the systematic review. If the review question can be addressed by both quantitative and qualitative research designs, the convergent integrated approach should be followed; if the focus of the review is on different aspects or dimensions of a particular phenomenon of interest, the convergent segregated approach is undertaken. Some example review questions are provided below, which delineate the different approaches.
Example 1:
“What are the barriers and enablers to the adoption of electronic health records to support self-management in adult patients with a chronic disease?”
- Here the focus is on barriers and enablers, which can be addressed through qualitative research (eg, through a phenomenological study of health care professionals involved in supporting adult patients with a chronic disease through the use of electronic health records) as well as quantitative research (eg, through a survey of health care professionals involved in the use of electronic health records conducted as part of a cross sectional study).
- Since this review question can be answered by both quantitative AND qualitative studies, it would follow a convergent integrated approach to its synthesis and integration.
Example 2:
“What are the effects of canine-assisted interventions (CAIs) on the health and social care of older people residing in long-term care?” and “What is the experience of older people residing in long-term care who receive CAIs?”
- Here both questions relate to a common phenomenon (ie, CAIs for older people), but they are addressing two different aspects associated with it: the effects these interventions have on older people (ie, the effect of the interventions on outcomes such as stress and anxiety), and how older people experience or perceive them. Questions of effectiveness are answered through quantitative research (eg, through a randomized controlled trial comparing CAIs with standard interventions), and questions of experience/perception are answered through qualitative research (eg, through an ethnographic study where the researcher undertakes fieldwork on a group of older people receiving these interventions).
- Since this review focuses on different dimensions of a phenomenon, it would follow a convergent segregated approach to its synthesis and integration.
The methodological guidance for the synthesis and integration of these two approaches is presented separately in the succeeding sections.
The convergent integrated approach to synthesis and integration
The convergent integrated approach, outlined in example 1 above, refers to a process of combining extracted data from quantitative studies (including data from the quantitative component of mixed methods studies) and qualitative studies (including data from the qualitative component of mixed methods studies), and involves data transformation. In order for qualitative and quantitative data to be integrated and thus fully inform the topic, one approach is for the data to be transformed into a mutually compatible format.27 Data transformation can occur either by converting qualitative data into quantitative data (ie, quantitizing) or by converting quantitative data into qualitative data (ie, qualitizing). Quantitizing is a process in which qualitative data are assigned numerical values, whereas qualitizing refers to quantitative data being converted into themes, categories, typologies, or narratives.2,3,23
For data transformation, JBI recommends that quantitative data be qualitized, as codifying quantitative data is less error-prone than attributing numerical values to qualitative data.22 Qualitizing involves extracting data from quantitative studies and translating or converting it into textual descriptions to allow integration with qualitative data. Qualitizing involves a narrative interpretation of the quantitative results. At the simplest level, qualitized data might comprise describing a sample (or members of it) using word categories based on supplementary descriptive statistics such as averages or percentage scores.28 Qualitized data can also include profiling of the sample using cluster or factor analysis.28 Data with a temporal or longitudinal component,28 or those that examine associations and relationships using inferential statistics such as linear or logistic regression analysis, also have narrative potential and can therefore be qualitized by identifying variables included in the analysis. By qualitizing, the reviewer converts the quantities into declarative standalone sentences in a way that answers the review question.
The textual descriptions (qualitized data) from quantitative studies are then assembled and pooled with the qualitative data extracted directly from qualitative studies. Reviewers are then required to undertake repeated, detailed examination of the assembled data to identify categories on the basis of similarity in meaning, much like the process of meta-aggregation for qualitative synthesis.29 A category will integrate two or more qualitative data, qualitized data or a combination of both. In some instances, however, data may not have the same meaning as others (ie, may not reciprocally translate across studies)30 and therefore cannot be combined to form a category. Where possible, categories are then aggregated to produce the overall integrated findings of the review. This process is illustrated in Figure 1.
Figure 1: JBI convergent integrated approach where qualitized findings are assembled into categories with other qualitative findings extracted directly from qualitative studies based on similarity of meaning
The convergent segregated approach to synthesis and integration
The convergent segregated approach consists of conducting separate quantitative synthesis and qualitative synthesis, followed by integration of evidence derived from both syntheses. By integrating the quantitative and qualitative synthesized findings, a greater depth of understanding of the phenomena of interest can be obtained, compared to undertaking two separate component syntheses without formally linking the two sets of evidence. The guidance developed for this approach currently focuses exclusively on reviews addressing questions of meaningfulness/experience (qualitative) and effectiveness (quantitative).
In example 2 above, quantitative data is synthesized in the form of a meta-analysis (or a narrative summary if meta-analysis is not possible) to determine the effects of canine-assisted interventions on older adults residing in long-term care. Additionally, the qualitative data are pooled (in the case of the JBI approach, through the process of meta-aggregation, or through a narrative summary if meta-aggregation is deemed inappropriate) to determine the experiences/perceptions of older adults receiving these interventions. There is no order to which synthesis is done first, as they are independent; however, both must be completed before moving to the next step: integration of quantitative and qualitative evidence. This next step involves juxtaposing the synthesized quantitative results with the synthesized qualitative findings, and organizing or linking the results and findings into a line or argument to produce an overall configured analysis. This is where the reviewer considers how (and if) the results and findings complement each other by using one type of evidence to explore, contextualize, or explain the findings of the other type of evidence. In this step, results and findings cannot be reduced but are organized into a coherent whole.3 In this approach, the reviewer repeatedly compares the results of the quantitative synthesis with the findings of the qualitative synthesis, analyzing the intervention that had been investigated for effectiveness (quantitative) in light of the experiences of the participants (qualitative). The following questions act as a guide for this process:
- Are the results/findings from individual syntheses supportive or contradictory?
- Does the qualitative evidence explain why the intervention is or is not effective?
- Does the qualitative evidence help explain differences in the direction and size of effect across the included quantitative studies?
- Which aspects of the quantitative evidence are or are not explored in the qualitative studies?
- Which aspects of the qualitative evidence are or are not tested in the quantitative evidence?
In some instances, the reviewer may find that the results of the quantitative synthesis are not complementary or have no relationship with the findings of the qualitative synthesis, or vice versa. In such cases, the reviewer may identify gaps where further research may be useful to explain the contradictory findings or when there is no relationship between the qualitative findings and quantitative results. The JBI convergent segregated approach to synthesis and integration is illustrated in Figure 2, while Figure 3 provides a summary of both approaches.
Figure 2: JBI convergent segregated approach where separate quantitative and qualitative syntheses are undertaken followed by integration of evidence derived from both syntheses
Figure 3: JBI approach for mixed methods systematic reviews (PICo: participants, phenomena of interest, context; PICO: population, intervention, comparison, outcomes)
Discussion
Mixed methods systematic reviews provide an innovative approach for addressing important questions in health care.31 The increasing interest in this type of review, and the variability and lack of clear detail in the methods to synthesize quantitative and qualitative data or integrate quantitative and qualitative evidence indicate the need for guidance on how MMSRs should be undertaken. Based on a review of the international literature on MMSRs, and with input from experienced researchers in this field, JBI updated its methodological guidance and identified two synthesis designs for conducting MMSRs: convergent integrated, and convergent segregated.
The JBI methodological approach is based on the typology developed by Hong et al.17 as well as the seminal work undertaken by Sandelowski and colleagues.3,32 The convergent integrated approach is similar to Sandelowski's integrated design, which involves direct assimilation, and is based on the assumption that quantitative and qualitative data can both address the same research question.3,32 As such, they can be combined once data have been transformed in the same format (ie, quantitized or qualitized). Comparable to JBI's convergent integrated approach and Sandelowski's integrated design is the data-based convergent design identified by Hong et al.,17 which typically involves a broad systematic review question (that can be answered by both quantitative and qualitative studies) and a synthesis that occurs following data extraction and transformation.17 On the other hand, the convergent segregated approach is analogous to Sandelowski's segregated design. In contrast to the integrated design, which allows direct assimilation, the segregated design involves the integration of evidence through a method known as configuration. Configuration refers to the arrangement of complementary evidence into a line of argument.3,32 According to Sandelowski et al., complementarity is based on the assumption that quantitative and qualitative evidence address different research questions that are related to the same phenomenon of interest.3,32 In other words, quantitative and qualitative evidence address different aspects or dimensions of a phenomenon of interest; therefore, they can neither corroborate nor refute each other but rather only complement each other. As such, the quantitative and qualitative evidence cannot be directly combined and can only be organized into a coherent whole. This approach to synthesis corresponds to Hong et al.'s17results-based convergent design that typically involves an overall systematic review question with sub-questions (some that can be addressed only by quantitative studies and others that can be addressed only by qualitative studies); there is a separate and simultaneous synthesis of quantitative and qualitative data, followed by the integration of the resulting quantitative and qualitative evidence.
Mixed methods systematic reviews appear to be the most complex and the least developed of all systematic review methods. The updated JBI guidance provides foundational work to this rapidly evolving methodology; however, it provides only a starting point for developing methods for combining quantitative and qualitative evidence in MMSRs, which may be conceived as a narrow conceptualization of mixed methods. It is hoped that in future iterations of JBI guidance, more sophisticated methods for integrating evidence will be developed and explored.
The methodological approach outlined in this paper also comes with some caveats. In the convergent segregated approach, the current JBI guidance specifically focuses on intervention/treatment or effectiveness questions for the quantitative component and on meaningfulness or experience questions for the qualitative component. However, the JBI Mixed Methods Review Methodology Group acknowledges that there are other types of review questions that lend themselves to a segregated approach. For example, an MMSR may ask a prevalence question or patterns of use of a specific treatment (which is quantitative in nature) along with the experiences of patients regarding that treatment (qualitative component). While the group believes that a segregated approach is broad enough to be applied to other types of MMSR questions, future iterations of the JBI methodology will provide explicit guidance on how such questions can be synthesized and integrated in a MMSRs.
One of the distinguishing features of an MMSR is the inclusion of not only primary quantitative and qualitative studies but also primary mixed methods studies. For primary mixed methods studies included in a JBI MMSR, data are extracted so that they can be classified as quantitative or qualitative. In the integrated approach, quantitative data are then qualitized to allow synthesis, whereas in a segregated approach, data are kept separate and then go through either meta-analysis or meta-aggregation (as appropriate), followed by the integration of the resulting evidence. This approach of categorizing data into quantitative or qualitative, particularly for the segregated approach, is ideal for primary mixed methods studies in which the quantitative component is published separately from the qualitative component. This is usually the case for mixed methods research that applies a sequential explanatory design33 (ie, where qualitative findings are used to interpret or explain quantitative results).34 However, for primary mixed methods research where the results represent the actual integration of the quantitative and qualitative data (such as those found in realist evaluation), categorizing data into quantitative or qualitative may not be ideal and philosophically would negate the strength of mixed methods studies. It would seem intuitive that in such instances, data are classified into three streams (ie, quantitative, qualitative, and mixed methods), followed by a configurative analysis to allow integration. This will be future work for the JBI Mixed Methods Review Methodology Group.
In addition to those identified above, the JBI Mixed Methods Review Methodology Group has identified a number of methodological projects that need to be undertaken to advance this field. First, as with other systematic reviews, critical appraisal is an essential component of MMSRs, and currently JBI advocates the use of the appropriate JBI quantitative tools (for quantitative studies and the quantitative component of mixed methods studies) and the JBI qualitative tool (for qualitative studies and the qualitative component of mixed methods studies). It may be necessary to develop a bespoke tool for mixed methods primary studies or identify an already existing critical appraisal tool for use in JBI MMSRs.24,25,35,36 Additionally, with regard to critical appraisal in the integrated approach, further investigation into how the appraisal results of quantitative studies (in which findings have been qualitized) are incorporated into the synthesis is needed.
One of the strengths of a systematic review, particularly JBI systematic reviews, is its ability to provide actionable and explicit practice recommendations. These recommendations are based on review findings that have been assessed using a structured approach: GRADE37 for systematic reviews of effectiveness, and ConQual38 for systematic reviews of qualitative studies. Due to the complexities associated with recommendations being derived from both streams of evidence and the impact of data transformation and/or integration on the grading process, an assessment of the certainty of the evidence using either the GRADE or ConQual approach is currently not recommended for JBI MMSRs following either the convergent integrated or convergent segregated approach. Modification to existing systems that assess the certainty of evidence may need to be investigated or, alternatively, a new system developed for evaluating results or findings from an MMSR. Finally, although this paper has focused on the conduct of reviews and not their reporting, it is evident that there is a lack of consensus in terms of reporting standards for MMSRs. This may be due to the lack of universally agreed and specific guidelines for such reviews. As the demand for this type of review increases along with significant methodological advancements in MMSRs, work can now be initiated to improve the standards for reporting.
Conclusion
This paper outlines an exciting development in the field of mixed methods synthesis. The update of the JBI methodological guidance for conducting an MMSR recommends that reviewers take a convergent approach to synthesis and integration whereby the specific method utilized is dictated by the nature/type of questions that are posed in the systematic review. If the review question can be addressed by both quantitative and qualitative research designs, then the convergent integrated approach should be followed, which involves data transformation and allows reviewers to combine quantitative and qualitative data. If the focus of the review is on different aspects or dimensions of a particular phenomenon of interest, then the convergent segregated approach should be undertaken, which involves independent synthesis of quantitative and qualitative data, leading to the generation of quantitative and qualitative evidence, which are then integrated together. Limitations to the current guidance are discussed, as are a series of methodological projects the Methodology Group will undertake to allow for further refinement of this methodology.
Acknowledgments
All past member of the JBI Mixed Methods Review Methodology Group, experts in the field who were contacted for advice, and those who provided feedback to members of the group when the guidance was presented at international scientific conferences.
References
1. Dixon-Woods M, Agarwal S, Jones D, Young B, Sutton A. Synthesising qualitative and quantitative evidence: a review of possible methods.
J Health Serv Res Policy 2005; 10 (1):45–53.
2. Heyvaert M, Maes B, Onghena P.
Mixed methods research
synthesis: definition, framework, and potential.
Qual Quant 2013; 47 (2):659–676.
3. Sandelowski M, Voils CI, Barroso J. Defining and designing mixed research
synthesis studies.
Res Sch 2006; 13 (1):29.
4. Pluye P, Hong QN. Combining the power of stories and the power of numbers:
mixed methods research and mixed studies reviews.
Ann Rev Public Health 2014; 35:29–45.
5. Classen S, Lopez E.
Mixed methods approach explaining process of an older driver safety systematic literature review.
Top Geriatr Rehabil 2006; 22 (2):99–112.
6. Abela G. Benefits of maggot debridement therapy on leg ulcers: a literature review.
Br J Community Nurs 2017; 22: (Suppl 6): S14–S19.
7. Arabloo J, Grey S, Mobinizadeh M, Olyaeemanesh A, Hamouzadeh P, Khamisabadi K. Safety, effectiveness and economic aspects of maggot debridement therapy for wound healing.
Med J Islam Repub Iran 2016; 30:319.
8. Sun X, Jiang K, Chen J, Wu L, Lu H, Wang A, et al. A
systematic review of maggot debridement therapy for chronically infected wounds and ulcers.
Int J Infect Dis 2014; 25:32–37.
9. Tian X, Liang X, Song G, Zhao Y, Yang X. Maggot debridement therapy for the treatment of diabetic foot ulcers: a meta-analysis.
J Wound Care 2013; 22 (9):462–469.
10. Wilasrusmee C, Marjareonrungrung M, Eamkong S, Attia J, Poprom N, Jirasisrithum S, et al. Maggot therapy for chronic ulcer: a retrospective cohort and a meta-analysis.
Asian J Surg 2014; 37 (3):138–147.
11. McCaughan D, Cullum N, Dumville J. Patients’ perceptions and experiences of venous leg ulceration and their attitudes to larval therapy: an in-depth qualitative study.
Health Expect 2015; 18 (4):527–541.
12. Menon J. Maggot therapy: a literature review of methods and patient experience.
Br J Nurs 2012; 21 (5):S38–42.
13. The Campbell Collaboration. Campbell Collaboration Systematic Reviews: Policies and Guidelines. Campbell Policies and Guidelines Series No. 1; 2019.
14. Pearson A, White H, Bath-Hextall F, Salmond S, Apóstolo J, Kirkpatrick P. A mixed-methods approach to systematic reviews.
Int J Evid Based Healthc 2015; 13 (3):121–131.
15. Noyes J, Popay J, Pearson P, Hannes K, Booth A. Cochrane Qualitative Research Methods Group. Chapter 20: Qualitative research and Cochrane reviews. In: Higgins JPT, Green S, editors. Cochrane handbook for systematic reviews of interventions, 2011.
16. Centre for Reviews and Dissemination. Chapter 6: Incorporating qualitative evidence in or alongside effectiveness reviews. In: Systematic reviews. CRD's guidance for undertaking reviews in health care. York, United Kingdom: University of York, 2009.
17. Hong QN, Pluye P, Bujold M, Wassef M. Convergent and sequential
synthesis designs: implications for conducting and reporting systematic reviews of qualitative and quantitative evidence.
Syst Rev 2017; 6 (1):61.
18. Harden A, Thomas J, Cargo M, Harris J, Pantoja T, Flemming K, et al. Cochrane Qualitative and Implementation Methods Group guidance series-paper 5: methods for integrating qualitative and implementation evidence within intervention effectiveness reviews.
J Clin Epidemiol 2018; 97:70–78.
19. Thomas J, Harden A, Oakley A, Oliver S, Sutcliffe K, Rees R, et al. Integrating qualitative research with trials in systematic reviews.
BMJ 2004; 328 (7446):1010–1012.
20. Harden A, Thomas J. Methodological issues in combining diverse study types in systematic reviews.
Int J Soc Res Method 2005; 8 (3):257–271.
21. Gough D, Thomas J, Oliver S. Clarifying differences between reviews within evidence ecosystems.
Syst Rev 2019; 8 (1):170.
22. JBI. Joanna Briggs Institute Reviewers’ Manual: 2014 edition/Supplement methodology for JBI
mixed methods systematic reviews. Adelaide, Australia: JBI; 2014.
23. Frantzen KK, Fetters MD. Meta-
integration for synthesizing data in a systematic mixed studies review: insights from research on autism spectrum disorder.
Qual Quant 2016; 50 (5):2251–2277.
24. Hong QN, Gonzalez-Reyes A, Pluye P. Improving the usefulness of a tool for appraising the quality of qualitative, quantitative and
mixed methods studies, the
Mixed Methods Appraisal Tool (MMAT).
J Eval Clin Pract 2018; 24 (3):459–467.
25. Hong QN, Pluye P, Fabregues S, Bartlett G, Boardman F, Cargo M, et al. Improving the content validity of the
Mixed Methods Appraisal Tool (MMAT): a modified e-Delphi study.
J Clin Epidemiol 2019.
26. Lizarondo L, Stern C, Carrier J, Godfrey C, Rieger K, Salmond S. Aromataris E, Munn Z, et al. Chapter 8:
Mixed methods systematic reviews.
JBI Reviewer's Manual [Internet]. Adelaide: JBI, 2017 [cited 1 May 2019]. Available from:
https://reviewersmanual.joannabriggs.org/.
27. Voils CI, Hassselblad V, Crandell JL, Chang Y, Lee E, Sandelowski M. A Bayesian method for the
synthesis of evidence from qualitative and quantitative reports: the example of antiretroviral medication adherence.
J Health Serv Res Policy 2009; 14 (4):226–233.
28. Bazeley P. Integrative analysis strategies for mixed data sources.
Am Behav Sci 2012; 56 (6):814–828.
29. Lockwood C, Porrit K, Munn Z, Rittenmeyer L, Salmond S, Bjerrum M. Aromataris E, Munn Z, et al. Chapter 2: Systematic reviews of qualitative evidence.
JBI Reviewer's Manual [Internet]. Adelaide: JBI, 2017 [cited 1 May 2019]. Available from:
https://reviewersmanual.joannabriggs.org/.
30. Melendez-Torres GJ, Grant S, Bonell C. A
systematic review and critical appraisal of qualitative metasynthetic practice in public health to develop a taxonomy of operations of reciprocal translation.
Res Synth Methods 2015; 6 (4):357–371.
31. Noyes J, Booth A, Moore G, Flemming K, Tuncalp O, Shakibazadeh E. Synthesising quantitative and qualitative evidence to inform guidelines on complex interventions: clarifying the purposes, designs and outlining some methods.
BMJ Glob Health 2019; 4: (Suppl 1): e000893.
32. Sandelowski M, Leeman J, Knafl K, Crandell JL. Text-in-context: a method for extracting findings in mixed-methods mixed research
synthesis studies.
J Adv Nurs 2013; 69 (6):1428–1437.
33. Fetters MD, Curry LA, Creswell JW. Achieving
integration in
mixed methods designs-principles and practices.
Health Serv Res 2013; 48 (6 Pt 2):2134–2156.
34. Ivankova NV, Creswell JW, Stick SL. Using mixed-methods sequential explanatory design: from theory to practice.
Field Methods 2006; 18 (1):3–20.
35. Heyvaert M, Hannes K, Maes B, Onghena P. Critical appraisal of
mixed methods studies.
J Mix Methods Res 2013; 7 (4):302–327.
36. Long AF, Godfrey M, Randall T, Brettle A. HCPRDU Evaluation tool for
mixed methods studies [Internet]. 2002 [cited 4 April 2019]. Available from:
http://usir.salford.ac.uk/id/eprint/13070/.
37. Guyatt GH, Oxman AD, Vist GE, Kunz R, Falck-Ytter Y, Alonso-Coello P, et al. GRADE: an emerging consensus on rating quality of evidence and strength of recommendations.
BMJ 2008; 336 (7650):924–926.
38. Munn Z, Porritt K, Lockwood C, Aromataris E, Pearson A. Establishing confidence in the output of qualitative research
synthesis: the ConQual approach.
BMC Med Res Methodol 2014; 14:108.
