There is a growing use of multiple qualitative methods and methodologies* as part of the public health practitioners' tool kit. Qualitative methods or research tools, such as in-depth interviews and case studies, play an important role in research and evaluation as they provide broader context to understand issues, patterns, and people's experiences.1,2 From an epistemological standpoint, qualitative methodologies are “concerned with the meanings people attach to their experiences of the social world and how people make sense of that world.”3(p3) Qualitative methodologies have been invaluable for both formative and summative program planning and evaluation. Traditionally, in-depth interviews, direct observation, and document review have been utilized to investigate program accomplishments and effectiveness, as well as to inform quality improvement (QI).2,4–7 Qualitative methods may be used independently or in complement with quantitative methods in mixed-methods studies to gain a more comprehensive understanding of the study topic.3
Qualitative methods are being used in more collaborative, participatory, and action-focused community-based public health practice. A more recent push for utilization-focused evaluation emphasizes “the extent to which intended users actually use the findings for decision making and program improvement.”2(p10) Public health practitioners and funders are increasingly recognizing participatory action research as a transformative research paradigm to address health disparities.8,9 Participatory action research “is meant to enlighten and empower ... motivating each individual to take up and use the information gathered in the research.”10(p250) Despite the expanded uptake of qualitative methods and methodologies in public health research, confusion remains among many practitioners as to available approaches, when to use them, and to what end.
The Primary Care Capacity Project (PCCP), implemented by the Louisiana Public Health Institute (LPHI), utilized qualitative methodologies and methods as an integral part of the mixed-methods project evaluation framework to identify project areas of focus, evaluate clinical transformation interventions in selected heath centers along the Gulf Coast, and provide data and support to foster ongoing QI approaches. Primary Care Capacity Project is part of the larger Gulf Regional Health Outreach Project (GRHOP), a series of 4 integrated, 5-year projects to strengthen health care in certain Gulf Coast communities in Louisiana, Mississippi, Alabama, and the Florida Panhandle. Gulf Regional Health Outreach Project was developed jointly by BP and the Plaintiffs' Steering Committee as part of the Deepwater Horizon Medical Benefits Class Action Settlement.11 Gulf Regional Health Outreach Project consists of 4 projects: PCCP, the Mental and Behavioral Health Capacity Project, the Environmental Health and Literacy Project, and the Community Health Workers Project, and also incorporates a Community Involvement Project.11,12 The qualitative evaluation of PCCP captured the work of several of the GRHOP partner activities, highlighting the synergy and integration of systems and services.
Qualitative methods and methodologies were specifically chosen and utilized as part of PCCP's formative, implementation, and summative evaluation. While the methods achieved their evaluation purposes, they also had unanticipated benefits for participating health centers and community stakeholders. This article highlights the qualitative methods selected at each point in the evaluation as well as how the participatory and action-oriented methodologies became interventions in themselves, leading to the empowerment of participants and stakeholders.
The LPHI PCCP team (hereafter referred to as PCCP) systematically and rigorously applied qualitative methods and methodologies throughout the course of PCCP as a key component of the project's mixed-methods evaluation approach. The major qualitative methods described in this article (community prioritization meetings, dyadic interviews, and case studies) were specifically considered and selected by trained qualitative evaluation professionals for the purposes of formative evaluation, implementation evaluation, and summative evaluation. Table 1 provides an overview of each, describing the method selected by evaluation type, methodology, and expected results.
As part of the formative evaluation, PCCP conducted a rapid mixed-methods assessment of primary and secondary health data for Gulf Coast communities at the start of the project. Community prioritization meetings were the core component of this formative work that helped identify and prioritize key community health needs and issues to inform the funding priorities and agenda for PCCP. This participatory group interview approach was selected because it promoted community engagement and brought together diverse stakeholders to discover or elaborate on health information in a short period of time.
Community prioritization meetings were held in Mississippi, Alabama, and Florida in December of 2012, and 2 community prioritization meetings were held in Louisiana in November of 2013. The community prioritization process developed by LPHI was informed by best practices from the Catholic Health Association and the National Association of County & City Health Officials' Mobilizing for Action through Planning and Partnerships processes for selecting measurements that summarize the state of health and quality of life in a community.13,14 Primary Care Capacity Project developed a semistructured interview guide, participant workbook, and data collection protocol tailored to gather data on Gulf Coast–specific community health needs and priorities. A comprehensive, though not exhaustive, list of potential stakeholders was compiled with input from GRHOP partners, state and local departments of health, primary care associations, public health institutes, and local community-based organizations.
During the meetings, the stakeholders participated in a 3-stage process of (1) synthesis of existing secondary data; (2) small group facilitation using semistructured interview guides; and (3) community prioritization through real-time polling. Demographic and health data at the subcounty and county level were presented to the group. Next, stakeholders broke out into small groups for facilitated discussion to identify health and health care needs, barriers to care, existing community resources, and potential solutions. Groups were also asked to identify target populations and/or geographic areas in their communities with the highest physical health, behavioral health, occupational health, and environmental health needs. Representatives from each of the small groups presented their findings to the larger group of stakeholders, and data were compiled into a list of health issues and health interventions. Finally, stakeholders were asked to rank their top health issues and interventions to address these issues using Turning Point, an audience response system.15
Dyadic interviewing is a qualitative method that produces data through the interaction of 2 participants.16–18 This approach shares many of the same benefits as focus groups over individual interviews in that the interaction between the 2 participants often guides the discussion and allows for more rich details and information to naturally flow out of the conversation. However, dyadic interviews can be more easily scheduled and conducted than focus groups as they require fewer participants. Primary Care Capacity Project utilized dyadic interviews to gain a deeper understanding of health centers' experiences participating in PCCP interventions and to receive feedback on technical assistance (TA) offerings. The dyadic interviews were conducted in 2016 during the regional care collaborative (RCC), a learning community organized by PCCP for participating health centers and GRHOP partners.11
Each dyadic interview focused on a key PCCP intervention component identified a priori by the PCCP programmatic team: dental care, transitions of care, reporting software, sustainability, workflow assessment, the community-centered health home demonstration project, environmental/occupational medicine, health departments operating federally qualified health centers, and the Greater New Orleans PCCP Quality Improvement Initiative (a project to reduce nonemergent emergency department visits). Participants from 2 health centers were paired together for each of the topic areas; typically, there were only 2 participants, although a few interviews had 2 participants per center. Pairs were purposively selected on the basis of the centers' experience or expertise with the interview topic. Some pairs were chosen because they had very similar experiences or challenges while other pairs were selected because they had different levels of expertise or experience with the topic. The facilitator used a semistructured interview guide, and participants were able to ask their own questions as well.
As part of the PCCP summative evaluation, the evaluation team conducted a series of case studies of selected health centers from 2015 to 2017. The purpose of the case studies was to evaluate clinical transformation and document GRHOP synergy and PCCP accomplishments through the experience of specific health centers, looking at their successes and challenges.
Primary Care Capacity Project utilized a RAP for the case studies. A RAP is an “intensive, team-based qualitative inquiry using triangulation, iterative data analysis and additional data collection to quickly develop a preliminary understanding of a situation from the insider's perspective.”19(p1) A RAP is similar to ethnography but it requires a team of at least 2 individuals, which allows triangulation of data to happen at a rapid rate. The RAP method was selected because it would allow a small team to become intimately knowledgeable with the centers and collect a large amount of data from a multitude of interviews and observations over a relatively short period of time.
Several health centers that represented the geography of PCCP and participated in multiple GRHOP projects or initiatives were selected to be case study sites. The case studies were mainly explanatory in nature, focusing predominantly on the health centers' PCCP experience and showcasing how the integration of PCCP and the other GRHOP projects impacted the centers. Case study topics of interest included experiences participating in PCCP and GRHOP; experiences integrating multiple GRHOP projects; impacts of PCCP on patient-centered medical home recognition, health information technology, and mental/behavioral health; challenges and lessons learned; and sustainability.
In addition to the case studies outlined by the PCCP evaluation plan, several health centers that were not originally selected as case study sites saw the value of the RAP and approached LPHI to request their own case studies. The centers were all relatively new and facing challenges with patient utilization of services and clinic sustainability. As a result, these case studies were exploratory, driven by research questions developed by the health centers' leadership. The major themes of these case studies included health needs and issues of the community, community access to and utilization of health services, community awareness of the center and services they offered, and community perceptions of the center.
Primary Care Capacity Project followed a similar process for each of the health centers. First, PCCP submitted the research protocol to the Sterling Institutional Review Board that classified the study as nonhuman subjects' research. Then, a team of 2 to 3 PCCP staff spent a week at each center, conducting interviews of staff and patients, as well as observations of the health center waiting rooms. For the exploratory case studies, the team also conducted intercept interviews with community members and key informant interviews with community stakeholders.
Community prioritization meetings gathered diverse stakeholders together, many of whom did not know one another or had not worked together previously. Community stakeholders thoughtfully contributed to the group discussions by sharing their top concerns based on their personal and professional experiences in their communities. Through the prioritization process, the stakeholders came to a consensus on the most pressing issues to be addressed by PCCP as well as GRHOP as a whole, and informed funding priorities at the health center and system levels. Following the meetings, PCCP developed final reports that synthesized existing data and the prioritization results. These reports were shared with the stakeholders.
Table 2 summarizes the top health issues identified during the prioritization meetings. Chronic disease, behavioral health, and access to care were common foci across all 4 states.
The dyadic interviews provided LPHI with critical insights into the context within which the centers were operating as well as a more in-depth understanding of the centers' experiences and perceptions of the TA provided through PCCP and the other GRHOP projects and initiatives. For instance, participants in the workflow assessment interview emphasized how the TA they received from LPHI had been very valuable, as LPHI identified issues and provided constructive ways to address them. One participant shared how they were now optimizing their electronic health record and another participant proudly stated that they were focusing on increasing efficiency and had cut wait times by one-third. Dyadic interview findings allowed PCCP to improve current programming and identify priority areas for additional TA.
In addition, this method provided health center leadership and staff the opportunity to engage in peer-to-peer learning as they were able to speak directly with peers using common technical language and explore issues of interest to them at greater depth. For example, during an interview on dental care, after the facilitator asked a few initial questions, the 2 participants engaged in free-flowing conversation, as 1 center had recently applied for a grant for a new mobile dental unit and the other center already had a mobile unit:
Participant 1: So I see the strength in [your mobile unit] model- not only from the revenue generating side, but an efficiency standpoint. You're able to streamline a lot through this tailored service. So did you learn that through lessons learned or did you start out that way?
Participant 2: We started out that way. The health department had been doing a sealant program and we looked at how they were doing it and modeled it after them.... You've got to have to a good relationship with the school principals [...]
Participant 1: It's funny that you mentioned the value of that relationship. When we were writing the grant, we also went after the support of the superintendent [...]
Participant 2 then asked participant 1 about their plans to buy a mobile unit and offered to share their vendor's contact information, as well as set up a tour of their own mobile unit. The participants also discussed common challenges that they were facing with patient access and began brainstorming potential solutions together.
Participant 1: Are there any strategies you think we can use to get more patients to come in?
Participant 2: We try to participate in every health fair type or outreach event. Even if we're not providing services, we'll bring the bus up there—the medical bus or the dental bus, let people get on it and look at it ... it's a big billboard basically. You can try to find a first Friday type of community event too.
Participant 1: That's a great idea! [The town] does have a new fishermen area with new programming.
The case studies provided comprehensive portrayals of the centers' PCCP and GRHOP stories, as health center staff and patients, as well as community members, shared varying perceptions, opinions, and experiences. During the exploratory case studies, many community members listed a number of community health issues and also explained the reasons why individuals were not seeking or accessing care. In addition, community members sometimes had negative perceptions of the health center or were unaware of the services offered by the center. For the explanatory case studies, clinic leadership described their centers' journeys throughout PCCP, highlighting their lessons learned, greatest achievements, and challenges around integrating multiple projects and future sustainability. Health center staff would typically speak about their own role and less about overall clinic vision, but often had different views than leadership of what they considered to be PCCP successes and challenges, especially around the electronic health record and behavioral health integration. Primary Care Capacity Project utilized the case study findings to inform actionable recommendations to the health centers around different themes, including community outreach, internal communication, operations, partnerships, and sustainability, among other areas.
The RAP approach itself proved to be beneficial for some of the centers. At 1 center, the PCCP team interviewed several neighboring industries that the center had never approached. These conversations helped forge new connections and potential business opportunities for the center. At another center, the site manager gained a new understanding of her staff as a result of the RAP. Because of a lack of available clinic space, many of the interviews were conducted in her office, with her present. At the end of the day, she pulled the evaluation team aside and became emotional as she described how listening to the interviews was a great experience because she heard so many concerns and such passion from her staff, helping her see how committed they are to their jobs.
The use of utilization-focused evaluation and participatory qualitative methods was highly congruent with PCCP's programmatic mission to expand high-quality primary care to low-income populations in Gulf Coast counties and parishes impacted by the BP oil spill. Qualitative methods were used throughout the project life cycle as part of a comprehensive mixed-methods evaluation plan, from genesis of the intervention components to project completion and sustainability guidance. Beyond the evaluation, these methods produced added value to the project by supporting community responsiveness and QI initiatives. For clinic and community partners, the networking and reflection will support the sustainability of project initiatives that are responsive to community needs.
In an effort to address the needs of the target communities, PCCP chose a participatory action research approach to inform the first-year funding priorities. The use of a participatory action research approach was invaluable, particularly as LPHI had never worked in the regional context and was unfamiliar with the health needs and disparities of many of the affected counties and parishes.
A critical unintended consequence of the community prioritization meetings was that they served as an intervention for the stakeholders to network and forge new relationships. There were several examples of community stakeholders “taking up and using the information gathered” from the community prioritization meetings.10(p250) These meetings served as a foundation for the continued partnerships throughout the course of PCCP and GRHOP overall. In 1 case, following the community prioritization meeting, stakeholders formed a community advisory group that continues to meet.
The prioritization meeting reports also proved to be extremely beneficial, beyond the initial purpose of summarizing the meetings themselves. Several health centers and hospitals, including those not formally part of or funded by GRHOP, utilized the findings and data from the reports for their own strategic planning, community health needs assessments, and community health improvement plans.
The dyadic interviews acted as action-oriented interventions, consistent with the peer-to-peer learning focus and goal of the RCC. Participants assumed the role of facilitator to ask one another additional questions and garner information relevant to their work. Rather than only answering questions of utmost interest to PCCP, participants steered the conversation to topics and deeper details that were most important and timely to them. Numerous participants exchanged advice, lessons learned, and resources with the other health center staff participating in the interview, and several interviews lasted longer than the assigned time, or participants continued meeting once the interviews finished.
Had it not been for the dyadic interviews, it is very likely that these participants from different parts of the Gulf South would not have connected at the RCC. Many of these new connections were maintained and strengthened following the interviews, leading to more meaningful collaboration between health centers often representing diverse geographies and patient populations. During the RCC the following year, some of the dyads voluntarily sat together and continued their conversations, sharing updates and lessons learned.
Researchers have argued that RAP has moved beyond “quick and dirty” assessment to a “public health tool for time-sensitive development of changes in intervention strategies, community-based organizational structure, program evaluation, and policy decisions.20 The case studies and RAP method allowed LPHI to provide value back to the health centers, as the findings helped answer their questions. For example, patient interviews and observations of one center's waiting room uncovered several issues with front desk staffing and policies that were associated with some negative community perceptions of the center. After determining that additional focus on these issues was crucial, PCCP brought in a technical expert to conduct an integrated workflow assessment that evaluated clinic operations and identified opportunities to improve community engagement, patient experience, productivity, and workflow efficiency.
Implications for Policy & Practice
- Qualitative methods are critical tools for public health practitioners to support project planning and improvement as well as community change.
- There are a variety of qualitative methods available. It is important for practitioners to work with evaluators or researchers to identify appropriate methods and methodologies for each community and organization, as well as the stage of development of the project.
- The iterative nature of research, action, and reflection is valuable throughout the life cycle of a project including pre-project (planning), in-project (implementation), and post-project (follow-up) phases for not only evaluators and practitioners but also stakeholders.21
- A critical component of PCCP's success was inclusion, transparency, and rapid information exchange.
For some health centers, their participation in the RAP acted as an intervention. For instance, at 1 center, as the community health worker went out into the community multiple times with the PCCP team, she saw opportunities to speak with community members and disseminate information on the center's services. The RAP empowered this community health worker to reframe her role and increase her presence in both the community and the health center. This was especially critical in changing the community's often incorrect notions or negative perceptions of the center.
1. Darlington Y, Scott D. Qualitative Research in Practice: Stories from the Field. Crows Nest, New South Wales, Australia: Allen & Unwin; 2002.
2. Patton MQ. Qualitative Research & Evaluation
Methods. 3rd ed. Thousand Oaks, CA: Sage Publications, Inc; 2002.
3. Pope C, Mays N. Qualitative Research in Health Care. 2nd ed. London: BMJ Books; 2000.
4. Alderfer MA, Sood E. Using qualitative research methods to improve clinical care in pediatric psychology. Clin Prac Pediatr Psychol. 2016;4(4):358–361.
5. Brantlinger E, Jimenez R, Klingner J, Pugach M, Richardson V. Qualitative studies in special education. Except Child. 2005;71(2):195–207.
6. Gullick J, West S. Uncovering the common ground in qualitative inquiry: combining quality improvement
and phenomenology in clinical nursing research. Int J Health Care Qual Assur. 2012;25(6):532–548.
7. Marshall M, Mountford J, Gamet K, et al Understanding quality improvement
at scale in general practice: a qualitative evaluation
of a COPD improvement programme. Br J Gen Pract. 2014;64(629):e745–e751.
8. Mercer S, Green L. Federal funding and support for participatory research in public health and health care. In: Minkler M, Wallerstein N, eds. Community Based Participatory Research for Health: Process to Outcomes. 2nd ed. San Francisco, CA: Jossey-Bass; 2008.
9. Wallerstein N, Duran B. Community-based participatory research contributions to intervention research: the intersection of science and practice to improve health equity. Am J Public Health. 2010;100(suppl 1):S40–S46.
10. Berg BL. Qualitative Research Methods for the Social Sciences. 7th ed. Boston, MA: Pearson Education, Inc; 2009.
11. GRHOP. About the gulf region health outreach program. http://http://www.grhop.org
/pages/about.aspx. Published 2014. Accessed January 7, 2017.
12. Netters T, Clesi Giepert J, Baumgartner E. GRHOP [PCCP] how to orchestrate it: working with community and different sectors to achieve population health outcomes. In: Bialek R, Beitsch LM, Moran JW, eds. Solving Population Health Problems Through Collaboration. New York, NY: Routledge; 2017.
13. Catholic Health Association. Assessing and addressing community health needs: a summary of new requirements and recommended practices. http://www.chausa.org
/store/products/product?id=3008. Published 2015. Accessed January 7, 2017.
14. National Association of County & City Health Officials. MAPP framework. http://archived.naccho.org/topics/infrastructure/mapp/framework/index.cfm. Published 2017. Accessed January 7, 2017.
16. Eisikovits Z, Koren C. Approaches to and outcomes of dyadic interview analysis. Qual Health Res. 2010;20(12):1642–1655.
17. Morgan DL, Eliot S, Lowe RA, Gorman P. Dyadic interviews as a tool for qualitative evaluation
. Am J Eval. 2016;37(1):109–117.
18. Polak L, Green J. Using joint interviews to add analytic value. Qual Health Res. 2016;26(12):1638–1648.
19. Beebe J. Rapid Assessment Process: An Introduction. Walnut Creek, CA: Altamira Press; 2001.
20. Trotter R, Needle R, Goosby E, Bates C, Singer M. A methodological model for rapid assessment, response, and evaluation
: the RARE Program in public health. Field Methods. 2001;13(2):137–159.
21. McAllister K, Vernooy R. Action and Reflection: A Guide for Monitoring and Evaluating Participatory Research. Ottawa, Canada: International Development Research Centre; 1999.
*The term “method” refers to the research technique or tool, whereas methodology describes the rationale and approach for the implementation of a given method.