Secondary Logo

Journal Logo

Invited Commentaries

Pause, Persist, Pivot: Key Decisions Health Professions Education Researchers Must Make About Conducting Studies During Extreme Events

O’Brien, Bridget C. PhD; Teherani, Arianne PhD; Boscardin, Christy K. PhD; O’Sullivan, Patricia S. EdD

Author Information
doi: 10.1097/ACM.0000000000003535
  • Free

Abstract

During this pandemic, many faculty members have found that their mental energy is redirected toward efforts to adapt and adjust their clinical, educational, and research work to a rapidly shifting reality of uncertain duration.1,2 Clinicians are delivering care in new ways. Educators are teaching in virtual worlds. Researchers have stopped studies or are creating innovative work-arounds. The literature is replete with suggestions for these various audiences.3,4 For researchers, national and local institutions focus much of their guidance on policies, principles, and technical details that pertain primarily to clinical and lab-based research.5–7 Meanwhile, health professions education (HPE) researchers need guidance about whether and how to proceed with education studies they have either begun or were about to begin. Should such research simply stop or at least wait for the “new normal”? We believe there are viable options for education research to continue.

Extreme Events and the Researcher’s Dilemma

Extreme events such as earthquakes, hurricanes, political conflicts, and pandemics are “occurrences that, relative to some class of related occurrences, are either notable, rare, unique, profound, or otherwise significant in terms of their impacts, effects, or outcomes.”8(p407) While disruptions to researchers’ plans occur frequently and often can be addressed through contingency plans for typical occurrences,9 extreme events are so unanticipated and thoroughly disruptive that they necessitate review and reconsideration of the research project.

When these events arise, the researcher’s dilemma is “what to do now?” Some researchers will abandon or reduce their studies to data already collected, perhaps to pursue new studies. Others will continue their studies.10–15 For those who continue, options include pausing the research for a period of time with criteria for when to resume, persisting with the research through the event—often with modifications to the study procedures, or pivoting the focus of their research. These choices are not mutually exclusive, as some research may continue with the original question (pause or persist) and add new questions or modify existing questions (pivot). In this Invited Commentary, we offer a framework, presented as guiding questions, to assist HPE researchers in deciding how to proceed with their research. The questions address the stage of research and the main components of the research process (study purpose, design or approach, procedures, and findings). These are summarized in Chart 1. Chart 2 demonstrates using the framework with an example. We acknowledge that extreme events will spawn their own set of novel education studies, and researchers will follow traditional guidance for quality research.

Chart 1
Chart 1:
Questions, Options, and Considerations for Health Professions Education Researchers Conducting Studies During Extreme Events
Chart 2
Chart 2:
Example of an Exploratory, Qualitative Study Initiated Before the COVID-19 Pandemic and Discussions of Considerations Leading to a Decision on How to Proceed With This Study During the PandemicIn fall 2019, a research team launched a study of physicians’ willingness to share vulnerability with trainees. The team approached the study from a constructivist orientation, beginning analysis of data as soon as data collection began. The team completed interviews with 30 physicians from multiple specialties and career stages by January 2020 and had completed the first phase of data analysis in mid-March when concerns about the COVID-19 pandemic escalated. The team had just begun preparing for the second phase of data collection in which they planned to interview trainees about their perceptions and observations of faculty members. Initially, the team shifted attention to clinical and personal responsibilities and concerns related to the pandemic, but eventually they returned to regular research meetings (virtually) and discussed how to proceed with the study. The discussion posed concerns aligned with research components as detailed below.

Key Guiding Questions

What is the study’s stage?

If a study is still in the design stage, researchers can adjust their plans depending on how intertwined the topic is with the extreme event and whether they can study the topic in other ways or at another time. For example, if the event occurs in a limited region, can the research be conducted elsewhere? Is it worthwhile to adjust the design to compare affected and unaffected regions or populations? If the study is already underway, researchers will need to decide how to manage the event’s impact on the study. Some researchers may decide to pivot, particularly if new funding sources support research in their topic areas with a focus on how the extreme event affects education. Funding may also provide access to new data sources related to the extreme event and encourage collaboration across institutions, disciplines, and fields. Which stage the study is in, along with considerations of the research process, will help with pause, persist, or pivot decisions.

How will the extreme event affect the purpose of the study?

Studies serve a variety of purposes, including description of a phenomenon (e.g., prevalence of burnout in a population), exploration for deeper understanding of a phenomenon (e.g., how do learners perceive and experience inequity in clinical learning environments), testing hypotheses or interventions (e.g., is team-based learning as effective for teaching diversity, equity, inclusion topics as it is for biochemistry topics), and translating theory or knowledge from research to educational practices (e.g., how best to apply retrieval practice in clinical education).16 Extreme events may affect these purposes in different ways. Consider, for example, a descriptive research study examining how students use online modules to study for exams. When the study started, modules were optional or supplemental to in-person lectures. Now, as a result of the pandemic, online modules are the primary option for learning course material. Does this new situation still yield a valid description of the phenomenon? Probably not. Is it still useful to know what patterns looked like before the pandemic? Perhaps—particularly if the purpose of the study could be revised to compare use pre–post or to document trends over time.

Considerations for the pause, persist, or pivot decision relate to the usefulness and credibility of a study. First, is the purpose of the study still relevant and useful? In many cases, the answer is yes, and the study should persist on this basis but may require modifications. Then, the decision will depend on whether these modifications are possible (e.g., ethically, feasibly). Questions about students’ use of online modules are still relevant, but the new data collection may expand to better represent the impact of the pandemic or capitalize on the opportunity to ask about different, yet relevant, outcomes. For example, researchers might incorporate into their study faculty development needs for successful teaching online. Second, can the purpose of the study and research questions still be answered with adequate validity, credibility, or trustworthiness if the study pauses or persists? If not, should it pivot? The answer depends on how the extreme event will affect the phenomenon of interest or intervention, both in the short and long terms. Descriptive research might pivot to focus on trends or comparisons (as in the study of online learning). Exploratory research might persist and remain attuned to shifts or patterns emerging over time as data collection and analysis continue. Research testing a hypothesis or intervention may not be particularly affected or the modified intervention can serve as an additional intervention arm, particularly if conducted in a controlled environment.

How is the study design or approach affected?

Study design relies on the researcher’s philosophical orientation.17 Qualitative research conducted from a constructivist orientation needs to reflect contextual changes. Correspondingly, this research may incorporate changing circumstances into the original research question. Quantitative descriptive studies aim to represent a phenomenon or state of affairs in a designated population and time frame, often from a positivist or postpositivist stance that strives for statistically generalizable claims. How events threaten the external validity of the study outcomes will be important to determine.

How should study procedures change?

Extreme events can affect many aspects of study procedures, including recruitment, intervention, data collection, and data analysis. There are numerous procedural considerations, so we mention a few general points here and acknowledge that many require exploration in the context of a particular study.

Recruitment.

Recruitment may be difficult or impossible during or after the extreme event. Certain groups may be disproportionately affected by the event and harder to access than others. Researchers may use statistical techniques to address this issue or recruit in an unaffected area. Recruitment strategies could change. For example, trying to recruit residents may be challenging using email, but announcements at pandemic-generated, virtual social events may work effectively.

Interventions.

Interventions may need to be paused either until delivery of the original intervention can be resumed or, if that is not possible, until the intervention has been reconceptualized. A key deciding factor is how much the intent, fidelity, or integrity of the intervention would change if altered. A complete redesign may be necessary as educational systems change or content is deemed more or less important during or after an extreme event.

Data collection.

Data collection methods could be scrutinized. New questions or scales might be added to surveys as the study design either pivots or additional information is required for context. Interviews may include new questions to collect information about participants’ current circumstances and experiences of the extreme event or to reflect on changes in their thinking, perspectives, and beliefs. Researchers should examine data collection procedures such as format (e.g., shifting from in-person to video interviews or focus groups or from paper to electronic surveys), consent procedures (e.g., if signature cannot be obtained in person, if additional risks or benefits need to be identified), and researcher training (e.g., to collect data in a new way, to ensure safety, to recognize signs of distress). Any changes made partway through data collection require careful review for changes in the quality and integrity of the data. If initial data sources are limited or no longer available, researchers could consider entirely different data sources and techniques that require flexible or no recruitment, such as the use of text mining methods to study social media content to answer research questions.

Data analysis.

Data analysis may require checks for differences in responses, ratings, patterns, or themes before and after the event. Additional focus groups or interviews could check resonance and nonresonance of findings. The analysis in qualitative studies should be attentive to interpretations related to the extreme event. Researchers might turn to realist framing, which invokes the primacy of context.18 Studies involving data collection before and after the event can use techniques such as interrupted time series and the unexpected event during survey design to determine the significance of the event on the outcome.19 Missing data analysis techniques, such as intent to treat or multiple imputations, should be considered when the data collection has been disrupted or impeded by the event.20 Changes or modifications in the analytic techniques may also require additional collaborators or team members to appropriately address these new challenges.

How should findings address the extreme event?

When researchers report their findings, they need to consider how design and process decisions may affect their findings. We suggest paying careful attention in 2 areas.

Representativeness/validity/trustworthiness.

The researchers will need to ponder whether the data they report on still represent the phenomenon of interest, the population intended, and the perspectives of that population. The researchers may want to describe their participants in phases and clarify any changes in representation in the sample based on the extreme events. Discrepancies may need to be reported as findings or limitations.

Framing/interpretation.

The experience and effects of the extreme event may bring new insights to include in the findings. Researchers should consider how to clarify the additional interpretations that come to light in the aftermath of the extreme event. For example, education studies that examine learning during extreme events such as war and forced migration will need to be interpreted carefully, underscoring the context, politics, and policies that circumscribe the findings from the study.

How should the impact of the extreme event on the study be acknowledged in the manuscript?

The researchers’ decisions have implications for the writing of the manuscript. The researchers will have to decide where to best position the extreme event—introduction, method, discussion, or all 3? If the study pivoted, the extreme event may be best positioned up front, as part of the framing in the introduction. The method section likely will include details explaining the decision to pause, persist, or pivot and associated changes. In qualitative studies, researcher reflexivity will require careful attention to how members of the research team experienced the extreme event and how their experiences influence their interpretations.

To enrich the discussion, researchers might think through the relevance and implications of their findings in light of the extreme event. Some questions include the following: Can we still learn something from the findings, especially if all data were collected before the extreme event? Are data useful for comparison or benchmarking in future studies? Are the research questions still meaningful and important despite changes that have occurred and will likely continue to occur after the event (e.g., if a technology studied before the event becomes obsolete or a policy changes)? The limitations section may elaborate on context and explain modifications, particularly for studies that paused or persisted.

Final Reflections

Extreme events are likely to disrupt all aspects of education1,21 and, correspondingly, education research. Researchers may question the relevance of topics and studies chosen before an extreme event. Such reservations stem from concern about pressing social issues, guilt for using limited resources or burdening participants, and observed shifts in funding and publication priorities.22 For example, several funding agencies, journals, and academic outlets have shifted attention to pandemic issues (the “covidisation” of research)23 and may distract researchers from focus on important educational issues that are fundamental to education even in a time of crisis.

While some of the challenges faced in education research are similar to those of clinical and other social and behavioral science research, the lack of funding for and small-scale and local focus of most HPE research can bring these studies to a halt, especially during times of scarcity and heightened competition for limited resources. We provided questions as reminders of issues that we encourage those who continue their work to consider when determining how to proceed and how to do so in a scholarly manner. These questions should provide a way to think through an educational research study during any unexpected event. They should provide ways forward when such movement initially feels overwhelming or impossible.

References

1. Kim CS, Lynch JB, Cohen S, et al. One academic health system’s early (and ongoing) experience responding to COVID-19: Recommendations from the initial epicenter of the pandemic in the United States. Acad Med. 2020;95:1146–1148
2. Rose S. Medical student education in the time of COVID-19. JAMA. 2020;323:2131–2132.
3. Nicol GE, Piccirillo JF, Mulsant BH, Lenze EJ. Action at a distance: Geriatric research during a pandemic. J Am Geriatr Soc. 2020;68:922–925
4. Padala PR, Jendro AM, Padala KP. Conducting clinical research during the COVID-19 pandemic: Investigator and participant perspectives. JMIR Public Health Surveill. 2020;6:e18887
5. National Institutes of Health. NOT-OD-20-087 Guidance for NIH-funded clinical trials and human subjects studies affected by COVID-19. https://grants.nih.gov/grants/guide/notice-files/NOT-OD-20-087.html. Published March 16, 2020 Accessed April 29, 2020
6. Berkeley Research. COVID-19 Current research operations and continuity planning. https://vcresearch.berkeley.edu/research-continuity-planning-covid-19#researchcontinuity. Accessed April 29, 2020.
7. University of California San Francisco Office of Research. Revised interim UCSF policy on human subjects-related research. https://research.ucsf.edu/revised-interim-ucsf-policy-human-subjects-related-research. Accessed April 29, 2020.
8. Sarewitz D, Pielke R. Extreme events: A research and policy framework for disasters in context. Int Geol Rev. 2001;43:406–418
9. Ruecker R, Svihla V. Ruecker R, Svihla V, eds. Introduction: Sanitized research descriptions and messy realities. In: Navigating Challenges in Qualitative Educational Research: Research, Interrupted. 2020New York, NY: Routledge;13–23
10. Harter R, Petty J, Kelly J. Conducting surveys when disasters strike. 2006. Conference of the American Statistical Association Section on Survey Research Methods; http://www.asasrms.org/Proceedings/y2006/Files/JSM2006-000815.pdf. Accessed April 25, 2020.
11. Narimatsu H, Sugawara Y, Morita T, Hayashi E, Kami M. The impact of the Great East Japan earthquake on research activity in a disaster-stricken area. Ther Innov Regul Sci. 2014;48:NP28–NP29
12. Paige J, Kozmenko V, Morgan B, et al. From the flight deck to the operating room: An initial pilot study of the feasibility and potential impact of true interdisciplinary team training using high-fidelity simulation. J Surg Educ. 2007;64:369–377
13. Paige JT, Kozmenko V, Yang T, et al. Henriksen K, Battles JB, Keyes MA, Grady ML, eds. The mobile mock operating room: Bringing team training to the point of care. In: Advances in Patient Safety: New Directions and Alternative Approaches (Vol. 3: Performance and Tools). 2008Rockville, MD: Agency for Healthcare Research and Quality;http://www.ncbi.nlm.nih.gov/books/NBK43685. Accessed April 25, 2020.
14. Saez AM, Kelly AH, Brown H. Notes from Case Zero: Anthropology in the time of Ebola. http://somatosphere.net/2014/notes-from-case-zero-anthropology-in-the-time-of-ebola.html. Published September 16, 2014. Accessed April 29, 2020.
15. Smith SL, Jones L, de Oliveira LC. Ruecker R, Svihla V, eds. Two steps forward, one step back: Obstacles and progress in conducting research in elementary classrooms. In: Navigating Challenges in Qualitative Educational Research: Research, Interrupted. 2020New York, NY: Routledge;150–160
16. Ringsted C, Hodges B, Scherpbier A. ‘The research compass’: An introduction to research in medical education: AMEE guide no. 56. Med Teach. 2011;33:695–709
17. Varpio L, MacLeod A. Philosophy of science series: Harnessing the multidisciplinary edge effect by exploring paradigms, ontologies, epistemologies, axiologies, and methodologies. Acad Med. 2020;95:686–689
18. Wong G, Greenhalgh T, Westhorp G, Pawson R. Realist methods in medical education research: What are they and what can they contribute? Med Educ. 2012;46:89–96
19. Munoz J, Falco-Gimeno A, Hernandez E. Unexpected event during survey design: Promise and pitfalls for causal inference. Polit Anal. 2020;28:186–206
20. Little RJA, Rubin DB. The analysis of social science data with missing values. Soc Methods Research. 1998;18:292–326
21. Whitcomb M. Responding to disasters: Lessons for academic health science centers. Acad Med. 2007;82:731–732
22. Eva KW. Strange days. Med Educ. 2020;54:492–493
23. Pai M. “Covidisation” of academic research: Opportunities and risks. Nature Research Microbiology Community. https://naturemicrobiologycommunity.nature.com/users/20892-madhukar-pai/posts/65638-covidisation-of-academic-research-opportunities-and-risks. Published April 10, 2020 Accessed April 29, 2020
Copyright © 2020 by the Association of American Medical Colleges