Literature Review and Knowledge Distribution During an Outbreak: A Methodology for Managing Infodemics

Abstract

Problem 

The COVID-19 pandemic has challenged health care systems in an unprecedented way by imposing new demands on health care resources and scientific knowledge. There has also been an exceedingly fast accumulation of new information on this novel virus. As the traditional peer-review process takes time, there is currently a significant gap between the ability to generate new data and the ability to critically evaluate them. This problem of an excess of mixed-quality data, or infodemic, is echoing throughout the scientific community.

Approach 

The authors aimed to help their colleagues at the Rambam Medical Center, Haifa, Israel, manage the COVID-19 infodemic with a methodologic solution: establishing an in-house mechanism for continuous literature review and knowledge distribution (March–April 2020). Their methodology included the following building blocks: a dedicated literature review team, artificial intelligence–based research algorithms, brief written updates in a graphical format, large-scale webinars and online meetings, and a feedback loop.

Outcomes 

During the first month (April 2020), the project produced 21 graphical updates. After consideration of feedback from colleagues and final editing, 13 graphical updates were uploaded to the center’s website; of these, 31% addressed the clinical presentation of the disease and 38% referred to specific treatments. This methodology as well as the graphical updates it generated were adopted by the Israeli Ministry of Health and distributed in a hospital preparation kit.

Next Steps 

The authors believe they have established a novel methodology that can assist in the battle against COVID-19 by making high-quality scientific data more accessible to clinicians. In the future, they expect this methodology to create a favorable uniform standard for evidence-guided health care during infodemics. Further evolution of the methodology may include evaluation of its long-term sustainability and impact on the day-to-day clinical practice and self-confidence of clinicians who treat COVID-19 patients.

Problem

The COVID-19 pandemic has challenged health care systems in an unprecedented way by imposing new demands on health care resources and scientific knowledge. For example, the pandemic has prompted a global rush for data regarding pathogenesis, clinical manifestations, risk factors, mitigation measures, and therapeutic options as well as other aspects of the disease. While knowledge flow and skills distribution are imperative for the clinical management of COVID-19, the exceedingly fast accumulation of new information on this novel virus presents yet another challenge for health care professionals. Given the time-consuming nature of the traditional peer-review process, there is currently a significant gap between the ability to generate new data and the ability to critically evaluate them. As a result, many manuscripts have been published as preprints in different archive platforms, ^1,2 have been circulated on social media, and have sometimes reached news networks before being properly evaluated. From an evidence-based practice point of view, this global health crisis has manifested in a chaotic flood of mixed-quality data, stemming from multiple sources.

Currently, this problem of an excess of mixed-quality data, or as the World Health Organization has called it the COVID-19 infodemic, ^3–5 is echoing throughout the scientific community, with different voices calling for proactive efforts to reduce instances of misinformation. ^3,4,6 During this infodemic, methodological approaches that ensure the generalizability of data have been abandoned in favor of rapid publication of research, some of which is low-quality and can unjustly lend support to specific diagnostic procedures or therapeutics. This, in turn, can lead to the use of evidence-free pseudoscience, such as reiki- and homeopathy-based treatments for COVID-19, ³ and also the speedy endorsement of new knowledge, which may only produce marginal benefits and have unknown adverse implications. ⁷

Approach

In the hospital setting, the infodemic is compounded by the clinical burden of COVID-19. During the pandemic, health care systems have faced increased numbers of patients, intense work in an austere environment, and unique requirements for safety measures. This additional workload may further increase the challenge of acquiring and implementing new clinical knowledge. Therefore, we aimed to help our colleagues at Rambam Medical Center, Haifa, Israel, manage these obstacles with a methodologic solution: establishing an in-house mechanism for continuous literature review and knowledge distribution (March–April 2020).

Our methodology included the following building blocks: a dedicated literature review team, artificial intelligence–based research algorithms, brief written updates in a graphical format, large-scale webinars and online meetings, and a feedback loop (Figure 1).

Figure 1: Flowchart depicting an infodemic and the methodology of an in-house mechanism for continuous literature review and knowledge distribution to help manage the COVID-19 infodemic, The B. Rappaport Faculty of Medicine, Technion, and Rambam Medical Center, Haifa, Israel, March–April 2020. Infodemics are rapid accumulations of data from various sources (i.e., popular and professional search engines) that may lack a proper peer-review process. To deal with this problem during the COVID-19 pandemic, the authors formed a designated literature review team, which screened all new incoming data with the help of artificial intelligence (AI)–based research algorithms. The team then chose the most relevant papers, created graphical updates, and presented the outcomes at large-scale webinars and online meetings. These webinars and meetings also allowed for a completed feedback loop, which guided the team in its future research.

Dedicated literature review team

A dedicated literature review team composed of 9 volunteers—2 senior physicians with experience in clinical research and 7 MD–PhD program researchers from the affiliated faculty of medicine—was established. The team (i.e., the authors of this article) reviewed scientific materials from PubMed, specific journal websites, social media (e.g., Facebook, Twitter), and preprint archives such as medRxiv and ChinaXiv. Then, the team held daily discussions to evaluate the different papers, select the most relevant and reliable data, and derive take-home messages with clinical value for the center’s medical staff. The derived take-home messages were then reviewed by senior physicians from the center’s COVID-19 management team and infectious disease unit.

Artificial intelligence–based research algorithms

Artificial intelligence–based research algorithms were incorporated into the literature review process to help manage the abundance of data and allow for appropriate data filtration. In addition to traditional search engines (e.g., Google, Google Scholar) and inquiries on specific websites (e.g., medRxiv, PubMed), the literature review team used these algorithms as a platform that could integrate all the relevant websites. The algorithms used by the team have been free to everyone throughout the COVID-19 crisis via the Socrates research engine (by Omnisol Information Systems, an Israeli artificial intelligence data science company; https://platform.socrates-insights.com).

This artificial intelligence–based data harvesting research engine simplified the integration of multiple online sources (including social media and preprint archives) and enabled complex literature review designs with an unlimited number of search terms. Additionally, every search term could be further defined for the level of impact it should have on the results: the user could mark a term as obligatory and harvest results that necessarily contained that term, the user could boost a term to increase the visibility of results that contained that term, or the user could give penalties to terms to decrease the visibility of results with that term.

Moreover, the algorithms could consider user feedback, provided via like and dislike buttons, when determining the significance of each result.

Another mode of interaction with Socrates was through its suggested terms. That is, the algorithms could analyze the text of each result, apply language processing to recognize additional possible key terms, and determine the significance of these terms based on their location in the text (e.g., title, abstract), the number of times they appeared, and their distinctiveness (compared with how common they are in the English language). The user could then accept or reject these suggested terms to refine the literature review design or take it in new directions.

Socrates also allowed directed interrogation of selected sources (e.g., Twitter, medRxiv) or genres (e.g., academic papers, news articles, social media posts). This ability proved useful, for example, when looking for valuable responses, opinions, and criticisms posted by clinicians on social media.

Lastly, the algorithms could run continuous searches behind the scenes and notify the team when new results were available. This ability suited the nature of this work and facilitated continuous research throughout the COVID-19 outbreak. This valuable technological support boosted the coverage of different sources of information and accelerated the team’s research.

Brief written updates in a graphical format

Brief written updates in a graphical format were generated by the team from the selected data. These graphical updates were designed to be user-friendly and concise to allow medical staff to acquire focused updates in a short time. The methodological limitations of any presented research were included on the graphical updates.

Large-scale webinars and online meetings

Large-scale webinars and online meetings served as the platform for presenting the graphical updates on a daily basis. The participants in these webinars and meetings ranged from interns to department heads and included representatives from a variety of disciplines. In addition, select final graphical updates (see below) were uploaded to the center’s website, where they could also be accessed by the broader public.

Feedback loop

A feedback loop was established. Questions and remarks from the webinar and meeting participants guided further research by the team. Additionally, an email address was given on the website where select final graphical updates were posted to allow for ongoing queries from clinicians. In this way, future research was better able to reflect the needs of clinical professionals.

Outcomes

This methodology was developed while the COVID-19 epidemic was spreading in Israel and hospitals were preparing themselves to receive contagious COVID-19 patients. Within an initial 4-day period (March 28–31, 2020) the project was launched: the review team was assembled, a graphical update template was created, and the first graphical update was presented at a webinar.

Overall, during the first month of the project (April 2020), many dozens of papers were reviewed by the team, 32 papers were selected and synthesized into 21 graphical updates, and presentations were given at the center’s daily large-scale webinars and online meetings. These numbers reflect the strict screening process the team used that highlighted only a handful of important papers from among the haystack.

Specifically, the screening took place on 2 levels: (1) the individual research conducted by each literature review team member, in which each researcher had the freedom to choose papers as they saw fit, and (2) daily joint discussions in which the senior physicians set the tone and made final inclusion decisions. The leading principles for selecting data emerged rather early in the first joint discussions and favored clinically applicable data (e.g., recommendations for pregnant women), high-profile publicized issues (e.g., use of tocilizumab as a treatment), and criticism regarding key paradigms (e.g., COVID-19 screening policies or the use of hydroxychloroquine as a treatment). Apart from these screening principles, an additional selection criterion considered the requests of webinar and meeting participants (e.g., thromboembolic events in COVID-19 patients).

In a retrospective analysis, every graphical update was assigned to a category based on its topic. Out of all the updates initially prepared by the team in April 2020 (n = 21), 33% addressed the clinical presentation of the disease, 29% referred to specific treatments, 9% presented screening and safety issues, 19% dealt with virologic characteristics of the pathogen, and 10% were more general reviews (e.g., reviewing all the major COVID-19 treatments that were investigated rather than a case for a specific treatment; Figure 2A). Interestingly, after consideration of feedback from colleagues from the webinars and meetings and final editing by the senior physicians on the literature review team, 13 graphical updates were uploaded to the center’s website. Among these final graphical updates, the balance of topics changed, giving extra weight to the clinical presentation, specific treatments, and general review topics collectively (31%, 38%, and 15%, respectively; Figure 2B). Screening and safety issues and virology each accounted for 8% of the graphical updates in this final screening.

Figure 2: Retrospective analysis of the topics addressed by the graphical updates generated by an in-house mechanism for continuous literature review and knowledge distribution to help manage the COVID-19 infodemic, The B. Rappaport Faculty of Medicine, Technion, and Rambam Medical Center, Haifa, Israel, March–April 2020. (Panel A) The percentage of each topic was calculated out of the total number of graphical updates prepared by the team in April 2020 (n = 21). (Panel B) After consideration of feedback from colleagues from the large-scale webinars and online meetings and final editing by the senior physicians on the literature review team, select revised graphical updates were uploaded to the center’s website. The percentage of each topic was then calculated out of the final number of graphical updates uploaded to the website (n = 13). Notably, this process favored the clinical presentation, specific treatments, and general review topics collectively.

This methodology as well as the graphical updates it generated were adopted by the Israeli Ministry of Health and were distributed in a hospital preparation kit as part of the Ministry of Health program for preparing hospitals for the COVID-19 pandemic and its expected secondary waves.

Next Steps

In summary, we believe we have established a novel methodology that can assist in the battle against COVID-19 by making high-quality scientific data more accessible to clinicians. The challenge of identifying relevant and high-quality scientific work has been pushed to the extreme in the current pandemic and resulting infodemic. The combination of a framework of trained research personnel and artificial intelligence–based research algorithms with real-time feedback allowed us to effectively handle the current infodemic.

Importantly, by sustaining the flow of high-quality evidence to clinicians, this methodology also aimed to minimize factors that may limit individuals’ abilities to sift through an excess of information, such as personal workload, research skills, or access to feedback from colleagues. In the future, we expect this methodology to create a favorable uniform standard for evidence-guided health care during infodemics.

The methodology has evolved since it was first conceived, leading to the outcomes described here. We believe that our current framework (as described above) can serve as a model for the future management of infodemics beyond COVID-19. Further evolution of the methodology is likely and may include evaluation of its long-term sustainability and its impact on the day-to-day clinical practice and self-confidence of clinicians who treat COVID-19 patients.

Acknowledgments:

The authors wish to thank Omnisol Information Systems for the permission to use Socrates, the research engine, and for their technical support throughout the COVID-19 crisis.