Systematic reviews and meta-analyses play an invaluable role in the practice of evidence-based medicine. Unfortunately, the process is time-consuming, on average requiring 67 weeks to sift through all available literature, collate relevant data, and analyze results to form conclusions. However, recent advances in natural language processing (NLP) and machine learning have enabled “artificial intelligence” (AI) to “learn” through algorithms and assist with text classification and data extraction. Semi-automation, with “human-in-the-loop” systems, can potentially assist with several labor-intensive steps of the systematic review process and make it faster. Nevertheless, skepticism as to the accuracy of automated tools exists which presents a barrier to their widespread acceptance.
Two independent investigators conducted a systematic database search of PubMed and ClinicalTrials.gov. SK conducted the search manually and SS performed the search using an AI. All the tools so used were developed in-house using hypertext preprocessor (PHP) language. The different steps so used are shown in Table 1. The difference between the manual workflow and NLP-assisted workflow is shown in Table 2. The time taken for the search and data extraction was recorded. The machines used a mix of NLP and automation. By automation, the AI screened articles and put extracts of relevant articles in their database in a convenient format, for later use. NLP then used “bags-of-words” technique to extract the relevant lines that captured our curated keywords (statistical/genomic/metabolomics). The extracted data were then entered into Microsoft Excel (2010) after which SS filtered the relevant papers. A similar technique using NLP helped analyze the full-text papers.
We included trials that studied the efficacy of apremilast in hidradenitis suppurativa published in English, from database inception till January 2021. The process of article selection is detailed in Figure 1.
We found that the papers were selected and conclusions reached were the same by the semi-automated and completely manual methods. The time taken both for the article selection and data extraction was lower for the search conducted with AI assistance [Figure 1]. A little more than half the patients (54.2%; 19/35) treated with 30 mg twice daily of apremilast achieved ≥50% reduction in Hidradenitis Suppurativa Clinical Response (HiSCR50) from baseline at 16 weeks compared with none in the placebo group. [Table 3]
Recognition of the potential for AI to simplify and expedite the systematic review process led to the formation of the International Collaboration for Automation of Systematic Reviews. In this review, we found that the use of automation drastically reduced the total time used to process available literature. This will be critical in larger systematic review that retrieves large number of articles for screening. It also eliminates time lost due to unplanned disturbances and fatigue that inevitably creeps in after perusing a large amount of literature. Machine-assisted processing minimizes mundane tasks, such as extracting several sentences manually for review by peers. This leaves us free to work on critical tasks.
Through this preliminary and small-scale systematic review, we assessed the utility of semi-automation and NLP for systematic review. Our study was limited by the fact that we performed this systematic review for a topic which yielded only 15 articles. Other than the advantage of time, we were unable to find any other significant difference between the two methods. Further large-scale comparative systematic reviews are needed to assess machine accuracy and gain more confidence in using machines.
Financial support and sponsorship
Conflicts of interest
There are no conflicts of interest.
1. Beller E, Clark J, Tsafnat G, Adams C, Diehl H, Lund H, et al Making progress with the automation of systematic reviews: Principles of the International collaboration for the automation of systematic reviews (ICASR) Syst Rev. 2018;7:77
2. Borah R, Brown AW, Capers PL, Kaiser KA. Analysis of the time and workers needed to conduct systematic reviews of medical interventions using data from the PROSPERO registry BMJ Open. 2017;7:e012545
3. Marshall IJ, Wallace BC. Toward systematic review automation: A practical guide to using machine learning tools in research synthesis Syst Rev. 2019;8:163
4. Kerdel FR, Azevedo FA, Kerdel Don C, Don FA, Fabbrocini G, Kerdel FA. Apremilast for the treatment of mild-to-moderate hidradenitis suppurativa in a prospective, open-label, phase 2 study J Drugs Dermatol. 2019;18:170–6
5. Vossen ARJV, van Doorn MBA, van der Zee HH, Prens EP. Apremilast for moderate hidradenitis suppurativa: Results of a randomized controlled trial J Am Acad Dermatol. 2019;80:80–8