Meta-Analyses Do Not Establish Improved Mortality With Ivermectin Use in COVID-19

Ivermectin has been identified as an inexpensive, readily available drug with the potential to be repurposed as a treatment for COVID-19, especially in countries with limited access to vaccines. Although multiple studies have been published in an attempt to evaluate its usefulness in COVID-19, many are small and not constructed appropriately to detect differences in important clinical outcomes (ie, death). For this reason, researchers have turned to meta-analyses to combine study results and draw summary conclusions regarding ivermectin's effectiveness. Two such meta-analyses recently published in the American Journal of Therapeutics concluded that ivermectin decreased mortality and improved other surrogate end points in COVID-19.^1–4 A recently withdrawn article caused both authors to rework their meta-analyses without altering their main conclusions.^1–5 We feel that shortcomings within both sets of meta-analyses and limitations in the component studies are significant enough to invalidate their main finding that ivermectin reduces mortality. A review of other meta-analyses on the same subject, containing many of the same individual studies, were similarly limited by poor design.

KORY AND MARIK META-ANALYSES

In their updated meta-analysis, the authors missed an opportunity to improve methodologic weaknesses within their original study.^1,2 In both the original and follow-up meta-analyses, the authors did not list standard methodological items (eg, Preferred Reporting Items for Systematic Reviews-Metanalysis) and had no description of statistical techniques.⁶ Absence of these descriptors makes meta-analyses nonreproducible and nonreplicable, thereby limiting external validity.⁷ Their absence automatically classifies this meta-analysis as having critically low quality.^8,9

The authors implied their follow-up was performed to exclude a retracted study. However, without explanation, the authors deleted a second study while adding 2 others.¹⁰ One newly added study was unblinded and a second enrolled patients with negative PCR tests and did not control for underling comorbid risks.^11,12 In a larger omission, the authors did not include 3 double-blind placebo-controlled trials showing no effect on mortality—biasing results in favor of ivermectin.^13–15

In their follow-up, the authors did not correct errors from the original meta-analysis that overstated ivermectin's mortality benefit. In their description of the study by Ravikirti et al, mortality was described as “improved” with ivermectin compared with controls “0% versus 6.9%, P = 0.019.”^1,16 However, there was no statistical difference in mortality (0/55 vs. 4/57, Fisher exact P value = 0.12). A mortality benefit with ivermectin in severely ill patients was stated to be “of borderline statistical significance, 0% (0/11) versus 27.3% (6/22), P = 0.052” in the description of Hashim's study.^1,17 This was incorrect, as the Fisher exact test, P value is 0.08. The phrases “borderline significant,” “approached” and “nearly statistically significant” used in this meta-analysis have been described as inappropriate and misleading as “results do not include movement and cannot approach significance because of the dichotomous definition” of statistically significant.^1,18,19

Inclusion of unblinded, single-blinded, and nonrandomized studies limited the authors' ability to calculate reliable, reproducible, pooled estimates of treatment effects. The largest study within the reworked meta-analysis comprising 39% of all cases and 48% of all deaths was a chart review where only 3.5% of patients were given ivermectin.²⁰ Patients also received multiple other treatments in an uncontrolled manner (eg, steroids, tocilizumab) with no comparison of disease severity or comorbidity between the groups. In the study by Khan et al, nonrandomization resulted in a sicker control population (46% control vs. 10% study oxygen requirement).²¹ Nonrandomization led to different treatment (39.8% dexamethasone in ivermectin vs. 19.6% in control group) in the retrospective study by Rajter et al.²² The authors inappropriately adjusted for baseline risk between the groups by constructing a prediction model using binary logistic regression that overfitted their data using 13 variables to create a model that predicted 53 outcomes.²² The ratio of studied prediction variables to outcome should be, at most, 1:10 or 1:20.²³

In other studies, described as double-blind, randomized controlled trials (RCTs), study and control groups differed. In the study by Niaee et al,²⁴ control patients had fewer positive PCR tests, lower oxygen saturations, higher body mass indices (BMI), and more initial CT scans compared with ivermectin patients. The authors miscalculated P values when comparing PCR positivity and initial CT performance between the groups. The values should be <0.001 for both comparisons and not 0.421, 0.527 as reported.²⁴ Vital sign data appear to be in error as a diastolic blood pressure median and interquartile range of 80 mm Hg (80–80) was reported in 4 of 6 groups in the study. It is unlikely that half of all patients (60/120 patients) in the first 4 groups had the exact same diastolic blood pressure. In the study by Mahmud et al, 32 cases were lost to follow-up (LTFU).²⁵ Study validity is affected when LTFU is >5% (9% in this study), LTFUs exceed the outcome (32 LTFUs vs. 3 deaths) or a worst-case scenario imputation alters results (ie, all study LTFU and no control LTFUs have adverse outcome).^26–28 This imputation would result in 8.2% mortality in ivermectin cases versus 1.7% in controls (P < 0.01) and render the overall mortality difference between the groups nonsignificant [odds ratio, 0.48; 95% confidence interval (CI) 0.21–1.07, RevMan 5.4; MedCalc 19.7, Osteen Belgium] within this meta-analysis.

BRYANT META-ANALYSES

Bryant's meta-analyses corrected many methodological shortcomings within the first set of meta-analyses by including many Preferred Reporting Items for Systematic Reviews-Metanalysis items. The authors, however, did not appear to register their meta-analysis, a critical item that categorizes a meta-analysis as low quality.^8,9 Study registration improves transparency and minimizes the risk of selective outcome reporting bias. Selection bias may have been introduced when authors of 2 previous pro-ivermectin (or pro ivermectin) meta-analyses were contacted to aid in identifying additional articles for this analysis. The authors did not list inclusion or exclusion criteria within their methods only stating that they searched for RCTs. Their final included articles were described as double blind, open label, and quasi-RCTs. Some experts believe quasiexperimental studies are not RCTs, thus, requiring exclusion because true randomization is absent.^29,30

Many studies were undersized with no sample size calculations for any outcome in 6 studies with other studies calculating samples sizes to detect differences in surrogate end points: clinical recovery (n = 5), viral clearance (2), and CIs around combined group mortality (1).^3,4 No study was constructed to detect mortality differences between ivermectin and control groups. Eleven studies were small (<100 patients) increasing the likelihood that findings are false and inflating the effects of treatments.^31,32 Further misinterpretation of results can occur in undersized studies that make multiple comparisons without any statistical corrections. For example, Okumus et al and Shahbaznejad et al conducted 69 and 58 statistical comparisons in studies with only 69 and 60 total patients, respectively.^11,12

Nine of 14 studies were described as having adequate random sequence generation and allocation concealment.³ Failed randomization/allocation occurred in 2 of these “adequate” studies with a different viral cycle threshold plus different polymerase chain reaction (PCR) positivity, BMI, and oxygen saturation between ivermectin and control groups.^24,33 In 5 studies described as not having adequate allocation, one had different baseline SOFA scores and 4 did not analyze comorbidity or other treatments between the groups.^{11,13,14,17,34} These confounding biases can alter the results of individual studies, thus, altering results of the meta-analysis.

The authors describe 6 of 14 studies as having an unclear or high risk for inadequate blinding.³ They state blinding is “less important” for evaluating evidence related to mortality.³ We disagree. Inadequate blinding can alter treatments, aggressiveness of care, and other actions taken to manage patients potentially resulting in different outcomes. Five studies lacked placebo controls,^{11,12,17,33,35} 4 contained nonplacebo controls (ritonavir/lopinavir, doxycycline, chloroquine, hydroxychloroquine),^12,13,33,35 and 3 added doxycycline to ivermectin treatment groups.^17,25,36 Studies with active or nonplacebo controls, or that add drugs to treatment groups, do not allow ivermectin's effects to be isolated from those of the other drugs. Moreover, it is possible that active controls might worsen outcomes, falsely skewing results toward ivermectin's effectiveness.

Several other aspects of studies within each set of meta-analysis limits the ability to draw summary conclusions. Variable ivermectin doses, duration of ivermectin use, admission status, variable testing or no COVID-19 testing, and timing of drug administration during the disease course obscure the interpretation of results. Mixing studies with varying disease severity definitions further limits the ability to make conclusions. Moreover, inclusion of 2 studies enrolling children was inappropriate because mortality differs dramatically between adults and children.^12,17

In their original meta-analysis, Bryant et al³ performed sensitivity analyses to assess robustness of their results by performing trial sequential analysis to confirm that ivermectin reduced mortality. Reanalysis of the study by Niaee et al showed that it should have been recategorized as having a high risk of bias and excluded during sensitivity analysis.²⁴ This study had data and statistical errors previously described and included PCR-negative patients with different between-group PCR positivity, BMI, oxygen saturation, and initial CT performance, indicating randomization failure. In our opinion, these issues require removal of this article from any meta-analysis. Importantly, this was the only article within the Bryant meta-analysis showing that ivermectin reduced mortality.⁴ Removal of this single article alters their results so that no significant difference now exists between ivermectin and control mortality (risk ratio, 0.73; 95% CI, 0.45–1.18).

OTHER META-ANALYSES

We performed a literature search (EMBASE, PubMed, Web of Science, bioRxiv, medRxiv, Google Scholar, January 12,020 to October 15, 2021) and found 18 additional meta-analyses (now 20 total) that evaluated ivermectin's effect on COVID-19 mortality (Table 1).^37–54 There was substantial overlap of studies within meta-analyses. Fourteen of 18 had the majority of their included studies contained within the Bryant or Marik meta-analyses (Table 1). Eleven concluded that ivermectin prevented mortality (Table 1). Removing the withdrawn study by Elgazzar et al left 9 and removing the study by Niaee et al left only 5 meta-analyses concluding that ivermectin is effective (Table 1). These 5 include Marik's meta-analysis (previously critiqued), meta-analyses by Kow and Nardlli whose individual studies overlap completely with Marik's and Bryant's meta-analyses, a meta-analysis by Karale et al that found no mortality difference in subsets with RCTs or without active controls, and a 3-study meta-analysis by Padhy et al with the quality of evidence, Cochrane GRADE, rated as very low.^{2,4,5,24,44,46–48}

Table 1. - Meta-analyses evaluating Ivermectin's association with mortality in patients with COVID-19.

Author* publication date	Mortality studies (without Elgazzar)	Number (%) overlap with revised Bryant†	Number (%) overlap with revised Marik†	Ivermectin and overall mortality‡	Ivermectin and mortality excluding Elgazzar§	Ivermectin and mortality excluding Elgazzar/Niaee§	AMSTAR 2 meta-analysis quality¶
Siemeniuk51 July 30, 2020 Living meta-analysis, updates periodically	8 (7)	7/7 (100%)	3/7 (43%)	Network relative estimate 0.31 (0.14 to 0.72)	RR, 0.33 (0.14 to 0.77)	RR, 0.54 (0.21 to 1.36)	Mod/high
Padhy48 November 23, 2020	3	0	2/3 (67%)	OR, 0.53 (0.29 to 0.96)	NA	NA	Critically low
Kim45 December 30, 2020	2	0	2/2 (100%)	Moderate/severe OR, 0.76 (0.25 to 2.33) Critical (one study) OR, 0.15 (0.04 to 0.57)	NA	NA	Mod/high
Castenada-Sabogal#,37 January 27, 2021	4	0	3 (75%)	RR, 0.7 (0.31 to 2.28)	NA	NA	Critically low
Kow46 March 29, 2021	6 (5)	5/5 (100%)	5/5 (100%)	OR, 0.21 (0.11 to 0.42)	OR, 0.27 (0.14 to 0.55)	OR, 0.37 (0.16 to 0.89)	Critically low
Nardelli47 May 8, 2021	7 (6)	6/6 (100%)	5/6 (83%)	OR, 0.19 (0.1 to 0.34)	OR, 0.27 (0.14 to 0.55)	OR, 0.37 (0.16 to 0.89)	Critically low
Hariyanto41 June 6, 2021	8 (7)	7/7 (100%)	5/7 (71%)	RR, 0.31 (0.15 to 0.62)**	RR, 0.42 (0.24 to 0.74)	RR, 0.55 (0.3 to 1)	Mod/high
Zein53 June 27, 2021	9 (8)	7/8 (88%)	5/8 (63%)	RR, 0.39 (0.2 to 0.74)	RR, 0.56 (0.34 to 0.93)	RR, 0.6 (0.34 to 1.05)	Low
Roman50 June 28, 2021	5	5/5 (100%)	2/5 (40%)	RR, 0.37 (0.12 to 1.13)	NA	RR, 0.62 (0.24 to 1.65)	Critically low
Hill42 July 6, 2021	11 (10)	9/10 (90%)	6/10 (60%)	RR, 0.44 (0.25 to 0.77)	RR, 0.57 (0.35 to 0.95)	RR, 0.77 (0.51 to 1.16)	Critically low
Popp49 Jul 28, 2021	4	4/4 (100%)	1/4 (25%)	Moderate–severe RR, 0.6 (0.14 to 2.51) mild to RR, 0.33 (0.01 to 8.05)	NA	NA	Mod/high
Izcovich43 August 21, 2021	12 (11)	9/11 (82%)	6/11 (55%)	RR, 0.5 (0.28 to 0.88); RR, 0.96†† (0.58 to 1.96)	RR, 0.64 (0.4 to 1.05)	RR, 0.84 (0.57 to 1.24)	Mod/high
Bryant4 August 27, 2021	14	14/14 (100%)	6/14 (43%)	RR, 0.51 (0.27 to 0.95)		RR, 0.73 (0.45 to 1.18)	Low
Marik2 September 2, 2021	10	6/10 (60%)	10/10 (100%)	OR, 0.39 (0.25 to 0.6)	NA	OR, 0.44 (0.28 to 0.71)	Critically low
Deng40 September 2, 2021	13	9/13 (69%)	4/13 (31%)	OR 0.77 (0.5 to 1.19)	NA	NA	Low
Cruciani39 September 8, 2021	8 (7)	7/7 (100%)	4/7 (57%)	RD to 0.02 (−0.05 to 0.01)‖	RD −0.01 (−0.04 to 0.01)	NA	Low
Cheng38 September 16, 2021	3	2/3 (67%)	1/3 (33%)	RR, 0.53 (0.11 to 2.61)	NA	NA	Critically low
Karale44 September 17, 2021	29	11/29 (38%)	10/29 (34%)	OR, 0.54 (0.34 to 0.86)	NA	OR, 0.58 (0.36 to 0.92)	Critically low
Zhang‡‡,54 September 28, 2021	10	6/10 (60%)	4/10 (40%)	OR, 0.61 (0.37 to 1)	NA	OR, 0.87 (0.51 to 1.49)	Critically low
Singh52 October 10, 2021	2	2/2 (100%)	2/2 (100%)	OR, 0.45 (0.17 to 0.18)	NA	NA	Low

^*Dates may be inaccurate when publication on preprint server changes to online or print journal publication. Within meta-analyses, listed first authors for included individual studies may have differed between preprint and final published articles (eg, Cepelowicz = Rajter, Fonseca = Bermijo Galan = Galan, Shahbazneiad = Rezai, Ravikirti = Kirti, Beltran-Gonzales = Gonzales).

^†Number (%) of individual studies within these meta-analyses that are also contained within Marik and Bryant revised meta-analyses.

^‡RD, RR, OR with 95% CIs in parenthesis.

^§Recalculated RD, OR, RR with deletion of Elgazzar study and with deletion of both Elgazzar/Niaee studies,^5,24 Multiple meta-analyses have different total mortality and total cases for the same individual studies. When available, the mortality and total numbers used by each meta-analysis were used for each re-calculation. NA within columns indicatesnot applicable because Elgazzar or Niaee article is not contained within these meta-analyses. RevMan 5.4 and MedCalc 19.7, osteen Belgium were used to calculate values repeating the same statistical technique used in each study.

^¶Two reviewers (by consensus, disagreements to be settled by third reviewer–unnecessary) independently assessed each meta-analysis for the presence or absence of 7 critical items to classify its quality as critically low, low, or moderate/high.^8,9 If all critical items are present, the meta-analysis is moderate or high quality. The absence of one item categorizes a meta-analysis as low quality and 2 or more items categorize an article as having critically low quality. (kappa = 0.72, 95% confidence interval, 0.49–0.94 for initial 2 reviewer agreement).

^‖Cruciani left off 4 deaths in control group, adding these back in changes initial RD to −0.03 (−0.06 to 0.01).

^#These authors treated the same article by Cepelowicz-Rajter [preprint (Cepelowicz) and final published article (Rajter)] as 2 separate studies when they were the same study. Thus, there were only 4 total studies and their initial calculations appear incorrect. Deleting the preprint version of the article alters their calculated OR to 0.68 (0.24–1.97).

^**The authors mislabeled and transposed their mortality and PCR Forest plots within their article. Figure 2 (a) and 2 (b) descriptions are transposed.

^††Excluding studies with a high risk of bias.

^‡‡The 10 included articles within this ivermectin mortality analysis were identified by comparing the ivermectin studies within Table S3 (Detailed Trial Characteristics) with those within Table S5 [Evaluation of risk of bias (mortality)].

OR, odds ratio; RD, risk difference; RR, risk ratio (relative risk).

A quality appraisal of all 20 meta-analyses using the AMSTAR-2 tool found that 15 had low or critically low quality.^8,9 The 5 meta-analyses concluding that ivermectin was effective after article exclusions all had critically low quality^2,44,46–48 (Table 1). Others have noted similar low quality in a large number of meta-analyses and recommended remediation by following best-practice guidelines for meta-analyses and using anonymized individual patient data (IPD) obtained directly from study authors.^55–57 Using IPD would increase transparency, improve data quality, offset inadequate reporting within studies, allow for better subgroup analysis, allow for standardizing common measures, and avoid inclusion of studies with a high risk of bias.

In-depth assessment of “other” meta-analyses indicates that many were not transparent, had inadequate inclusion/exclusion criteria, and lacked proper risk of bias assessments, across study quality assessments and sensitivity analyses. The same failures occurred in many of these meta-analyses. The combination of including flawed studies and using inadequate meta-analysis techniques rendered many of their conclusions unreliable and useless for clinical decision making.

Unfortunately, inadequate and improperly conducted studies of this drug have added to confusion both inside and outside of the medical community. In an August 26, 2021 health advisory, the Centers for Disease Control and Prevention reported a 24-fold increase in ivermectin prescriptions per week since the prepandemic period, indicating that a substantial number of physicians are prescribing this drug for COVID-19.⁵⁸ In July 2021, the Centers for Disease Control and Prevention reported a 5-fold increase in calls to US poison control centers related to ivermectin with increased emergency department visits for adverse effects.⁵⁸ A more recent report from Oregon detailed 21 poison center calls with patients developing seizures, hypotension, confusion, and ataxia and 4 requiring intensive care unit admission after ivermectin ingestion. Seventeen of 21 patients had purchased veterinary formulations, only 3 had received prescriptions and 1 had an unknown source for their drug.⁵⁹ These reports indicate that despite its absence from NIH COVID-19 treatment guidelines and lack of United States Food and Drug Administration approval, off-label use of ivermectin by lay people and physicians has continued throughout the pandemic.⁵⁸ It is possible that inconclusive ivermectin studies and meta-analyses contributed to this activity.

NEXT STEPS

Ultimately, imperfect studies need to be replaced by larger, adequately powered, double-blind, placebo-controlled trials, with meta-analyses, ideally using IPD, based on these studies. Assuming an overall COVID-19 case fatality rate of 2%, a study with 4638 patients (2319 each arm) would be needed to have 80% power (alpha 0.05, 1:1 enrollment) to detect a 1% or 50% relative drop in mortality from any intervention (eg, ivermectin).⁶⁰ Assuming a sicker population (7% mortality, Bryant meta-analysis), a study with 1274 patients (637 each study arm) would be needed to have 80% power to detect a 3.5% (50% relative) drop in mortality. We await results of such studies before concluding that treatment with ivermectin can decrease mortality in all or a subset of patients with COVID-19.