About the time our article appeared, the New England Journal of Medicine (NEJM) published a multi‐author article on ivermectin’s effects on COVID patients in Brazil. The authors conducted a large‐scale trial known as TOGETHER that looked at both ivermectin and the antidepressant fluvoxamine as possible treatments, and they concluded that ivermectin is not useful against the disease. According to the article, “Treatment with ivermectin did not result in a lower incidence of medical admission to a hospital due to progression of Covid‐19 or of prolonged emergency department observation among outpatients with an early diagnosis of Covid‐19.” Reporting on the article, the New York Times quoted one infectious disease expert who had read the study, Dr. David Boulware of the University of Minnesota, stating, “There’s really no sign of any benefit,” while another, Dr. Paul Sax of Brigham and Women’s Hospital in Boston, said, “At some point it will become a waste of resources to continue studying an unpromising approach.”
Given this negative news, it appeared, ivermectin had reached the end of its COVID road.
However, a careful reading of the NEJM article finds it is not nearly as conclusive and persuasive as the two doctors’ quotes and other media coverage would lead us to believe. In fact, because the results of the TOGETHER Trial suggest that ivermectin actually did benefit the Brazilians in the treatment group — results that are in agreement with 87% of the other clinical trials that have tested ivermectin — there is still good reason to continue studying the drug as a possible preventative or treatment for COVID-19.
Clinical trials and the truth / By the very nature of clinical trials, there is only an indirect linkage between their results and the truth. Ideally, a trial uses a relatively small sample to represent a population — say, a thousand people to represent all of humanity — some of whom receive the treatment under investigation while others do not. Investigators then try to determine if the treatment, or “active,” group has a different outcome than the control group, with the hope that the only difference between the groups is the treatment under investigation and with the further hope that the sample truly is representative of the population.
Running clinical trials on medications is difficult and many things can go wrong. We must scrutinize each trial to see its strengths and weaknesses and then look at the whole body of evidence concerning the possible intervention that is under investigation. Here’s a partial list of factors to consider when evaluating a drug study:
- Was the correct dose given? If not, was the dose too low or too high?
- Was the treatment given at the correct time? Was it given too late in the course of the illness to be effective?
- Was the drug correctly formulated? Was the active ingredient actually active?
- Were the study participants split properly between active and control groups? Were there material differences between the two?
- Was something else happening in the background that might have limited the ability of the study to tease out the results of interest?
- Was the study properly administered or were there errors that could have compromised its integrity?
- Was the study adequately powered — meaning did it include enough test subjects — to detect the intended result? All studies are powered to a certain level, meaning that even if the drug actually works, there is some probability that the study won’t uncover that efficacy.
- Were the investigators potentially biased?
- Did the study truly find a negative result or was it an artifact of how the researchers looked at the data?
With these questions in mind, we offer the following criticisms of the TOGETHER ivermectin trial and resulting report.
Study issues / Many of the outcomes specified in the TOGETHER trial protocol for ivermectin are missing from the final report. The reason for this, in part, is that several mid‐trial protocol changes were made. Trial protocols are typically set before a trial begins and are not subsequently changed. Yet, in the case of the TOGETHER ivermectin study, all‐cause, cardiovascular, and respiratory mortality outcomes were removed, and inclusion/exclusion criteria were changed from including to excluding vaccinated patients.
The TOGETHER team published the fluvoxamine portion of their research in August 2021. It is unclear why the ivermectin results were reported six months later. Was there a problem with the ivermectin data? The authors promised last October to release those data to outside researchers, but that has not yet happened.
The control groups for the two halves of the study (ivermectin versus placebo and fluvoxamine versus placebo) that were conducted almost simultaneously should have had similar characteristics, but they didn’t. That is hard to understand.
Every clinical trial is required to have an independent Data and Safety Monitoring Committee (DSMC). The integrity and independence of the committee are critical. The DSMC for this trial had deep connections to the co‐principal investigator, McMaster University health science professor Edward Mills, and to a key funder of the study, the Bill & Melinda Gates Foundation. For instance, Kristian Thorlund, chair of the DSMC and senior vice president of Cytel, the company analyzing the clinical data, has written over 100 papers with Mills. The two also started a company, MTEK Sciences, together with Jonas Haggstrom, another member of the DSMC. MTEK Sciences provided data analytics for life science companies until it was acquired by Cytel in 2019. Two other members of the DSMC have also published papers with Mills. In noting this, we do not accuse any of these people of acting unethically, but rather note that they do not appear to be impartial.
The placebo used in the trial was not specified in the NEJM article. An earlier trial announcement said it would be a vitamin C pill. Vitamin C has been studied in 42 clinical trials as a treatment for COVID-19, with some indications of efficacy. Obviously, a potentially efficacious substance is not a good placebo.
Also, this clinical trial was powered at 80%. That means there was a 20% chance of a false negative result even if the trial had been conducted flawlessly.
Background issues / Ivermectin treatment of parasitic infection is common in Brazil, and researchers needed to take care that trial participants had not recently used the drug. Yet, recent ivermectin use was not a formal exclusionary criterion for the study. The authors say that such patients were excluded via “extensive screening,” but if prior ivermectin use was not part of the official exclusion criteria for the trial (and it wasn’t), then we don’t know how widespread this screening was and what form it took.
Further, ivermectin is widely available in Brazil as an over‐the‐counter drug — unlike in most clinical trials, where the drug under study is available only via the trial. Prospective participants who wanted ivermectin because they believed they had COVID could have taken it on their own and thus would have been disinclined to enroll in a trial where they faced a 50% chance of getting a placebo. Further, those who wanted ivermectin likely would have had a serious case of COVID, hence their desire for the drug. Therefore, we can assume that the trial participants skewed toward those who considered themselves at low risk from the illness. This conflicts with the stated goal of the trial, which was to study high‐risk patients.
Reporting issues / There are some data inconsistencies in the tables and figures in the NEJM article. In one place, it reports on 288 patients who were studied, but in another it states 228. The article is even inconsistent about the number of patients who died while in the trial.
The subgroup analysis is missing some patient data. For instance, the time since onset of symptoms is missing for 23% of patients. Similar data on patient age are missing. That information is important for good analysis.
The missing data lead to a curious result when the authors compare the outcomes of patients identified as having received early treatment with the outcomes of those identified as having received it later. Both groups did worse than what is shown as the average outcome for treated patients. The only way to explain this result mathematically is if the ivermectin recipients with missing timing data experienced efficacy that was seven times the average — something that is highly unlikely. Many other similar problems are in the analysis.
Trial implementation issues / The randomization of patients in the trial does not match the protocol. This suggests major problems with the study.
One problem is that the patients in the control and ivermectin treatment groups faced different virus variants because the control group was generally treated earlier in the pandemic than the active group. Based on an analysis over time of the patients on placebo, the case fatality rate may have been twice as high during the period when most ivermectin‐receiving patients were enrolled — that is, ivermectin recipients faced a more formidable virus.
Another problem: many of the placebo patients were treated when vaccination was an inclusion criterion (patients may or may not have been vaccinated) while many of the ivermectin patients were treated after vaccinations were considered an exclusion criterion (patients were not vaccinated). In other words, there were material differences between the control and active groups other than the administration of ivermectin.
Blinding / Patients who received a placebo had a treatment duration of one, three, 10, or 14 days, while those who received ivermectin had a treatment duration of three days. This meant that doctors treating patients receiving one, 10, or 14 days of treatment could have figured out that their patients were on a placebo.
Suggesting that did indeed happen, 92% of ivermectin recipients claimed to adhere completely to the dosing regimen, while those on placebo had only 34% or 42% adherence (the NEJM article shows inconsistent numbers). This suggests the clinical trial wasn’t properly blinded.
Treatment timing / Other studies strongly suggest that ivermectin works better when administered early in an infection. The TOGETHER study allowed for and apparently included many patients treated late in their infection. Patients were randomized within seven days but didn’t receive treatment until the next day, meaning that some patients received treatment eight days after symptom onset. Eight days is a very long period for COVID-19. The results of other trials show that the effect of ivermectin drops to about zero at eight days.
Treatment dose / In the TOGETHER trial, ivermectin was administered to patients on an empty stomach, reducing the absorption rate of the drug. That makes the effective dose about 15% to 40% of what current clinical practice suggests. Further, as previously noted, treatment was limited to three days. In addition, the dose of 0.4 milligrams per kilogram of bodyweight was capped for patients weighing more than 90 kg (200 lbs.), meaning that heavier patients got an even lower dose relative to body weight. Half of all patients in the study had a body mass index of 30 or more, suggesting that 30%–50% of patients had their dose capped.
A comparison of the side effects observed in the study should show a greater incidence of diarrhea in the ivermectin group — a known problem with the medicine — but there was a lower incidence of all gastrointestinal disorders among those who supposedly got ivermectin. For comparison, a different trial found 3.6 times the incidence of diarrhea among patients given ivermectin. (“Efficacy of Ivermectin Treatment on Disease Progression Among Adults With Mild to Moderate COVID-19 and Comorbidities,” by Steven Chee Loon Lim et al., JAMA Internal Medicine 182[4]: 426–435 [2022].) The low dose of ivermectin in this study could have contributed to the disappointing findings. If patients hadn’t received a full dose, they could have had a low incidence of diarrhea.
Primary outcome / The primary outcome by which patients’ success was measured and the inclusion criteria for the study were nebulous and subjective. For instance, one inclusion criterion was “patients less likely to need treatment beyond [standard of care] to recover.” One primary outcome criterion was “emergency room visit for >6 hours,” but there was no clarity as to whether this was treatment time or included waiting time.
Potential conflicts of interest / Some of the researchers involved in the TOGETHER trial had performed paid services for Pfizer, Merck, Regeneron, and AstraZeneca, all companies involved in developing COVID-19 therapeutics and vaccines that nominally compete with ivermectin. This does not prove that they were biased, but it does raise the possibility. Again, we point this out not to accuse anyone of unethical behavior, but to note the possibility of unrecognized influence.
Divergence of data results and study conclusions / If a scientist told you that a study showed that ivermectin “did not result in a lower incidence of medical admission to a hospital due to progression of Covid‐19 or of prolonged emergency department observation,” you would expect that result to show up in the data analysis. Yet, the TOGETHER study found that ivermectin was associated with a 12% lower risk of death, a 23% lower risk of mechanical ventilation, a 17% lower risk of hospitalization, and a 10% lower risk of extended ER observation or hospitalization. So what gives?
This underscores the discussion in our earlier article about statistical significance. If the confidence level of the results does not eclipse a stipulated threshold, it is often said that the treatment did not work. However, in this case, the results suggest that the drug did work, but the results weren’t as definitive as the researchers might have wanted. A more accurate interpretation of the findings would be to say that the drug showed promise and that a larger trial may yield the desired statistical significance.
Based on our analysis of the published study results, we have estimated the probability that ivermectin helped patients in the TOGETHER trial. The results are shown in Table 1. To compute these probabilities, we used the point estimates and the 95% Bayesian Credible Intervals from the NEJM article’s Table 3. (To better understand our methodology, see “Metalog Distributions,” by Tom Kreelin, www.metalogdistributions.org.) Based on our results, it is difficult to agree with the conclusion that the TOGETHER trial showed “no sign of any benefit” for ivermectin.
