PLEASE NOTE: The Viewpoints on our website are to be read and freely shared by all. If they are republished, the following text should be used: “This Viewpoint was originally published on the REVIVE website revive.gardp.org, an activity of the Global Antibiotic Research & Development Partnership (GARDP).”
The development of vaccines for COVID-19 has been a magnificent achievement. At least five of these vaccines have been trialled in various European countries or North America. Table 1 summarises the vaccines and their characteristics. For further reading, I recommend Comparing the COVID-19 Vaccines: How Are They Different? by Kathy Katella1. In this article, I shall discuss various statistical aspects of five large phase 3 trials used to support vaccine registration in order to learn what key aspects of their success could inform other fields such as antimicrobial R&D.
Table 1 Summary of results from five large vaccine studies.
The most important thing to note from Table 1 is that four of the sponsors used a 1:1 allocation for vaccine to placebo. The exception is AstraZeneca/Oxford who used 2:1. It should also be noted that the J&J Janssen vaccine is administered as one dose; the others are administered as two doses.
The morals are clear. Be very cautious in accepting claims from observational studies. This applies not only to studies of vaccines for COVID-19 but more widely for studies of treatments of infections generally and, indeed, for all studies of treatment efficacy.
The efficacy of vaccines for COVID-19
Figure 1 is a graphical display showing estimated vaccine efficacy and the associated 95% confidence intervals for the five trials. Vaccine efficacy is calculated as a relative risk reduction as follows:
Here rp is the risk of catching COVID-19 if receiving the placebo and the corresponding risk of receiving the vaccine. The numerators to calculate these risks will be the numbers of cases receiving the placebo and vaccine, respectively, and the denominators will be ‘exposure’, either measured in terms of number of persons or person-years.
Figure 1 Vaccine efficacy for five vaccines based on the corresponding phase 3 trials. Total numbers of cases/subjects are indicated.
The sponsoring vaccine manufacturers may have made different choices for the denominator and will have adjusted for different covariates. In one case, the Pfizer/BioNTech vaccine, the analysis was Bayesian7, but in the other cases, it was frequentist. However, my analysis, which was frequentist, made use neither of any denominator nor of covariates. I simply considered what might be called the risk proportion, let us represent this by the risk of getting COVID-19 when given a vaccine as a proportion of the sum of the risks when given a vaccine or placebo. So, we have:
I then calculated exact confidence intervals for this proportion using cases and allocation ratio only and converted the results to the vaccine efficacy scale.
What is remarkable about this is how close my simple calculations are to the analyses carried out by the sponsors. There are various technical reasons for this, in particular, that a major source of uncertainty is the number of cases. Where numbers are small, so that the total numbers of subjects, which are large are unimportant in comparison as regards contribution to uncertainty. However, it is also a tribute to the value of concurrent control in these well-designed clinical trials.
Figure 2 shows the importance of concurrent control. This shows the estimated infection rate (the ratio of cases to subjects) when individuals are given a placebo and associated confidence intervals calculated in two different ways: using the binomial distribution, which I prefer because it is ‘exact’, and using the normal approximation, which lends itself to a formal summary of results using a so-called ‘meta-analysis’. The placebo risk seems to be very different from study to study and this is confirmed by the results of a meta-analysis.
What can be learnt from these trials?
The meta-analysis shows considerable evidence of heterogeneity, that is to say, the results vary more from trial to trial than can be explained by simple random variation. One plausible explanation is that infection rates have varied considerably from place to place and at different times. In consequence, the confidence interval for the random effects analysis is much wider than for the fixed-effects meta-analysis. The result is that if we were forced to use historical figures for judging the efficacy of a new vaccine, rather than concurrent controls, the estimate of efficacy and the estimate of the uncertainty of efficacy would be very unreliable. One might argue that this is to raise an unnecessary worry: no responsible sponsor would rely on controls of this sort and no regulator would accept this approach.
Figure 2 Placebo infection rates for the five studies together with the result of a fixed and random effect meta-analysis.
However, for other matters, we may have no choice. Consider the efficacy of vaccines against new variants of COVID-19. There we may have to use observational studies rather than randomised clinical trials. Figure 2 shows the problem: controls from observational studies are plausibly unreliable in the same way that historical controls would be for a trial. If we calculate the standard errors and the confidence intervals using the statistical approach used for Figure 1, then we are likely to underestimate considerable the uncertainty involved.
What is remarkable about this is how close my simple calculations are to the analyses carried out by the sponsors. There are various technical reasons for this, in particular, that a major source of uncertainty is the number of cases.
The morals are clear. Be very cautious in accepting claims from observational studies. This applies not only to studies of vaccines for COVID-19 but more widely for studies of treatments of infections generally and, indeed, for all studies of treatment efficacy. Observational studies may be relevant for answering some questions when studying antimicrobials and may be unavoidable on occasion, but the statistical methods used to evaluate them run the risk of exaggerating their precision. One should also be wary of using historical controls for antimicrobial trials. Randomised trials using concurrent controls are valuable.