#9: The power of challenge trials

This week: How to set the scene for a Zika virus vaccine. Plus a lot of new links and podcasts.

This is my ninth post of Scientific Discovery, a weekly newsletter where I’ll share great new scientific research that you may have missed. Check out the About page if you’re interested in why I’m writing this.

You may have heard of ‘challenge trials’, which are experiments where people are deliberately infected to test the efficacy of a new drug or vaccine.

In this week’s post, I’ll explain how they work, and why they might be our best hope to develop vaccines against Zika virus.

---

How to test a Zika virus vaccine

Let’s start at the beginning. How do regular vaccine trials work?

In a typical trial, participants are given a vaccine or a placebo. Then they’re followed up over time to see how many get infected naturally and develop the disease in each group. 

But these trials can face a major obstacle: if only a fraction of the people in the trial develop the disease at all, even in the placebo group, then it becomes difficult to see whether the vaccine had a benefit. So trials like this can take several years, thousands of participants, and an enormous amount of resources before enough people in the trial get infected, to see a result.

That’s the case for Zika virus disease.

The virus causes no symptoms in most people it infects. Instead, the main risks occur if it's caught early in pregnancy – when it can cross the placenta and attack the growing foetus, suffocating its blood flow. In 5–14% of infections, it attacks the growing brain, leading to severe birth defects. This is called “congenital Zika syndrome.”

If Zika virus is caught during pregnancy, what are its risks to the growing foetus? This diagram comes from this great overview by Musso et al. (2019)

The procedure to test a single vaccine against this syndrome can require recruiting tens of thousands of participants (potentially more than a hundred thousand) and each trial can take several years.

This is because: (1) only a fraction of participants become pregnant during a short period of time, (2) the chances of catching an infection during pregnancy are low, and (3) most foetuses are (fortunately) asymptomatic. Together, this makes it difficult to run trials even in places that already have an outbreak.

There are other alternatives – or at least there were. Instead of testing the vaccine’s efficacy against congenital Zika syndrome, you could test its efficacy against any detectable infection – by testing people’s blood, saliva and urine frequently, to see if the virus was present in them. That way, it’s likely they’d avoid passing the virus on to the foetus.

These simple differences – which outcome is tested in a clinical trial – can make a huge difference to how many participants are needed to see a result.¹ You can see this in the chart below.

These charts show how many participants would need to be recruited into a trial to reliably see the effect of a Zika virus vaccine. The number needed depends heavily on which outcome is tested. If you wanted to test the protection a vaccine delivers against congenital Zika syndrome, you’d need to recruit tens of thousands of participants (to reach the threshold of 80% statistical power). Possibly even hundreds of thousands. But if you wanted to test its protection against an infection regardless of its severity – and you tested participants weekly – then you’d need fewer than 10,000 participants. The number of participants you’d need would also be lower if you wanted to see if the vaccine was very effective, not just moderately effective. This great chart comes from this great simulation study by Mercaldo and Bellan (2019).

But there’s a problem with this approach too, because things have changed over time.

In the past, Zika virus was spread by mosquitoes that tended to affect other non-human primates, and caused only small sporadic outbreaks in humans. This carried on until the 2000s, when it sprung up in tropical countries around the world: the virus had mutated into a new strain with the ability to infect mosquitoes that tended to bite humans.

This chart shows how the Zika virus outbreak spread across tropical countries over time. Source: Musso et al. (2019)

The new outbreaks that sprung up around the world were large. Within a short period, the virus had infected so many people in the affected regions that those populations built up enough immunity to slow down its spread. Now, though, the disease is in a trough.

This chart shows the number of reported cases from Zika virus over time, and the number of countries that have reported cases over time. Source: Musso et al. (2019)

Unfortunately, herd immunity doesn’t mean a virus disappears. Viruses can linger in the population at low levels, and hang around in people with chronic infections.

As time goes by, they can find new pockets of people to infect, because of migration, new births, waning immunity, virus evolution, changes in the habitat of mosquitoes, and so on. This is why we tend to see new waves of disease after infections have been seeded around the world.

And while a reduction in the prevalence of Zika virus is great news for the burden of the disease now, it means testing new vaccines or treatments has become basically infeasible: you would have to enroll an enormous number of people into a trial and/or wait a very long time just to test a single vaccine. This makes it very difficult to prepare for another outbreak.

So how would we prepare for a new wave?

That’s where challenge trials come in. While a regular trial means waiting and watching for people to get infected on their own, a challenge trial involves doing it deliberately and seeing the results in a small number of people in a short amount of time. It’s crucial that this would only be for volunteers who are willing and informed.

Experts now believe these trials will be very important for testing new Zika vaccines.

Obviously, it would be very unsafe for these challenge trials to recruit pregnant women and infect them; that’s not how they would be designed. Instead, they could recruit volunteers who aren't planning to conceive and give them highly-effective contraception to take during the study, alongside the vaccine or placebo. This would avoid the serious risks of Zika virus. Then, the volunteers would be deliberately exposed and monitored closely to see if they could quickly eliminate the virus from their bodies.

Just pause to think about how useful this could be: a trial could test the efficacy of a vaccine in just a short amount of time, with only a small number of volunteers. This is incredibly useful if there are a large number of candidate vaccines, because it helps to weed out those that don’t work and spot those that do.² It could save a lot of time, and that could mean a lot of lives. And it’s now likely the only feasible way to test Zika vaccines directly.

Challenge trials aren’t only helpful when outbreaks have subsided. Even in usual circumstances, they can have great power. Literally – they tend to have much greater statistical power.

‘Statistical power’ is a concept with this definition: the probability that a test correctly rejects the null hypothesis. Here, you can think of it as the chances that the results of a study will correctly show a benefit of the vaccine, when the vaccine actually has a benefit.³

In a challenge trial, where all the participants are exposed to the virus, a large share of participants in the placebo group would become infected. And since they can be monitored more closely and frequently, it would be easier to detect most or all of the infections that occur.

So, while a typical trial might need many thousands of participants to see the difference between the vaccine group and the placebo group, a challenge trial tends to need only around a hundred.

I’ll elaborate on that because it’s often misunderstood. Statistical power isn’t just about sample size (e.g. the number of participants in a study). It’s also about:

• how frequent the outcome is

• how common the risk factor or treatment is⁴

• how big of an effect you’re testing for

• how confident you want to be in making a conclusion

That all sounds more complicated than ‘more participants = better study’, but it’s important to know when you’re reading research.

It’s why challenge trials can have great power despite having small samples.⁵ The risk factor, which is the virus in this case, is more common (every participant is exposed to it). It's also why seemingly-large studies can still be inadequate, as we saw earlier.

There are other examples too.

You could see and understand the harmful effects of radiation much more easily if you looked at people in the past who worked with X-rays without protection, because they had a lot of exposure to them.

You can see the benefits of flu vaccines in reducing the risks of stroke in the elderly much more easily than in young people (who are pretty unlikely to have a stroke regardless).

Now, this doesn’t mean the effects vary between people.

For example, flu vaccines might reduce the risk of stroke in young people just as much as they do in old people⁶, but it’s harder to detect the reduction in young people. Similarly, people who worked with X-rays weren’t inherently more susceptible to them, but they received much more exposure. This led to worse consequences for them, which were more noticeable.

Think of it this way: when are you most likely to see a difference, if it exists?⁷

---

I’ve gone down a rabbit hole of reading about challenge trials, so there'll be more content from me on the topic soon. But here are some other things I’ve been reading and listening to.

More

• I answered lots of questions for Big Think on progress and how we nurture it. It was part of a special issue containing lots of great content, including Hannah Ritchie on optimism and why we should be ‘impatient optimists’

• A very nice (and kind) post from Anton Howes on how to get writing

• India has a new HPV vaccine, which they say will soon be produced for the wider world too – that’s huge news

• And there’s a new meningococcal vaccine against 5 strains all together

• Some very fun episodes by the Ridiculous History podcast on inventors that died trying to fly with their own inventions and the history of Pringles

• Stuff You Missed in History Class on the invention of dishwashers and the Great London Smog

• I ate a lot of gelato while I was in Italy earlier this month. But more importantly I listened to this great collection of free audio tour guides for Europe – it’s awesome and great to hear at your own pace if you’re sightseeing⁸

• You might have heard of the White House memo on scientific publishing, which is meant to ensure ‘free, immediate, and equitable access to federally funded research’. How effective will it be? Brian Nosek explains in this great episode of Everything Hertz

• Joey Politano is now writing his Substack on macroeconomics full-time – check it out!

• This great study on life expectancy declines during Covid and how much they’ve been recovered in countries with high vaccination rates

• A study from Tamil Nadu finding that learning losses from Covid can be largely recovered

• The statistic about most books selling <12 copies is fairly exaggerated, although there is a long tail in book sales

• My great colleague Bastian Herre wrote on how the world has recently become less democratic

• The population of Hong Kong has shrunk from migration: 10–20% of young adults in Hong Kong have left in just the last half year

• And I found this chart very helpful, on scientific jargon that’s often misunderstood:

That’s all for now! As always, let me know if I’ve gotten anything wrong in this post, and I hope you subscribe if you haven’t already.

See you next time :)

– Saloni

1. Of course, it might be more important for you to test for the effects of the vaccine on some outcomes more than others.

2. There were around 13 candidate vaccines against Zika virus, as of 2020

3. This doesn’t mean it tells us about whether there is any effect regardless of its size. Instead it tells us about whether it has a given effect size or greater, with some degree of confidence.

4. There’s a balance here in how common a risk factor would need to be, to see a result. As I’ve explained before with a different example, ‘if no one in the world received antidepressants, we wouldn’t know what their effects were. But the opposite is also true. If everyone received antidepressants, we wouldn’t know how things would be without them.’ Instead, power would be at its highest if the treatment had a prevalence of 50%, assuming everything else is equal.

5. Like any study, you should still be careful of taking the results of challenge trials at face value, because it’s still possible for there to be research misconduct (p-hacking, publication bias, fraud, etc.) or other problems. For example, challenge trials might involve giving people an enormous exposure to a virus, that they wouldn't be able to experience in the real world. Then, a vaccine that would be protective in real world situations may still fail in the trial. So they usually need initial studies to find the right dosing. Another problem is that people who participate in a challenge trial might be quite different from the general population, and there might be differences in the efficacy depending on who receives the vaccine. So you usually need further studies with other groups too, but these don’t have to be full-scale trials.

6. As far as I’m aware, it’s unknown whether this effect size varies between age groups.

7. You could look at the people who are most at risk (like in a challenge trial, where everyone's exposed to the virus). You could pick an outcome that’s common (any infection, rather than congenital Zika syndrome). You could pick an outcome that’s easily measured. You could bluntly increase your sample size, etc.

8. Thanks to my friend Alasdair for recommending this!