This transcript has been edited for clarity.
Welcome to Impact Factor, your weekly dose of commentary on a new medical study.
I am here today to talk about the effectiveness of COVID vaccine boosters in the midst of 2023. The reason I want to talk about this isn’t necessarily to dig into exactly how effective vaccines are. This is an area that’s been trod upon multiple times. But it does give me an opportunity to talk about a neat study design called the “test-negative case-control” design, which has some unique properties when you’re trying to evaluate the effect of something outside of the context of a randomized trial.
So, just a little bit of background to remind everyone where we are. These are the number of doses of COVID vaccines administered over time throughout the pandemic.
Centers for Disease Control and Prevention
You can see that it’s stratified by age. The orange lines are adults ages 18-49, for example. You can see a big wave of vaccination when the vaccine first came out at the start of 2021. Then subsequently, you can see smaller waves after the first and second booster authorizations, and maybe a bit of a pickup, particularly among older adults, when the bivalent boosters were authorized. But still very little overall pickup of the bivalent booster, compared with the monovalent vaccines, which might suggest vaccine fatigue going on this far into the pandemic. But it’s important to try to understand exactly how effective those new boosters are, at least at this point in time.
I’m talking about Early Estimates of Bivalent mRNA Booster Dose Vaccine Effectiveness in Preventing Symptomatic SARS-CoV-2 Infection Attributable to Omicron BA.5– and XBB/XBB.1.5–Related Sublineages Among Immunocompetent Adults – Increasing Community Access to Testing Program, United States, December 2022–January 2023, which came out in the Morbidity and Mortality Weekly Report very recently, which uses this test-negative case-control design to evaluate the ability of bivalent mRNA vaccines to prevent hospitalization.
The question is: Does receipt of a bivalent COVID vaccine booster prevent hospitalizations, ICU stay, or death? That may not be the question that is of interest to everyone. I know people are interested in symptoms, missed work, and transmission, but this paper was looking at hospitalization, ICU stay, and death.
What’s kind of tricky here is that the data they’re using are in people who are hospitalized with various diseases. You might look at that on the surface and say: “Well, you can’t – that’s impossible.” But you can, actually, with this cool test-negative case-control design.
Here’s basically how it works. You take a population of people who are hospitalized and confirmed to have COVID. Some of them will be vaccinated and some of them will be unvaccinated. And the proportion of vaccinated and unvaccinated people doesn’t tell you very much because it depends on how that compares with the rates in the general population, for instance. Let me clarify this. If 100% of the population were vaccinated, then 100% of the people hospitalized with COVID would be vaccinated. That doesn’t mean vaccines are bad. Put another way, if 90% of the population were vaccinated and 60% of people hospitalized with COVID were vaccinated, that would actually show that the vaccines were working to some extent, all else being equal. So it’s not just the raw percentages that tell you anything. Some people are vaccinated, some people aren’t. You need to understand what the baseline rate is.
The test-negative case-control design looks at people who are hospitalized without COVID. Now who those people are (who the controls are, in this case) is something you really need to think about. In the case of this CDC study, they used people who were hospitalized with COVID-like illnesses – flu-like illnesses, respiratory illnesses, pneumonia, influenza, etc. This is a pretty good idea because it standardizes a little bit for people who have access to healthcare. They can get to a hospital and they’re the type of person who would go to a hospital when they’re feeling sick. That’s a better control than the general population overall, which is something I like about this design.
Some of those people who don’t have COVID (they’re in the hospital for flu or whatever) will have been vaccinated for COVID, and some will not have been vaccinated for COVID. And of course, we don’t expect COVID vaccines necessarily to protect against the flu or pneumonia, but that gives us a way to standardize.
Dr. F. Perry Wilson
If you look at these Venn diagrams, I’ve got vaccinated/unvaccinated being exactly the same proportion, which would suggest that you’re just as likely to be hospitalized with COVID if you’re vaccinated as you are to be hospitalized with some other respiratory illness, which suggests that the vaccine isn’t particularly effective.
Dr. F. Perry Wilson
However, if you saw something like this, looking at all those patients with flu and other non-COVID illnesses, a lot more of them had been vaccinated for COVID. What that tells you is that we’re seeing fewer vaccinated people hospitalized with COVID than we would expect because we have this standardization from other respiratory infections. We expect this many vaccinated people because that’s how many vaccinated people there are who show up with flu. But in the COVID population, there are fewer, and that would suggest that the vaccines are effective. So that is the test-negative case-control design. You can do the same thing with ICU stays and death.
There are some assumptions here which you might already be thinking about. The most important one is that vaccination status is not associated with the risk for the disease. I always think of older people in this context. During the pandemic, at least in the United States, older people were much more likely to be vaccinated but were also much more likely to contract COVID and be hospitalized with COVID. The test-negative design actually accounts for this in some sense, because older people are also more likely to be hospitalized for things like flu and pneumonia. So there’s some control there.
But to the extent that older people are uniquely susceptible to COVID compared with other respiratory illnesses, that would bias your results to make the vaccines look worse. So the standard approach here is to adjust for these things. I think the CDC adjusted for age, sex, race, ethnicity, and a few other things to settle down and see how effective the vaccines were.
Let’s get to a worked example.
Dr. F. Perry Wilson
This is the actual data from the CDC paper. They had 6,907 individuals who were hospitalized with COVID, and 26% of them were unvaccinated. What’s the baseline rate that we would expect to be unvaccinated? A total of 59,234 individuals were hospitalized with a non-COVID respiratory illness, and 23% of them were unvaccinated. So you can see that there were more unvaccinated people than you would think in the COVID group. In other words, fewer vaccinated people, which suggests that the vaccine works to some degree because it’s keeping some people out of the hospital.
Now, 26% versus 23% is not a very impressive difference. But it gets more interesting when you break it down by the type of vaccine and how long ago the individual was vaccinated.
Dr. F. Perry Wilson
Let’s walk through the “all” group on this figure. What you can see is the calculated vaccine effectiveness. If you look at just the monovalent vaccine here, we see a 20% vaccine effectiveness. This means that you’re preventing 20% of hospitalizations basically due to COVID by people getting vaccinated. That’s okay but it’s certainly not anything to write home about. But we see much better vaccine effectiveness with the bivalent vaccine if it had been received within 60 days.
This compares people who received the bivalent vaccine within 60 days in the COVID group and the non-COVID group. The concern that the vaccine was given very recently affects both groups equally so it shouldn’t result in bias there. You see a step-off in vaccine effectiveness from 60 days, 60-120 days, and greater than 120 days. This is 4 months, and you’ve gone from 60% to 20%. When you break that down by age, you can see a similar pattern in the 18-to-65 group and potentially some more protection the greater than 65 age group.
Why is vaccine efficacy going down? The study doesn’t tell us, but we can hypothesize that this might be an immunologic effect – the antibodies or the protective T cells are waning over time. This could also reflect changes in the virus in the environment as the virus seeks to evade certain immune responses. But overall, this suggests that waiting a year between booster doses may leave you exposed for quite some time, although the take-home here is that bivalent vaccines in general are probably a good idea for the proportion of people who haven’t gotten them.
When we look at critical illness and death, the numbers look a little bit better.
Dr. F. Perry Wilson
You can see that bivalent is better than monovalent – certainly pretty good if you’ve received it within 60 days. It does tend to wane a little bit, but not nearly as much. You’ve still got about 50% vaccine efficacy beyond 120 days when we’re looking at critical illness, which is stays in the ICU and death.
The overriding thing to think about when we think about vaccine policy is that the way you get immunized against COVID is either by vaccine or by getting infected with COVID, or both.
Centers for Disease Control and Prevention
This really interesting graph from the CDC (although it’s updated only through quarter three of 2022) shows the proportion of Americans, based on routine lab tests, who have varying degrees of protection against COVID. What you can see is that, by quarter three of 2022, just 3.6% of people who had blood drawn at a commercial laboratory had no evidence of infection or vaccination. In other words, almost no one was totally naive. Then 26% of people had never been infected – they only have vaccine antibodies – plus 22% of people had only been infected but had never been vaccinated. And then 50% of people had both. So there’s a tremendous amount of existing immunity out there.
The really interesting question about future vaccination and future booster doses is, how does it work on the background of this pattern? The CDC study doesn’t tell us, and I don’t think they have the data to tell us the vaccine efficacy in these different groups. Is it more effective in people who have only had an infection, for example? Is it more effective in people who have only had vaccination versus people who had both, or people who have no protection whatsoever? Those are the really interesting questions that need to be answered going forward as vaccine policy gets developed in the future.
I hope this was a helpful primer on how the test-negative case-control design can answer questions that seem a little bit unanswerable.
F. Perry Wilson, MD, MSCE, is an associate professor of medicine and director of Yale’s Clinical and Translational Research Accelerator. He disclosed no relevant conflicts of interest.
A version of this article first appeared on Medscape.com.