What Does Safe And Effective Mean?

How statistics can be used in less than purely mathematical ways.

The post this week will be a bit different, starting with the Pet Peeves Department. This will be trivial perhaps to those in the sciences or other fields who are used to dealing with statistical concepts, if so feel free to skip ahead. For most other people who are not comfortable with confidence intervals and statistical significance, we rely on interpretations presented in lay publications, or government public health communications. It is important to understand how the presentation of data can have a profound influence on your impression of its significance, and I would like to explore this a bit. The CDC most often presents data to the public in easily understood terms such as, “The risk of hospitalization is three times higher in people of a certain age group who have not taken the bivalent booster, compared to people who have.” Let’s take a closer look at this first type of statement; and say, for arguments sake, that you were weighing whether you want to take a booster, perhaps because you had side effects from previous shots. If you thought, well three out of ten people people going to the hospital compared to one is very significant—that’s 20% more so I’d better take the booster. But that is not what the data statement is telling you. You don’t get the denominator from this kind of representation. Suppose it was three people out of 100 verses one person, that’s three times as many but only 2% more. Going further consider it might have been three people in hospital versus one out of 100,000. The risk picture changes rapidly as the denominator goes up. The statement that three times as many people get hospitalized without the booster is correct, but is it a fair way to present information that you presumably might be weighing in a personal cost benefit analysis? I have seen this repeatedly in some journalist’s reports of results from the epidemiology literature. Whether by design or subconscious slip, the authors will present data which is slightly favorable to their obvious bias as a positive multiple of the comparator group—and when the outcome is unfavorable to their bias, they will represent the group outcomes as percentages, which often makes the difference appear smaller. A similar form of reader manipulation can easily occur when the outcomes are expressed as a percentage difference between the respective groups. and you are not aware of the actual real numbers of events in each group. As an example, let’s suppose that you are told that people given a certain drug for early COVID had a 33% lower chance of landing in the hospital than the untreated group. Sounds impressive, and a near certainty you would want to take it, if side effects were fairly low. But you might come to a different conclusion if you knew that the study encompassed only a few hundred people and the outcome of hospitalization was quite low in both groups. Suppose the hospitalization rate in the untreated group was 1.6%, and in the treated group it was 1.2%. Perhaps there were only one or two fewer treated people who end up in the hospital. Now you are much less impressed—but it is true that the Difference between the two groups is a 33% reduction.

Now to look at a second type of declaration, what I will call a Blanket Statement. If you have been reading these pages for a while, you know that I’m a fan of the maxim, “One size doesn’t fit all”. That is especially true in medicine, NBA sneakers, and even national approaches to Pandemic management. A little digression is in order here to consider exactly what is meant when the FDA says a drug is “Safe and Effective”. The effective part is pretty clear; the drug causes some predetermined level of improvement in survival, recovery or symptoms. The Safe part of the statement is a little more nuanced. Obviously it doesn’t mean the drug is without side effects, there is no such drug. It means that the side effects, at least as understood from the clinical trials thus far, are acceptable when balanced against the potential benefit. Cancer drugs can have terrible side effects, up to and including death; but they are considered safe enough when balanced against the certain outcome of untreated cancer. Getting back to a Blanket Statement like, “The risk of myocarditis from COVID is greater than the risk of vaccine induced myocarditis”. This has been repeated frequently and generally without qualification. It appears in bold headlines on the CDC page devoted to this tropic. This is true enough when applied to the overall population, and importantly when applied to a certain time frame, when a significant proportion of the population were naive to COVID with no prior natural immunity. We know that COVID disease severe enough to lead to hospitalization is much less now than in the earlier phases of the Pandemic. The Omicron family mercifully appears to be a bit less pathogenic, and the population has acquired a very significant degree of immunity from vaccination, infection alone or hybrid immunity. The CDC studies of seropositivity are around 97-98% at this point, which means that virtually every unvaccinated person has been infected (along with a majority of vaccinated people).

We also have solid evidence that natural infection, especially in young healthy people, provides long lasting protection from severe disease, which would include myocarditis. It is also clear that the risk for vaccine caused myocarditis is not uniformly distributed within the population, but is much higher in teenage boys and males in their early 20’s The exact risk of myocarditis in teenage males has varied from study to study, and population to population; but the most important question remains, does the risk of COVID infection really pose a greater risk of myocarditis than the vaccine—or is the Blanket Statement misleading by way of omission of important information? There has been a very recent study preprint suggesting that the vaccine may cause more myopericarditis than it prevents, but it has been heavily been criticized for relying on the US VAERS reporting system of vaccine side effects, which admittedly has many pitfalls. Instead let’s have a look at the following study of 40 million British folks published in the American Heart Association Journal Circulation. It takes advantage of the robust data collected by their NHS, which frankly we can’t compete with, and specifically looks at this question by age group. They came to the conclusions I will quote below with the reference, which is quite lengthly. I have not seen any detailed, specific challenges to this paper’s methodology or conclusions.

The risk of vaccine-associated myocarditis is consistently higher in younger men, particularly after a second dose of mRNA-1273, where the number of additional events during 28 days was estimated to be 97 per million people exposed. An important consideration for this group is that the risk of myocarditis after a second dose of mRNA-1273 was higher than the risk after infection. Indeed, in younger women, although the relative risks of myocarditis were lower than in younger men, the number of additional events per million after a second dose of mRNA-1273 was similar to the number after infection. These findings may justify some reconsideration of the selection of vaccine type, the timing of vaccine doses, and the net benefit of booster doses in young people, particularly in young men.

Risk of Myocarditis After Sequential Doses of COVID-19 Vaccine and SARS-CoV-2 Infection by Age and Sex https://doi.org/10.1161/CIRCULATIONAHA.122.059970Circulation. 2022;146:743–754

Now consider that the data this paper is based upon was collected in the pre-Omicron era, when the severity of infection and the rates of hospitalization were much greater. Far fewer teenagers are being admitted to the hospital today and a logical conclusion is that far fewer are developing COVID myocarditis—while there is no reason to posit that the rate of vaccine induced myocarditis has diminished. As of November 1st, 2022, Sweden stopped recommending COVID vaccinations for children age 12-17, citing the diminished risk of severe disease for this age group in the Omicron era, and the counter balancing risk of side effects. At the end of September, the UK also stopped recommending COVID vaccinations for children under age 11— a far cry from the CDC not only continuing to push for vaccinations in this age group but also further boosters.

One size does not fit all. Reasonable people can look at the same data set and draw different conclusions. Other nations, and other people, are not crazy because they chose a different path trough the Pandemic, it didn’t come with a road map or a crystal ball. Remain flexible in your thinking.

Thanks as always for your time. Comments and critiques are welcome here. Please share with your circle of friends. Below the buttons you will find several pictures of me: receiving the Presidential Freedom Medal, the Nobel Prize and my Knighting by the late Queen Elizabeth. Normally my modesty would prevent me from sharing these, and I do so only because I see a potential opportunity to run for office should Congressman George Santos seat become suddenly available.