Reaching Conclusions From Limited Data Holds Pitfalls

Maurie Markman, MD

There are most interesting and complex days in the continuing evolution of the COVID-19 pandemic. On one hand, in the United States and a number of other countries, mass vaccination strategies have finally begun to have a major impact on the incidence and severity of infections, with a substantial proportion of the adult population being fully or at least partially vaccinated.

Further, success with the production of several vaccine products in the United States has enabled federal and state government officials to declare that vaccination should be available for all adults, at least in theory, although perhaps not necessarily with the required vaccine supply in specific regions. And plans are underway to initiate widespread vaccinations of adolescents, likely to be followed later this year or early next year by children. This is a critically relevant development as variants continue to become widespread within the population.¹

Unfortunately, in many countries the availability of COVID-19 vaccines has been highly problematic and the situation in the developing world is particularly dire. Case numbers in many countries continue to rise as concern about the impact of variants on infectivity and the severity of illness heightens. Added to these major issues is the finding that certain vaccine products appear to be associated with a very small—perhaps 1 in a million—but highly clinically relevant risk of serious, even fatal, blood clots.^2,3 Considering the extremely rare nature of these events, investigators have had difficulty characterizing what has been observed except to say the clots have been seen in women between the ages of 18 and 59 years (median, 37 years) and the events are associated with low platelet counts.2 Finally, this analysis is made more complex by knowledge that a diagnosis of a COVID-19 infection itself substantially increases the risk of blood clots.⁴

International and national public health agencies as well as leading academics around the world have strug-gled in very real time and in full view of the public to explain what is occurring, how these events might be predicted and prevented, and, ultimately, successfully treated. Under nonpandemic circumstances, discus-sions and deliberations regarding vaccine safety and efficacy by these agencies and experts would likely have been conducted outside public view with the ability to obtain additional essential information over time before definitive decisions must be made. This will hopefully be the case as novel vaccines are developed to prevent future pandemics, including events associated with other coronaviruses.⁵

REAL-WORLD EXPERIENCE

The challenges of assessing COVID-19 vaccines shows that events in the real world may differ from those of a formal objective scientific analysis, especially in a setting where such evaluations of necessity involve very small numbers.

As of press time, health agencies in the United States and Europe have reasonably concluded that the benefits of vaccination outweigh the potential risk of serious—even fatal—blood clots, considering the seriousness of a COVID-19 infection, including the risks posed by new and evolving variants.^2,6 Based on existing data, it is difficult to disagree with these decisions.

However, from the perspective of specific individuals considering receiving a COVID-19 vaccine, the actual potential for becoming infected with COVID-19 and, most importantly, experiencing a serious infection is essentially unknown, whereas for the person who develops a major blood clot, regardless of how rare this event may be, the risk of severe complications is 100%. Further, younger individuals (for example, those 40 years and younger) appear to have a higher risk of developing a blood clot from certain COVID-19 vaccines and a lower risk of experiencing a serious COVID-19 infection than older people.

The point here is that although formal statistical evaluation is very relevant, such analyses must be viewed against a background of views and interpretations that involve individuals in the real world who will inquire about how the evaluation pertains to them. And as the numbers available to draw statistical conclusions become smaller, the ability to draw scientifically valid and clinically meaningful conclusions will decrease.

Although this discussion of a rare toxicity from COVID-19 vaccination is certainly not directly relevant in the cancer arena, the use of statistical tools involving very small numbers can be an equally meaningful and concerning issue.

Consider, for example, a report in an oncology journal that sought to assess the impact of complementary medicine (CM) on cancer-specific outcomes through an analysis based on very small numbers.⁷ This evaluation included 1,901,815 individuals in the National Cancer Database who received diagnoses between January 1, 2004, and December 31, 2013, of nonmetastatic breast, lung, prostate, or colorectal cancers. Investigators found a total of 258 patients who reported that they used CM techniques, defined as “other-unproven: cancer treatments administered by nonmedical personnel” in addition to standard therapeutic strategies.

The investigators attempted to match patients in the larger data-base with these 258 individuals. They concluded that “patients who received CM were more likely to refuse additional conventional cancer treatment and had a higher risk of death”6 than those who did not use CM. This conclusion was reached with an at-risk population of CM believers that made up 0.0001% of the entire study population.

It is relevant to note the public health care agencies and experts in the epidemiology of vaccine use were very careful to not overly gener-alize their conclusions, knowing they were dealing with very limited patient numbers. Unfortunately, such caution did not appear to be relevant to the investigators and reviewers of the CM study. Perhaps investigators in all disciplines can learn something of importance from their academic colleagues in the vaccine arena.

References

1. SARS-CoV-2 variant classifications and definitions. Centers for Disease Control and Prevention. Updated April 27, 2021. Accessed April 28, 2021. https://bit.ly/2QxSLns
2. FDA and CDC lift recommended pause on Johnson & Johnson (Janssen) COVID-19 vaccine use following thorough safety review. Centers for Disease Control and Prevention. April 23, 2021. Accessed April 29, 2021. https://bit.ly/3nDyYPp
3. Kupferschmidt K, Vogel G. Vaccine link to serious clotting disorder firms up. Science. 2021; 372(6539):220-221. doi:10.1126/science.372.6539.220
4. Taquet M, Husain M, Geddes JR, Luciano S, Harrison PJ. Cerebral venous thrombosis and portal vein thrombosis: a retrospective cohort study of 537,913 COVID-19 cases. Open Science Framework. April 14, 2021. Updated April 26, 2021. Accessed April 29, 2021. doi:10.17605/OSF.IO/H2MT7
5. Cohen J. The dream vaccine. Science. 2021;372(6539):227-231. doi:10.1126/science.372.6539.227
6. AstraZeneca’s COVID-19 vaccine: benefits and risks in context. European Medicines Agency. April 23, 2021. Accessed April 29, 2021. https://bit.ly/3vyhgQd
7. Johnson SB, Park HS, Gross CR, Yu JB. Complementary medicine, refusal of conventional cancer therapy, and survival among patients with curable cancers. JAMA Oncol. 2018;4(10):1375-1381. doi:10.1001/jamaoncol.2018.2487