The Scientific Validity of Peer-Reviewed Publications: A Concerning Situation With Implications for Oncology

Maurie Markman, MD

We are at an interesting, even paradigm-changing juncture in the academic exercise of scientific peer-reviewed publications. This long-standing process has served society well in establishing criteria for an objective evaluation of observations from the laboratory or clinic, including claims and conclusions made following the completion of clinical trials.

There is an unofficial ladder of scientific journals, just as in the realm of the academic world, with many referring to the “impact factor” (or a similar measure of how other investigators cite previously published works) as a guide to a publication’s place within this hierarchy. As a card-carrying academic for several decades, this commentator is well aware of the influence of an article published, for example, in Nature, Science, or the New England Journal of Medicine vs other quite excellent journals that might be considered by some to be somewhat or even substantially lower in the recognized continuum of scientific publications. Although clinical oncology will have its own listing of premier vs nonpremier journals, the unofficial lists clearly exist. The effect of the publication venue is relevant not only for the scientific community and society at large but may have a major influence on the process of academic advancement as well as subsequent recruitment and job retention, among other potential personal benefits.

Unfortunately, a number of disquieting events have shown how much this rather idealized view of the academic peer-reviewed publi- cation process does not represent an objectively adequate view of the current state of affairs, raising a vexing challenge for the scientific community. Who would have thought even a decade ago that a scientific superstar a now ex-president of Stanford University in California, would be forced to resign because of question- able publications?¹ Yes, the claim is that others under his leadership were responsible for the multiple documented concerning issues, but he was ultimately responsible for the submitted and subsequently published manuscripts. Similarly, in the oncology realm, could one imagine that multiple senior investigators at Dana-Farber Cancer Institute in Boston, Massachusetts, would be embroiled in serious controversy regarding the quality of their research publications, with a final “to-be-determined” number of peer-reviewed publications required to be withdrawn?²

To be clear, the academic commu- nity has witnessed in the more distant past the need for withdrawal of manuscripts following publica- tion after they had undergone what was declared to be appropriate peer review. Consider, for example, the extremely seriously flawed research paper by Andrew Wakefield in The Lancet that inappropriately claimed an association between measles, mumps, and rubella vaccination and the subsequent development of autism,³ which was finally with- drawn 12 years after publication.⁴

Although there is absolutely no intent in this discussion to equate the Wakefield scandal to the previously noted reports of withdrawal of scientific publications, news reports highlighting these incidences have the legitimate potential to undermine, perhaps seriously, the faith of many in the scientific stablishment.

Further, these events seem to be far from isolated. Recent reports have noted “egregious conduct” in Alzheimer studies,⁷ data manipulation by a prominent Harvard “honesty researcher,”⁸ required retractions of multiple publications dealing with sports-related concussions from the former editor in chief of a sports medicine journal,⁹ the retractions of several papers by a physicist studying superconductivity,¹⁰ and the ongoing investigation of manuscripts coauthored by a Nobel Prize–winning neuroscientist.¹¹

In addition to these reports of questionable publications from high-profile scientists appearing in the peer-reviewed literature, other issues raise more general concerns. There has been an explosion of journals over recent years, many unfortunately of quite dubious quality. The term predatory journals has been coined to help identify publication efforts that “do not provide scientifically rigorous peer review; their feedback is rubbish. They also have dishonest business practices.”¹² Unfortunately, even when a publication of this nature is identified, the company may simply rebrand itself under a different name and be back in business.¹³

If what has already been noted is not enough, there is increasing evidence of rather large-scale fraud in certain publications, which, when discovered, has required the withdrawal of hundreds or even thousands of papers¹⁴; unabashed bribery of editors to ensure publication¹⁵; and the direct selling of coauthorship,¹⁶ including to individuals from foreign countries attempting to improve their chances to obtain a medical residency in the United States.¹⁷

In a recent commentary in Scientific American, Naomi Oreskes notes the massive expansion of scientific publication over the past several decades.¹⁸ She highlights a study finding that there are currently more than 7 million such publications each year, whereas that number in 1980 was less than 1 million. Further, she comments on “[results from] another study that found 265 academic authors—two-thirds of whom were in the medical and life sciences—who published a paper every 5 days on average.”

Who is providing the peer review for this massive number of manuscripts? Do we need to wait for the limited group of “science sleuths” to discover the errors (inadvertent or intentional) that make it through this increasingly porous process?¹⁹ The concerns being emphasized here are the multiple challenges facing what has been traditionally considered the gold standard for the objective evaluation of scientific efforts, including clinical investigation: rigorously undertaken and adequately overseen peer review.

In the opinion of this commentator, it is critical that now, and even more so in the future, the scientific community be able to effectively communicate observations from investigative efforts as well as their relevance for our individual and societal welfare, both internally and with the public. Unfortunately, if the academic peer review process is unable to be fully trusted, it is not difficult to contemplate the possible ramifications.

References

1. Kaiser J. ‘I should have done better’: Stanford head steps down. Science. 2023;381(6656):366-367.doi:10.1126/science.adj9568
2. Cancer studies face retractions. Science. 2024;383(6681):351.
3. Horton R. A statement by the editors of The Lancet. Lancet. 2004;363(9411):820-821. doi:10.1016/S0140-6736(04)15699-7
4. Eggertson L. Lancet retracts 12-year-old article linking autism to MMR vaccines. CMAJ.2010;182(4):E199-E200. doi:10.1503/cmaj.109-3179
5. Subbaraman N. Harvard hospital seeks study retractions. Wall Street Journal. January 23, 2024:A3.
6. Subbaraman N. Scientist engaged in misconduct, probe finds. Wall Street Journal. March 21, 2024:A6.
7. Piller C. Probe of Alzheimer’s studies finds ‘egregious misconduct’. Science. 2023;382(6668):251-252.doi:10.1126/science.adl4276
8. O’Grady C. Embattled Harvard honesty professor accused of plagiarism. Science. 2024;384(6692):148-149. doi:1126/science.adp7470
9. Concussion publications pulled. Science. 2024;384(6697):719.
10. Garisto D. Third retraction looms for superconductivity physicist. Science. 2023;382(6666):19-20.doi:10.1126/science.adl1808
11. Else H. Journals investigating Nobel winner’s papers. Science. 2024;384(6695):496-497. doi:10.1126/science.adq1774
12. Moher D, Moher E. Stop predatory publishers now: act collaboratively. Ann Intern Med.2016;164(9):616-617. doi:10.7326/M15-3015
13. Predatory publisher rebrands. Science. 2021;374(6567):517.
14. Subbaraman N. Counterfeit studies are infecting scientific journals. Wall Street Journal. May 15, 2024:A1.
15. Joelving F; Retraction Watch. Paper trail. Science. 2024;383(6680):252-255. doi:10.1126/science.ado0309
16. Chawla DS. How a site peddles author slots in reputable publishers’ journals. Science.2022;376(6590):231-232. doi:10.1126/science.abq4276
17. Joelving F; Retraction Watch. Prescription for controversy. Science. 2024;384(6696):612-614.doi:10.1126/science.adq2869
18. Oreskes N. Trouble in the fast lane: scientific research needs to slow down, not speed up. Sci Am. 2024;330(4):69.doi:10.1038/scientificamerican0424-69
19. Subbaraman N. Science sleuth says papers use copied images. Wall Street Journal. February 2, 2024:A5.