Scientists Blame “Publish or Perish” Culture for Reproducibility Crisis

Science has a reproducibility crisis on its hands, and according to a new study in PLoS Biology, biomedical researchers believe the infamous “publish or perish” research culture is behind it.  

Irreproducibility in biomedical research

Reproducibility builds trust in science; it enables science to be progressive and it ensures scientific research can have a meaningful impact on our world.

Sadly, evidence indicates that we are facing a reproducibility crisis in science. A 2016 Nature survey of 1,500 researchers found that over 70% of respondents could not reproduce another scientist’s experiment. Further still, more than 50% could not reproduce their own work.

The same survey found that 83% of respondents agree that there is a reproducibility crisis in science, with 52% stating that they feel this crisis is “significant”.

Building on this survey (but, importantly, not seeking to reproduce it), the new work in PLoS Biology aimed to explore this complex issue, specifically within the biomedical science community.

“There is immediate importance to ensuring biomedical research is reproducible: here, studies that were subsequently not reproducible have led to patient harms,” the authors wrote. “By capturing a diverse and global group of biomedical researchers’ perceptions of reproducibility within the field we hope to better understand how to ensure reproducibility in biomedicine.”

Over 1600 biomedical scientists offer their take on reproducibility in science

The researchers, led by Dr. Kelly Cobey, an associate professor in the School of Epidemiology and Public Health at the University of Ottawa Heart Institute, randomly selected 400 journals in MEDLINE published between October 1, 2020–October 1, 2021.

The authors of these papers were contacted and invited to complete an anonymous online survey that collected demographic data, perceptions about a reproducibility crisis, their perceived cause of irreproducibility in research, their experience conducting reproducibility studies and their knowledge of funding and training for research on reproducibility. 

“We define reproducibility as re-doing a study using similar methods and obtaining findings consistent with the original study and as irreproducible when the findings are not consistent with the original study. This definition allows for variation in methods (e.g., conceptual and direct replications) between the original study and the reproducibility study as well as different definitions of how “consistent results” are defined (i.e., using p-value, observing results in the same direction, comparing effect sizes),” Cobey and colleagues described.

Over 1600 researchers, representing over 80 countries, responded. The majority of participants were either faculty members or primary investigators, and over 50% were male. Roughly half of the participants worked in clinical research.

Quantity over quality drives irreproducibility in science

Cobey and colleagues found 72% of survey respondents agree that a reproducibility crisis exists in biomedicine. Approximately 27% of these participants believe this crisis is “significant”.

“The concern appears to apply to biomedicine overall, but also specifically to clinical research, in vivo research, and in vitro research (11% or fewer participants indicated that they think more than 80% of papers in each category were reproducible),” Cobey and colleagues said.

More than 62% of participants attributed irreproducibility in science to the “publish or perish” culture.

“Publish or perish” reflects the unfortunate reality that, oftentimes, researchers must consistently publish in prestigious journals to secure long-term career prospects. This creates a perpetual cycle of fear and pressure that is unconducive to a flourishing research environment. 

Continue reading below...

Article

Almost 50% of Scientists Are Leaving Academia Within 10 Years – Here’s Why

Read more

“Concerns about how the current system of academic rewards stresses quantity over quality have been expressed for decades – a sentiment supported by this study’s data, which suggests that researchers’ performance is negatively impacted, in terms of producing reproducible research, by what the academic system incentivizes,” Cobey and colleagues emphasized.

Respondents highlighted other perceived causes of irreproducibility, including but not limited to poor study design, fraud, low quality peer review and lack of training in reproducibility. Only 16% of participants felt that their own institutions had established procedures to enhance reproducibility in biomedical research. Sixty-seven percent of participants reported feeling that their institution valued new research over replication studies.

Supporting scientists with reproducible science efforts

Cobey and colleagues believe that their work could be used to guide training and interventions that aim to improve reproducibility. They also state that conducting the same survey over time could help to explore how perceptions and behaviors are evolving.

“This international survey provides a contemporary cross-section of the biomedical community. While our survey approach and direction of findings are consistent with the previous Nature study, ongoing monitoring of perceptions of reproducibility in the community is critical to gauge shifts over time. Indeed, conducting this same survey again in the future would allow for a temporal comparison on how perceptions and behaviors shift over time.”

Continue reading below...

Article

Addressing the Reproducibility Crisis, One Step at a Time

Read more

Reference: Cobey KD, Ebrahimzadeh S, Page MJ, et al. Biomedical researchers’ perspectives on the reproducibility of research. PLOS Biol. 2024;22(11):e3002870. doi: 10.1371/journal.pbio.3002870