Is Research Quality Being Compromised During the COVID-19 Pandemic?

Exploring research exceptionalism

Every day, the COVID-19 pandemic continues to erode at the "norms" of scientific research. As the world observes the ongoing mission to understand, challenge and defeat the SARS-CoV-2 virus, we are seeing laboratory and clinical studies conducted at a revolutionary speed, generating data in volumes that are somewhat incomprehensible.

It is an extraordinary time, but it is not an excuse for lowering scientific standards, argue Professor Jonathan Kimmelman, Director of the Biomedical Ethics Unit at McGill University, and Professor Alex John London, Director of the Center for Ethics and Policy at Carnegie Mellon University in a Policy Forum published in the journal Science.

Opening the article, the authors immediately address a core issue at the center of scientific research in a pandemic – the overwhelming sense of urgency: "A palpable sense of urgency and a lingering concern that in critical situations, large randomized controlled trials are not always feasible or ethical' perpetuate the perception that, when it comes to the rigors of science, crisis situations demand exceptions to high standards for quality." This is not the case, Kimmelman and London stress throughout the piece. The article comprehensively discusses the concerning issue of research exceptionalism, which Kimmelman defines as "making exceptions to well-established standards of research design, ethics and reporting" when speaking to Technology Networks in a recent interview.

Does research exceptionalism only occur, or increase, in the context of a pandemic? Kimmelman seems to think it is universal, regardless of the circumstances: "What is different here is that many researchers, funders, policy-makers, and other parties to the research enterprise seem to think that we can 'except' those deficiencies from our normal standards; that these lapses in study design or justification are acceptable given crisis conditions. Alex [Professor London] and I strongly disagree."

What factors contribute to research exceptionalism?

In the article, the authors suggest that research exceptionalism is underpinned by three problematic assumptions:

1. Some evidence now, even if flawed, seems preferable to expending greater resources on more demanding studies whose benefits only materialize later
   
   
2. The view that key features of rigorous research, like randomization or the use of placebo comparators, conflict with clinicians’ care obligations
   
   
3. The expectation that researchers and sponsors are generally free to exercise broad discretion over the organization and design of research

Expanding on the first point, Kimmelman says: "The proverb – 'a little bit of information is dangerous' – is operative here. For example, if people perceive – on the basis of poorly run studies – that a treatment is effective for COVID-19, they may be more reluctant to volunteer for placebo-controlled trials that rigorously evaluate that efficacy." This is an issue because it can slow the process of determining if a treatment is effective.

He adds: "Many treatments that look promising in small, preliminary reports – arguably most treatments – turn out when tested rigorously to be at best ineffective and at worst unsafe." The information that research provides is a public good, on which "caregivers, health systems and policy makers rely to efficiently discharge important moral responsibilities," the authors state in the article.

They discuss five conditions of informativeness and social value that research should embody to help the fulfil such responsibilities:

1. Importance

2. Rigorous design

3. Analytical integrity

4. Trials should be reported completely, promptly and consistently with prespecified analyses

5. Feasibility

The article describes several examples of how, in Kimmelman and London's opinion, these conditions are not being fulfilled.

Clinical trials

The process of developing, testing, and manufacturing a therapeutic or preventive agent is an arduous and lengthy process at the best of times. Add the pressures of a global pandemic to the mix and the complexity reaches a new level.

Nonetheless, Kimmelman and London strongly advocate the importance of adhering to the rigorous process of drug design and development that is typically enforced by regulatory agencies across the globe.

This includes the proper testing of therapeutics and preventives through clinical trials, and the scientists allude to some failings in this area: "We have in mind vaccine Phase I trials that skipped proper evaluation in animal models, or clinical trials that are not adopting blinding procedures – that is, preventing patients and physicians from knowing whether they are receiving drug or control."

In the article, the scientists also stress that "sponsors, research consortia and health agencies should prioritize research approaches that test multiple interventions".

A flurry of clinical trials that are of varying scales have emerged in recent months, taking place across different regions and sometimes testing the same therapeutic. "This massive commitment concentrates resources on nearly identical clinical hypotheses, creates competition for recruitment, and neglects opportunities to test other clinical hypotheses," Kimmelman and London say.

Can we look to work smarter in the face of a pandemic, and if so, how?

The scientists nod to recent innovations in approaches to trial design: "There are a number of new trial designs that better economize on time when evaluating treatments. Rather than testing five different treatments in five different clinical trials, you can test the five treatments in one clinical trial with one comparator arm – six arms in total. This allows you to compare treatments side by side, and it gives you better control over some of the statistical issues that arise when testing a lot of hypotheses."

Preprints: Valuable or detrimental?

In addition to the quality of research design, Kimmelman and London also address the methods by which research is disseminated in the Policy Forum, placing particular focus on the concept of study preprints. An increasing number of preprints are being made available daily via online servers.

"Normally, we subject scientific findings to scrutiny through peer review before they are disseminated to people who might use those findings. Via pre-print server, that filtering mechanism is not operating properly, meaning that decision-makers are often accessing information that is potentially unreliable or misleading," Kimmelman comments.

Preprints remain a relatively new concept in scientific publication, and thus, discussions surrounding their value and potential issues pertaining to their use are ongoing and pertinent.

On such topic, Kimmelman expresses uncertainty. "I’ve described above a lot of misgivings. I think if we had a coordinated effort involving an initial quality check before deposition, followed by a peer review within 24 hours of deposition, I’d feel a lot better."

Interpreting a scientific report can be quite the challenge for an individual that is not a specialist in that particular research area, meaning their critical evaluation could be inadequate. Kimmelman suggests that a mechanism to make preprints scrutable for non-specialists on a fast timeline would be a plausible solution here.

Medical journals are also placed under the spotlight by Kimmelman and London, who suggest that conclusions are being drawn from data without assessing the full context of the experiment. Kimmelman told Technology Networks: "Publications in prestigious medical journals [are] claiming that patients receiving a drug through “compassionate use” benefited, even though there was no control group or study protocol. Here is a classic example of exceptionalism. Normally a top tier medical journal would not publish a paper like this; it would be more likely to use the paper it is printed on as bedding in a hamster cage."

"One lesson of the current outbreak is that expeditious research in a crisis situation is feasible," - Kimmelman. 

"Getting to the finish line"

Akin to the story of the tortoise and the hare, it seems the take home message from Kimmelman and London's Policy Forum is that haste, with little regard for standards, is not the answer in this current situation. Rather, the researchers eloquently bring to light the sheer importance of "combining efforts, dividing labor and triaging out low-value research at all times".

Photo credit: Steven Lelham on Unsplash.


"The most efficient way to get to the finish line is to demand exacting standards of evidence. Relaxing those standards – even if the individuals doing so have good intentions – only threatens to prolong our uncertainty and waste resources," Kimmelman concludes.

Jonathan Kimmelman was speaking to Molly Campbell, Science Writer, Technology Networks.

Reference: Alex John London and Jonathan Kimmelman. (2020). Against pandemic research exceptionalism. Science. DOI: https://science.sciencemag.org/cgi/doi/10.1126/science.abc1731.