Irreproducibility: The soft underbelly of science

Is reproducibility of research a mirage? Moments of ‘Eureka’ are frequently becoming isolated instances for lone researchers rather than for the entire science community. The inability of reproducing a vast number of scientific findings is proving to be the soft underbelly of science, and one that needs to be addressed with priority.

According to a recently published report, irreproducible biology research costs the US a whopping $28 billion per year. Therefore, although the global spending on research and development is increasing steadily, the issue of irreproducibility is making R&D investment a bone of contention among funders, institutions, and researchers. Most disciplines, especially cosmology, psychology, medicine, and genetics are faced with findings that seem path breaking but lose their luster when another team of researchers tries to reproduce it. “Colleagues who once got excited now shake their heads and joke about 'yet another dark-matter candidate',” says Jan Conrad, a professor of astroparticle physics at the Oskar Klein Centre for Cosmoparticle Physics, Stockholm University, Sweden.

Findings that cannot be reproduced not only pose an economic challenge, but can also have an adverse impact on the understanding of science. While fraudulent research findings such as the STAP cell research are followed by retractions, in most cases, irreproducible results due to erroneous data continue to get cited. Scientific advancements are based on established findings and theories. Thus, one irreproducible research, on which several hypotheses might come to be based, can lead to a domino  effect in the understanding of a particular aspect in a discipline.

Research also forms the basis for health care policies, and results that cannot be reproduced can jeopardize public health and their trust in science. A recent instance of this is the much-publicized influential study that ran in the late '90s in Kenya according to which deworming improved children’s school performance and overall health. This study, published in 2004, led many developing countries to launch large-scale deworming initiatives. However, when a group of UK epidemiologists attempted to replicate the results a decade later, it was found that the study had many flaws. Irreproducible research, can thus, be consequential in undermining public’s trust in science.

Irreproducible research is rooted in a myriad of factors, but the pressure to publish lies at the heart of this problem. Breakthrough research ensures publication in a high impact factor journal and increases the chances of receiving funding; and this leads researchers to broadcast their findings without confirming the accuracy of their research. According to John Ioannidis, Professor of Health Research and Policy at Stanford School of Medicine and the author of the paper “Why Most Published Research Findings Are False,” most researchers fall into the trap of ‘data dredging.’ He explains that when an initial hypothesis fails to hold water, researchers scan their data to find “something that looks like a story” or “a bump in the data” that they believe is significant and base their findings on this statistically inaccurate detail. Unsurprisingly, another researcher who justly ignores the statistical fluke is unable to reproduce the findings. 

The slew of highly publicized results that fail to be replicated or reproduced, such as the evidence for gravitational waves, also spring from the trend of broadcasting findings publicly through press releases and preprint uploads before they are peer reviewed. Conrad comments, “Incorrect papers posted on arXiv do more than add to the noise of irrelevant results. Funding decisions are skewed; theorists waste a lot of time trying to devise explanations; and the public is misled through news reports.” Staying ahead of competition and securing positions has led many researchers to announce their findings prematurely, only to find out that the study is flawed.

Is there a way to combat this issue? Joined efforts of all major stakeholders of science and research can rein in irreproducibility. Researchers need to be made aware of the potential pitfalls they could encounter while doing their research that could lead to erroneous findings, such as the challenges associated with using big datasets. Some researchers have suggested making replication of at least one study from PhD students’ own specialism a mandatory part of their education with the intention of making them aware of the importance of transparency and reproducibility in research. 

Another means of ensuring scientific accuracy is that each discipline should have a protocol or guidelines that researchers can refer to; for instance, Conrad suggests that physicists could be asked to apply the 5-sigma rule (or an equivalent) to reaffirm their discoveries. Online posting of papers, a practice that is gaining popularity, should also be regulated to ensure the quality of research. Findings that seem significant should be accompanied with named reviewers, which would in turn ascertain stringent peer review and curtail the premature publicity of unconfirmed findings.

Related Infographic

Behind the science: Quick facts about scientific publishing

Check out this infographic to know more STM publishing…

4k

Journals and publishers too have joined the efforts to tackle the problem of irreproducibility. Most journals have made it mandatory for authors to submit their data along with their paper as it would facilitate replication and reanalysis. Some top-tier journals such as Nature and Science and other major publishers have agreed on a common set of guidelines that would promote transparency and reproducibility of research. To further the fight against reproducibility, Science Exchange has introduced Reproducibility Initiative, which is attempting to reproduce 50 most impactful cancer studies for independent validation.   

There is a surge in the science community’s initiatives to address the problem of irreproducibility. However, some experts have expressed caution in this regard because reproducible results may not necessarily equate to correct science. As Roger Peng, Associate Professor in the Department of Biostatistics at the Johns Hopkins Bloomberg School of Public Health, puts it, “One must always remember that a study can be reproducible and still be wrong. By ‘wrong’, I mean that the conclusion or claim can be wrong.” According to him, “Reproducibility is important because it is the only thing that an investigator can guarantee about a study.”

Reproducibility is the heart of science, but it is admittedly a complex parameter. Whether a particular set of findings are reproducible or not, it would inadvertently further the understanding of science. Science should, therefore, strive for reproducibility with the aim of lending transparency to research.