Irreproducibility: Is the lack of an accepted definition a problem in itself?

Irreproducibility has rapidly gained the reputation of being one of the most persistent and concerning problems of science in recent times. Nature published a report of a survey it conducted to gain an insight into researchers' views about this issue. It came to light that the majority of researchers opined that “there is a significant crisis of reproducibility.” While the crisis is common knowledge, finding a way around it has proved to be difficult, primarily because there is no commonly agreed definition of the term reproducibility. Different meanings assigned to reproducibility that are at times contradictory. In a meeting conducted by National Library of Medicine to discuss how reproducibility can be improved in preclinical research, a point of significant discussion was that addressing the problem of irreproducibility is difficult without an understanding of what it stands for.

Often, the terms ‘reproducible’ and ‘replicable’ are frequently used interchangeably. While reproducibility primarily refers to the ability of the results to withstand different analyses, replication means attempting to repeat the results using the same or new raw data. However, these definitions vary across disciplines. In fact, science professionals have differing personal opinions on what these terms mean.      

In an attempt to demystify this, researchers have proposed definitions for the widely used umbrella terms 'reproducibility' and 'replication.' In What does research reproducibility mean? lead author Steven Goodman suggests splitting the term into these definitions:

• Methods reproducibility comes closest to the term replication as it means providing enough details about study procedures and data so that the same procedures could be repeated.
• Results reproducibility is closely related to methods reproducibility,  and refers to “obtaining the same results from the conduct of an independent study whose procedures are as closely matched to the original experiment as possible”
• Inferential reproducibility is different from the previous types of reproducibility because in this case, researchers might draw the same set of inferences from different studies or may use the original data to infer different conclusions. Thus, inferential reproducibility refers to “the drawing of qualitatively similar conclusions from either an independent replication of a study or a reanalysis of the original study”

Another notable attempt at standardizing the definition of reproducibility is by Victoria Stodden, a data scientist at the University of Illinois at Urbana-Champaign. She splits the term reproducibility into:

• Empirical reproducibility refers to providing all the details necessary to physically repeat and corroborate an experiment. Stodden’s definition of this term is similar to Goodman’s definition of methods reproducibility.  
• Computational and statistical reproducibility, according to her, refers to providing resources that would be essential to redo computational and analytical findings in a study.

Giving a twist to the entire discussion on reproducibility, a group of researchers at the American Society for Cell Biology in Bethesda published a paper in which they dismissed the term reproducibility. Instead, they introduced four types of replication:

• Analytic replication simply means reproducing results by reanalysing the original data.
• Direct replication refers to the attempts to use the same conditions, materials and methods as mentioned in an original experiment.
• Systematic replication means efforts made to reproduce the findings using different experimental conditions, for instance, trying an experiment in a different cell line or mouse strain.
• Conceptual replication indicates the efforts made to demonstrate the general validity of a concept, which might include using different organisms. 

Related Infographic

8 Things to remember when writing a statement of the problem

A statement of the problem helps you delineate and…

26k

It is apparent that pinning down the exact connotation of reproducibility is not easy – it is likely to differ across disciplines, topics, and even personal views. At the heart of irreproducibility are several problems such as lack of transparency, untrained researchers attempting to redo complex experiments, incomplete information about the methods, and so on. Thus, specifying the particular problems with a study that have rendered it difficult to replicate or reproduce, as well as defining the purpose behind the replication or reproduction, would be the key to resolving these issues. Therefore, many experts believe that having a unanimously agreed definition of reproducibility will make it easier to tackle the issue. The ultimate aim behind any published study should be making it easy to study, repeat, or adapt. With this intention, researchers should share their data, and strive to be transparent and detailed about their research design and methods. This might act as a bridge to avoid the so-called crisis of reproducibility.  

Recommended reading:

• Irreproducibility: The soft underbelly of science
• Is the reproducibility crisis real? An overview of Nature's survey
• Can scientific inquiry harm science?

References:

• Muddled meanings hamper efforts to fix reproducibility crisis
• What does “reproducibility” mean? New paper seeks to standardize the lexicon
• What does scientific reproducibility mean, anyway?