The Reproducibility Project's first findings highlight reproducibility crisis

The Reproducibility Project: Cancer Biology has made available the results of the first five studies it attempted to replicate, and their report has taken the world of biology by storm. The findings published in eLife journal state that out of the five studies, only two were successfully repeated, while one failed replication and the remaining two concluded in “uninterpretable results.”  

The Reproducibility Project is not the first attempt to repeat published results. In 2012, a biotechnology firm, Amgen, had published a report stating that the company’s researchers had failed to replicate 47 of 53 groundbreaking cancer studies. Inspired by this, the Reproducibility Project was launched as a collaborated effort by Science Exchange and the Center for Open Science to replicate 29 high-profile cancer studies published from 2010 to 2012 that are in preclinical stage. However, unlike Amgen’s project, the process and findings of the Reproducibility Project will be made freely available.

These are the details of the studies that were reproduced as part of the Reproducibility Project:

• Of the five studies, the two reproducible studies were: a study led by Stanford computational biologist Atul Butte titled Discovery and preclinical validation of drug indications using compendia of public gene expression data and a study led by Constantine Mitsiades titled BET Bromodomain inhibition as a therapeutic strategy to target c-Myc while at Dana-Farber Cancer Institute.
• The studies that ended in uninterpretable results were Melanoma genome sequencing reveals frequent PREX2 mutations led by Petar Stojanov of Dana-Farber Cancer Institute and The CD47-signal regulatory protein alpha (SIRPa) interaction is a therapeutic target for human solid tumors led by Stanford stem cell biologist Irving Weissman.   
• The only study that was declared as irreproducible was the one led by Erkki Ruoslahti, a cancer biologist at the Sanford Burnham Prebys Medical Discovery Institute in California, titled Coadministration of a tumor-penetrating peptide enhances the efficacy of cancer drugs.

Many academics, particularly biologists, have expressed their support to the project as it exposes the lack of methodology details in published literature, and encourages researchers not to take results at face value. However, others have highlighted the fact that replicating biological studies is extremely difficult as there are several layers of complexity involved. Therefore, they advise against interpreting a failed replication as a permanent ruling on a study. However, John Ioannidis, epidemiologist of Stanford University in Palo Alto, California and an advisor to the project, summed up the project’s findings as: “The composite picture is, there is a reproducibility problem.”

References:      

Rigorous replication effort succeeds for just two of five cancer papers

Cancer reproducibility project releases first results