Journal conducts reproducibility trial, publishes one paper with two conclusions

The British Journal of Anaesthesia published a research with two conclusions as part of an extraordinary peer review trial with a view to broaden the scope of reproducibility of results. The journal published a paper that concluded the lack of a link between the depth of sedation during surgery and early death among the elderly. Interestingly, the journal published another paper based on the same research but with a different conclusion. According to the second paper, the number of patients included in the trial was not sufficient to reach any conclusion regarding the mortality rates among the elderly.

Hugh Hemmings, Editor-in-Chief of the British Journal of Anaesthesia and a neuropharmacologist at Weill Cornell Medical College, New York City, said that the unusual peer review is an effort to tackle “over-interpretation, spin and subjective bias” that is prevalent in the discussion sections of research papers. 

Under the rare peer review process, the journal now invites an independent expert to write the discussion section for some papers. The expert is able to access only the methods and results sections of the study, and not the conclusions. Based on their personal judgment, the expert is expected to write a discussion for the study, thereby extending the reproducibility aspect of a study beyond the methods section. The journal then publishes the two discussion sections at the same time outlining the differences and similarities. 

Welcoming the new approach, the lead author of the original paper, Frederick Sieber, a researcher in anaesthesiology and critical-care medicine at Johns Hopkins Bayview Medical Center, Baltimore said: “We’re all biased and this gives a second pair of eyes.” Sieber added that the original study is still valid as it aimed to measure the impact of the anaesthesia dosage not on death, but rather on delirium. The independent discussion also agrees with the original research’s conclusion regarding this, he pointed out.

While applauding the journal’s novel attempt, John Ioannidis, researcher at Stanford University in Palo Alto, California and one of the authors of the independent discussion emphasized the need to extend reproducibility to the inferences and conclusions of research papers. Ioannidis, who is an advocate for improving reproducibility, said that, “Out of very similar results with very similar methods people can make inferences or create narratives or tell stories that are very different.”

Departing from this view, Robert Sneyd, dean of the Plymouth University Peninsula Schools of Medicine and Dentistry, UK, noted that this process of writing independent discussions is likely to be assigned to the same pool of people who act as peer reviewers. Instead, he suggested that the existing rules and guidelines for authors be followed more strictly such as stating a study’s limitations with clarity.

According to Hemmings, the journal has in the pipeline another independent discussion of another paper, and that he will continue with this practice only if it is useful and received well by the academic community.

Related reading:

• eLife's new computationally reproducible article will allow users to modify figures
• The Reproducibility Project's first findings highlight reproducibility crisis
• Can we fix the reproducibility crisis that is plaguing science?

Reference:

• Reproducibility trial publishes two conclusions for one paper