Further evidence continues to accumulate about the extent of the irreproducibility crisis—the combination of flawed statistics, arbitrary research techniques, and groupthink that afflict modern scientific research. A recent article at Nature Communications, “Regulation of REM and Non-REM Sleep by Periaqueductal GABAergic Neurons,” appears to be a poster-child for the irreproducibility crisis, for it states in the discussion of its procedures that “we continuously increased the number of animals until statistical significance was reached to support our conclusions.” This reads as an open avowal of p-hacking to achieve a false-positive result that was still publishable—you are supposed to determine your sample size first, not just stop collecting data at the moment that you happen to get a statistically significant result. Is this what happened? Nature claims to be committed to “substantive steps” to address the irreproducibility crisis, and we trust that the commitment also applies to Nature Communications. We have sent an inquiry to Nature Communications—our letter is at the bottom of this post.
We found out about this article from People for the Ethical Treatment of Animals (PETA), whose primary concern is that the article required painful, fatal animal research. The NAS’s fundamental commitment is rather to freedom of (scientific) inquiry. Yet we don’t relish the thought of p-hacking compounded by hacking apart mice. If animals are going to die for science, we’d rather it was for science done right.
It is possible that the articles’ authors simply explained badly what was actually a properly done scientific experiment. But we believe Nature Communications ought to clarify the facts at issue.
***
Dear Nature Communications,
Nature recently published “Scientific Rigour and Reproducibility” (https://www.nature.com/collections/byblhcfwhw), which stated that “There is growing alarm about results that cannot be reproduced. Explanations include increased levels of scrutiny, complexity of experiments and statistics, and pressures on researchers. Journals, scientists, institutions, and funders all have a part in tackling reproducibility. Nature has taken substantive steps to improve the transparency and robustness in what we publish, and to promote awareness within the scientific community.” Yet you recently published “Regulation of REM and Non-REM Sleep by Periaqueductal GABAergic Neurons” by Yang Dan, et al. (Nature Communications 9, 354, https://www.nature.com/articles/s41467-017-02765-w) , which contain this statement: “For optogenetic activation experiments, cell-type-specific ablation experiments, and in vivo recordings (optrode recordings and calcium imaging), we continuously increased the number of animals until statistical significance was reached to support our conclusions.” This reads as an open avowal of p-hacking to achieve a false-positive result that was still publishable. Is this what happened? I urge Nature Communications to re-examine this article to see whether it meets Nature’s standards—and to take appropriate measures if it does not. I also urge Nature Communications to provide a public explanation to its readership as to how it came to publish an article that seems openly to avow p-hacking.
Respectfully yours,
Peter W. Wood
President
National Association of Scholars
Image: Spark // Creative Commons
