Reproducibility Project Named Among Top Scientific Achievements of 2015

The journal Science has named a major attempt to replicate 100 papers published in top-tier psychology journals as one of the “breakthroughs of the year” for 2015.

This collaborative project, facilitated by the Center for Open Science and APS Fellow Brian Nosek, has been recognized as a major scientific achievement by psychology but also by science as a whole.  The results were sobering for the field — less than half of results replicated — but they also provided a valuable estimate of the replicability of psychology papers and are motivating new attempts to improve reproducibility. Science published the results of the project in August 2015.

The reproducibility project joins other breakthroughs such as the development of CRISPR, a genome-editing technology; explorations of dwarf planets Ceres and Pluto; and the discovery of a new human species called Homo naledi.

The project was also a reader favorite, named a “people’s choice” breakthrough. Six percent of Science readers voted that the project was the most influential development to be published in the journal in 2015, coming in a tie with the discovery of quantum entanglement, a theory that one quantum particle can influence another far-away particle. No other breakthroughs from the fields of behavioral or social sciences made the list.

“The innovation that may have the largest effect on the rest of science… lies in the design of the replication efforts,” science journalist John Bohannon wrote about the reproducibility project.

“The researchers followed a procedure called preregistration, publishing the methods and rationale of each study before the experiments were undertaken. Then they reported the results and statistical analysis no matter what the outcome.”

“That prevents researchers from teasing out positive results from their data or leaving negative results unpublished. If everyone followed that protocol, false positives might all but disappear from journals,” he continued.

APS is strengthening the reproducibility of psychological science through the publication of Registered Replication Reports, which are large-scale, multi-site replication efforts conducted in collaboration with original study authors. Also, the APS journal Psychological Science awards authors badges for preregistering studies, as well as for making data and materials open and accessible. Additionally, APS supports the Transparency and Openness Promotion guidelines, which attempt to define and encourage scientific values that improve reproducibility.