Science Publications Laud Reproducibility Efforts

As 2015 came to a close, media outlets were publishing their typical year-in-review lists, and the replication movement in psychological science received recognition as one of the notable scientific advances of the year.

The journal Science named a major attempt to replicate 100 papers published in top-tier psychology journals as one of the “breakthroughs of the year.” This collaborative project, facilitated by the Center for Open Science (COS) and APS Fellow Brian Nosek, was recognized as a major scientific achievement for psychology but also for science as a whole.

Meanwhile, Nosek, COS cofounder and executive director, was named as one of “Nature’s 10” in the international journal’s news feature highlighting people who had a major influence on science during the year. Other notable figures on that list included planetary scientist Alan Stern, principal investigator in NASA’s seminal New Horizons mission to Pluto; Christiana Figueres, the United Nations diplomat who helped forge the Paris Climate Agreement; and Ali Akbar Salehi, Iran’s nuclear program chief who helped iron out the technical aspects of the international agreement to limit the country’s nuclear development.

Nosek also was included in The Chronicle of Higher Education’s “2015 Influence List” — its third annual roster “of 10 influencers and agitators” in academia.

The media attention centers on Nosek’s achievements in improving transparency, openness, and reproducibility in science — an objective that also has been a top priority for APS.

Similarly to Nosek and COS, APS has been encouraging researchers to adopt practices that will improve reproducibility, including making data and materials available and publishing experiment results even when they fail to support the researchers’ hypotheses. APS’s efforts in this arena include Psychological Science’s badge program recognizing open-science practices in journal articles and the Registered Replication Report (RRR) initiative, which publishes large-scale replications of important findings in psychological research.

The results of the reproducibility project that Science published in August were sobering for the field — less than half of results replicated — but they also provided a valuable estimate of the replicability of psychology papers and are motivating new attempts to improve reproducibility.

Science included the project on its top breakthroughs list along with the development of CRISPR, a genome-editing technology; explorations of dwarf planets Ceres and Pluto; and the discovery of a new human species called Homo naledi.

The project was also a reader favorite, named a “people’s choice” breakthrough. Six percent of Science readers voted it as the most influential development to be published in the journal in 2015; it tied with the discovery of quantum entanglement, a theory that one quantum particle can influence another far-away particle. No other breakthroughs from the fields of behavioral or social sciences made the list.

“The innovation that may have the largest effect on the rest of science … lies in the design of the replication efforts,” science journalist John Bohannon wrote about the reproducibility project. “The researchers followed a procedure called preregistration, publishing the methods and rationale of each study before the experiments were undertaken. Then they reported the results and statistical analysis no matter what the outcome.

“That prevents researchers from teasing out positive results from their data or leaving negative results unpublished,” he continued. “If everyone followed that protocol, false positives might all but disappear from journals.”

Nature noted several of COS’s milestones, including achieving $18 million in funding, growing its staff, and devising a set of guidelines for transparency and openness coauthored by Nosek and including APS among the signatories. But COS’s most visible feat, said writer Brendan Maher, is the Reproducibility Project.

The Chronicle called Nosek a “Research Watchdog” and emphasized the importance of the Reproducibility Project’s results in revealing a replication problem in science.

COS is conducting a second reproducibility project focused on research in cancer biology, and results are expected to be released in 2016. Nosek also is pursuing similar projects in the fields of ecology and computer science.

Brian Nosek will be speaking at a workshop on “Improving Reproducibility of Our Research Practices” at the 2016 APS Annual Convention, May 26–29 in Chicago, Illinois. The workshop is sponsored by the Society of Multivariate Experimental Psychology.