Psychological Science Badge Program Encourages Open Practices, Study Shows

A new analysis indicates that the Open Practices Badge program launched two years ago in Psychological Science has positioned APS’s flagship journal at the forefront of the research transparency and openness movement in scientific publishing.

In an article published May 12, 2016, in the online journal PLoS Biology, a team of researchers from the Center for Open Science (COS) report an overall 10-fold increase in data sharing in papers published in Psychological Science since the badge program’s debut in May 2014.

The team, led by COS co-founder and executive director Brian Nosek, also an APS Fellow, found no change in rates of data sharing during that time among four other empirical psychology journals.

The open practices badge program encourages authors to engage in open research practices and was devised in partnership with the COS. Articles accepted for publication in Psychological Science may be awarded badges for meeting any or all of the following criteria:

• Open Data: The experiment’s data were submitted to an open-access repository.
• Open Materials: The experiment’s materials were submitted to an open-access repository.
• Preregistration: The study’s results were reported according to an open-access design and analysis plan formulated prior to data collection.

Twenty-three percent of articles published in Psychological Science have included open data since the badge program began, compared to less than 3% prior to the launch, according to the study, which examined nearly 840 papers from the journal. What’s more, nearly 40% of all papers published in the journal in the first half of 2015 included shared data.

The study assessed a total of 2,478 empirical articles published from 2012 through May of 2015 in Psychological Science; Journal of Personality and Social Psychology; Journal of Experimental Psychology: Learning, Memory, and Cognition; Developmental Psychology; and Clinical Psychological Science (also an APS publication). Of those journals, only Psychological Science awards badges for open science practices. In fact, only six academic journals have adopted badge programs thus far.

The COS researchers examined whether each article made data and materials available; where those data could be found (e.g. appendix, personal website, independent repository); whether the promised data and materials were actually found at the cited location; and whether they were complete, understandable, and could be used to reproduce the study’s originally-reported results.

Results show that 100% of the papers in Psychological Science that reported open data and materials actually lived up to that promise, with nearly 83% of that information being complete, 91% correct, and 76% usable. Prior to the badges program, only 20% of the reportedly available data in Psychological Science papers was complete, only 60% of it correct, and only half of it usable.

Across the other four journals, only 41% of articles with data reportedly available actually lived up to that claim, with only 16% of the information being complete, only 27% of it correct, and only 22% of it usable.

The COS team noted some limitations of their analyses, noting that the adoption of the badges program may have sparked a significant change in the population of authors submitting or earning acceptance to Psychological Science. Also, the evaluations of data and material accessibility, correctness, completeness, and usability were based on the researchers’ assessments and did not involve actual reanalysis of the data or reuse of the materials.

Nevertheless, the results of the COS analysis suggest that badge programs like that instituted by Psychological Science can increase data sharing substantially.

“Badges may seem more appropriate for scouts than scientists,” they write, “and some have suggested that badges are not needed. However, actual evidence suggests that this very simple intervention is sufficient to overcome some barriers to sharing data and materials. Badges signal a valued behavior, and the specifications for earning the badges offer simple guides for enacting that behavior.”

For an in-depth look at the badge program in its second year, see:

Open Practice Badges in Psychological Science: 18 Months On

Reference

Kidwell, M. C., Lazarević, L. B., Baranski, E., Hardwicke, T. E., Piechowski, S., Falkenberg, L., … Nosek, B. A. (2016). Badges to acknowledge open practices: A simple, low-cost, effective method for increasing transparency. PLoS Biology, 14(5). doi: 10.1371/journal.pbio.1002456.