Preregistration Becoming the Norm in Psychological Science

A methodological revolution is underway in psychology, with preregistration at the forefront. Methodologists have made the case for the value of preregistration — the specification of a research design, hypotheses, and analysis plan prior to observing the outcomes of a study. And indeed, it is hardly radical to hold that predictions should be specified before looking at the data.

Preregistration improves research in two ways. First, preregistration provides a clear distinction between confirmatory research that uses data to test hypotheses and exploratory research that uses data to generate hypotheses. Mistaking exploratory results for confirmatory tests leads to misplaced confidence in the replicability of reported results.

Second, preregistering may reduce the influence of publication bias on effect-size estimation. Journals favor submissions that report statistically significant effects, a bias that tends to inflate estimates of effect size in the published literature. If preregistrations are posted in searchable registries, then it is possible to discover all research on a topic, not just the research that got published.

Registries are available for depositing and discovering preregistration. An emerging research community is evaluating the extent to which scientists can embrace and practice preregistration. Many journals recognize articles reporting preregistered research with badges. A related trend is for journals to adopt Registered Reports, in which preregistrations are submitted for peer review before data collection begins.

As a consequence of all this, psychological scientists are preregistering research at unprecedented and accelerating rates. Change is happening, much of it driven by APS’s journals, and there is plenty of guidance available for scientists wanting to adopt this practice. Here are some examples:

• Psychological scientist E. J. Wagenmakers and his colleagues provided the theoretical rationale for preregistration in “An agenda for purely confirmatory research” — one of the most cited articles from an influential 2012 special issue of Perspectives on Psychological Science on improving research practices.
• In a 2016 Observer story, Psychological Science Editor-in-Chief Steve Lindsay, Dan Simons, Editor of the new APS journal Advances in Methods and Practices in Psychological Science, and Clinical Psychological Science Editor Scott Lilienfeld reviewed the fundamentals of preregistration and described how it is being incorporated into publishing at APS journals.
• Brian Nosek and colleagues have just published “The Preregistration Revolution,” an article in the Proceedings for the National Academy of Sciences addressing some of the pragmatic challenges for conducting preregistration, such as what to do when the data already exist or the study is multivariate or longitudinal.

Box 2: Universities leading the COS Preregistration Challenge as of February 19, 2018. For the full list, visit https://cos.io/prereg/.

Organizations in the social and behavioral sciences field have set up registries to make it easy to preregister. These groups include the American Economic Association’s RCT registry, eGAP for political research, RIDIE for developmental economics, and the free workflow service OSF (http:osf.io) for any kind of research. Registries such as OSF enable researchers to embargo preregistrations so they can complete their research before making their designs publicly accessible. AsPredicted.com also provides an easy way to generate a preregistration, though it is not a formal registry because its preregistrations can stay private forever. That has the advantage of protecting researchers’ privacy and the disadvantage of making some research nondiscoverable.

One might hope that researchers would preregister just because it is good practice, but presuming that would mean neglecting psychology’s insights on behavior change. Adopting new behaviors is hard, particularly if they are unfamiliar and if the incentives counter the adoption. For example, maximizing publishability of findings encourages
retaining as much flexibility in analysis and reporting as possible, even at the cost of the accuracy of the results. If preregistration is to be adopted widely, the incentives for doing it will need to outweigh the incentives against it. Some of that change is occurring already.

The most direct incentive change is Registered Reports, which integrate preregistration with publication. Authors submit their question, methodology, and analysis plan for review before conducting the research. If accepted, that protocol is a preregistration of the confirmatory aspects of the study that will be published regardless of outcome as long as outcome-independent quality control criteria are met. The final paper clearly distinguishes between confirmatory tests and any exploratory findings that were examined after observing the data.

The growth in preregistration, demonstrated by the surge in journals encouraging preregistration by offering badges or Registered Reports and the total number of registrations accumulating on OSF, has been dramatic. From just 38 registrations in 2012 to more than 12,000 in 2017, registrations are doubling yearly (see below).

Psychology is not the only community adopting preregistration, but psychological scientists are leading the way in initiating the behavior and in evaluating its effectiveness for improving research practices. Ongoing self-study of research practices will foster continuous improvement and thereby accelerate the pace of discovering replicable phenomena and determining the factors that modulate their occurrence and size.