Registered Replication Reports: An Update

When APS debuted the Registered Replication Report (RRR) initiative in 2013, it marked a milestone in the reproducibility movement that has been building in psychological science and other areas of scientific inquiry in recent years. The goal of the initiative is to publish large-scale, multicenter replications of important findings that can give more precise estimates of effect sizes; to this end, one RRR has been completed and several more are on the way.

Thirty-one labs participated in replicating APS Fellow Jonathan Schooler and Tonya Engstler-Schooler’s experiment demonstrating the verbal overshadowing effect, which shows that describing a previously seen face can actually make it more difficult to recognize that face later. The report was published in the September 2014 issue of Perspectives on Psychological Science, and a meta-analysis of the results found a small verbal overshadowing effect when participants described a suspect immediately after “witnessing” a staged bank robbery. Some researchers also ran another experiment in which subjects waited 20 minutes before giving a description and found that people were even less accurate when later identifying the suspect from a lineup under this condition, suggesting that time may play a role in this effect.

In addition to these meta-analytic findings, the completion of the first RRR presents the opportunity to reflect on some of the concerns raised before the initiative began. Critics of the “replication revolution” have expressed apprehension about how researchers who found their articles targeted for replication would respond to what some might perceive as suspicion regarding the veracity of their results.

However, in his commentary published alongside the final report, Schooler fully endorsed the process and outcome of the RRR. “The present project provides an unprecedented example of the value of a carefully conducted, large-scale, multisite replication effort,” he wrote.

APS Fellow Eli Finkel, lead author of the target article of a new RRR project, is similarly enthusiastic about his participation. “The process thus far has been highly collaborative and not at all antagonistic,” he says. “I found the proposal to be detailed and compelling, and I’ve been extremely impressed with [Perspectives Special Associate Editor] Dan Simons’s conscientiousness and open-mindedness.”

Building on the logistical and collaborative success of the Schooler and Engstler-Schooler replication, several more RRR projects are in the works. Data collection is complete and analysis is currently underway for the next RRR to be published, a replication of William Hart and APS Fellow Dolores Albarracín’s 2011 Psychological Science article showing that how an action is represented in time (“she was going” vs. “she went”), a construct known as grammatical aspect, influences a person’s perception of intentionality.

APS William James Fellow Roy Baumeister and colleagues’ concept of ego depletion, which frames self-control as a finite resource that is drained incrementally as a person uses it, has been the subject of much attention since it was first published in 1998. Uncertainty and disagreements among researchers within this burgeoning area of research make it an ideal subject for an RRR, and labs are currently collecting data.

The Many Labs 2 project, in which approximately two dozen effects are being examined in more than 100 unique samples, is also progressing. This project not only encompasses a direct replication of these important social and cognitive psychology findings, but it is also designed to provide insight into how these effects vary across different samples and settings. It is generally assumed that the same psychological factors and forces come to bear differently on two different people, or in the same person at different times or in different environmental contexts. One of the goals of Many Labs 2 is parsing out how much variation between samples is attributable to these individual discrepancies and how much can be chalked up to sampling error.

The newest RRR project focuses on a 2002 article by Finkel, APS Fellow Caryl E. Rusbult, Madoka Kumashiro, and Peggy A. Hannon titled, “Dealing With Betrayal in Close Relationships: Does Commitment Promote Forgiveness?” Previous research had established that increased commitment in a close relationship predicts prorelationship behaviors such as forgiveness, but Finkel et al. were the first to show that commitment levels can be experimentally manipulated and that inducing greater commitment through this manipulation led to greater levels of forgiveness.

The RRR initiative has now been shown to be an effective way to reliably estimate the true effect size for influential psychological phenomena, but as Schooler noted in his commentary, it is also an illuminating glimpse into the inner workings of the scientific process itself. “The outcome of the [verbal overshadowing] replication effort is in my opinion a genuine victory for the emerging field of metascience — the approach of turning the lens of science onto itself,” Schooler wrote. Not only did the results substantiate evidence for the effect, but they also “simultaneously reveal[ed] a number of factors that may have contributed to challenges in replicating verbal overshadowing in the past.”

Finkel is among those who see the value in the RRR process regardless of the result: “If this mechanism reveals a low replication rate, that would provide compelling support for the dire claims,” Finkel says, referring to what has been termed by some a “replicability crisis” in psychological science. “If it does not, that would suggest that the field may be on much stronger footing than many have feared.”