New Content from Advances in Methods and Practices in Psychological Science

Many Labs 5
To assess the role of poor replication designs in failures to replicate original results, several researchers from different labs attempted to replicate 10 Reproducibility Project: Psychology (RP:P) studies that were “not endorsed” and did not replicate the original findings. For each study, the researchers: 
1.     Revised the RP:P study protocol and subjected it to peer review; 
2.     Preregistered the protocols on the Open Science Framework, after the new protocols were accepted; 
3.     Conducted replications using both the revised protocols and the RP:P protocols.

The following summaries describe the effects found in the original studies, the differences among replication protocols, and the replication results.

Note: The differences among replication protocols were described by Charles R. Ebersole and colleagues in their article Many Labs 5: Testing Pre-Data-Collection Peer Review as an Intervention to Increase Replicability (Table 1), also part of this collection of articles.

Many Labs 5: Registered Replication of Albarracín et al. (2008), Experiment 5
Christopher R. Chartier et al.

Effect in the original study: Participants primed with concepts related to action demonstrated more cognitive output on a subsequent cognitive test than did those primed with concepts related to inaction.
Main differences between the Reproducibility Project: Psychology (RP:P) protocol and the revised protocol in ML5: In the ML5 RP:P protocol, participants were Amazon Mechanical Turk workers who completed the experiment online; in the revised protocol, participants were undergraduates who were tested in the lab.
Replication Results: The best estimated effect size for increased cognitive output following action primes versus inaction primes lies between the effect-size estimates of Albarracín et al. and the original replication.

Many Labs 5: Registered Replication of Albarracín et al. (2008), Experiment 7
Katherine S. Corker et al.

Effect in the original study: Participants primed with action or inaction goals, which are subsequently fulfilled or not, behave more or less actively in a later task compared with participants who receive a neutral prime.
Main differences between the Reproducibility Project: Psychology (RP:P) protocol and the revised protocol in ML5: The RP:P replication study was conducted in German; both ML5 studies were conducted in English. The revised protocol used scrambled sentences instead of word fragments to prime goals whereas the ML5 RP:P protocol used word fragments.
Replication Results: Scrambled sentences were a more effective priming method than word fragments, but heightened accessibility stemming from primes may not translate into downstream consequences for behavior on a thought-listing task.

Many Labs 5: Replication of Crosby, Monin, and Richardson (2008)
Hugh Rabagliati et al.

Effect in the original study: Subjects who had just heard an offensive remark looked longer at a potentially offended person, but only if that person could also hear the remark (i.e., social-referencing effect).
Main differences between the Reproducibility Project: Psychology (RP:P) protocol and the revised protocol in ML5: In the ML5 revised protocol, participants were presented with the experimental scenarios after they watched a video about affirmative action. The ML5 RP:P protocol did not include the video about affirmative action.
Replication Results: The researchers obtained the same effect, albeit smaller, and found that making subjects more aware of the offensive nature of the remark did not increase it—the protocol manipulation did not affect the size of the social-referencing effect.

Many Labs 5: Registered Replication of Förster, Liberman, and Kuschel’s (2008) Study 1
Hans IJzerman et al.

Effect in the original study: People assimilate a primed concept (e.g., “aggressive”) into their social judgments after a global prime (e.g., they rate a person as being more aggressive than do people in a no-prime condition) but contrast their judgment away from the primed concept after a local prime (e.g., they rate the person as being less aggressive than do people in a no prime-condition).
Main differences between the Reproducibility Project: Psychology (RP:P) protocol and the revised protocol in ML5: In ML5 stimuli for the revised protocol were pilot-tested for ambiguity and applicability; the RP:P protocol used the same stimuli as the previous RP:P replication study.
Replication Results: The researchers did not find moderation by protocol type, and judgment patterns in both protocols were inconsistent with the effects observed in the original study.

Many Labs 5: Registered Replication of LoBue and DeLoache (2008)
Ljiljana B. Lazarevic ́ et al.

Effect in the original study: Both young children and adults exhibit enhanced visual detection of evolutionarily relevant threat stimuli (e.g., snakes) relative to nonthreatening stimuli (e.g., frogs, caterpillars).
Main differences between the Reproducibility Project: Psychology (RP:P) protocol and the revised protocol in ML5: In ML5, the revised protocol: used frogs as control stimuli whereas the RP:P protocol used caterpillars as control stimuli; sampled only 3-year-olds along with their parents, instead of 3- to 5-year-olds, as in the RP:P protocol; used software that allowed the study to be run offline and therefore not hampered by Internet speed, as in the original studies.
Replication Results: Overall, participants were not significantly faster at detecting threatening stimuli. The effect from the RP:P protocol was similar to the effect from the revised protocol, and the results from both the RP:P and the revised protocols were more similar to those found in the original replication than to those found in the original study.

Many Labs 5: Registered Replication of Payne, Burkley, and Stokes (2008), Study 4
Charles R. Ebersole et al.

Effect in the original study: When participants were put under high pressure to respond without bias, the correlations between their implicit and explicit race attitudes were weaker, relative to when they were placed under low pressure.
Main differences between the Reproducibility Project: Psychology (RP:P) protocol and the revised protocol in ML5: In the ML5 RP:P protocol, the study was conducted in Italy, with materials in Italian; in the revised protocol, data were collected in the United States with materials in English.
Replication Results: The researchers detected a moderation by the sample’s country, but it was due to a reversal of the original effect in the United States and a lack of the original effect in Italy.

Many Labs 5: Registered Multisite Replication of the Tempting-Fate Effects in Risen and Gilovich (2008)
Maya B. Mathur et al.

Effect in the original study: Subjects believed that “tempting fate” (e.g., not doing homework) would be punished with ironic bad outcomes (e.g., being called upon in class), and that this effect was magnified when subjects were under cognitive load.
Main differences between the Reproducibility Project: Psychology (RP:P) protocol and the revised protocol in ML5: The ML5 revised protocol used undergraduates at elite universities, as in the original study whereas the RP:P protocol used Mechanical Turk workers.
Replication Results: Results from the university sites used in Many Labs 5 did not support the original effects. However, the original study’s results would not have been extremely unlikely in the estimated distribution of population effects. The results obtained from a new Mechanical Turk sample under the first replication study’s protocol were not meaningfully different from those obtained with the new protocol at universities similar to the original site.

Many Labs 5: Registered Replication of Shnabel and Nadler (2008), Study 4
Erica Baranski et al.

Effect in the original study: Participants’ willingness to reconcile improved after they received a message of acceptance or empowerment, although victims’ willingness to reconcile was greater in response to the non–theoretically prescribed message than in response to the theoretically prescribed message.
Main differences between the Reproducibility Project: Psychology (RP:P) protocol and the revised protocol in ML5: In the ML5 revised protocol, participants read a vignette about a student whereas in the RP:P protocol, participants read a vignette about an employee. This revision was meant to provide a more relatable experience regarding being the victim or perpetrator of a transgression.
Replication Results: Partial replication was observed only when participants received the revised and peer-reviewed version of the original protocol. These results underscore the importance of thoughtful consideration of how the content of the replication materials will be received by the population from which participants are selected.

Many Labs 5: Replication of van Dijk, van Kleef, Steinel, & van Beest (2008)
Lauren Skorb et al.

Effect in the original study: Subjects offered fewer chips to angry bargaining partners than to happy partners when the consequences of rejection were low.
Main differences between the Reproducibility Project: Psychology (RP:P) protocol and the revised protocol in ML5: Following the original study, the revised protocol excluded subjects who had prior experience with psychology or economics. Contrary to the RP:P study, participants also could not see or hear one another during the experiment.
Replication Results: The effect averaged one ninth the size of the originally reported effect and was significant only for the revised protocol. However, the difference between the original and revised protocols was not significant.

Many Labs 5: Registered Replication of Vohs and Schooler (2008), Experiment 1
Nicholas R. Buttrick et al.

Effect in the original study: Participants reading from a passage “debunking” free will showed decreased belief in free will and thus cheated more on experimental tasks than did those reading from a control passage.
Main differences between the Reproducibility Project: Psychology (RP:P) protocol and the revised protocol in ML5: The revised protocol used a different free-will-belief induction than the RP:P protocol did (a rewriting task instead of a reading task; text in the two protocols was pulled from the same source). Also, the revised protocol used a revised measure of free-will beliefs.
Replication Results: The researchers found that the protocol revisions did not matter. Free-will beliefs were unchanged by the manipulations, and in the focal test, there were no differences in cheating behavior between conditions. Expressed free-will beliefs did not mediate the link between the free-will-belief manipulation and cheating, and participants expressing lower beliefs in free will were not more likely to cheat in our task.