Replications Don’t Hinge on Sample and Setting Differences, Multilab Project Shows

Failures to reproduce psychological research findings are often attributed to differences in the study population being examined. But results from a massive research project have upended that claim.

A report on the international project, which involved replications of 28 classic and contemporary findings in psychological science, appears in Advances in Methods and Practices in Psychological Science. A team of 190 researchers involved in the effort found that population characteristics had little to no bearing on the failure of a finding to replicate.

The project, called Many Labs 2, was designed specifically to address the argument that variations in study samples may result in a failed replication. Each of the 28 studies was repeated in more than 60 labs across 36 nations and territories, and collectively generated sample sizes that on average were more than 60 times larger than the original samples.

Fourteen of the original findings replicated, although some at variable degrees across the different labs. But for the 14 studies that did not replicate, sample diversity had minimal if any effect on the results.

“We were surprised that our diversity in our samples from around the world did not result in substantial diversity in the findings that we observed” said Rick Klein, a researcher at the University of Grenoble Alpes in France and one of the project leaders, in a press release. “Among these studies at least, if the finding was replicable, we observed it in most samples, and the magnitude of the finding only varied somewhat depending on the sample.”

The studies selected included many studies published within the last 20 years but also some classics in the research literature, including the well-known framing effect on choices, identified by APS William James Fellows Amos Tversky and Daniel Kahneman; and a 1977 finding on the false consensus effect, in which people overestimate the consensus around their own beliefs and preferences. (Both of those findings were reproduced, although the framing effect proved only half as strong in the replications compared to the original finding.)

Among the findings that did not replicate were a 2014 study showing that people’s belief in climate change can be increased by priming heat-related words and a 2009 study showing that people prone to feelings of disgust are also more likely to view portrayals of gay relationships in media as intended to make a statement.

Many Labs 2 represents a far greater undertaking compared to the first Many Labs project from 2013, in which 36 labs collaborated to examine 13 findings — replicating 10 of them. As part of the initiative, the collaborating labs collected the original materials from each study and had the experimental procedures peer-reviewed in advance by experts and, in some cases, authors on the original work.

The results of the latest project do not definitively mean that did not replicate findings were invalid, said Michelangelo Vianello, a professor at the University of Padua in Italy and another of the project leads.

But, he added, “they do suggest that they are not as robust as might have been assumed. More research is needed to identify whether there are conditions in which the unreplicated findings can be observed. Many Labs 2 suggests that diversity in samples and settings may not be one of them.”

The pre-press version of the article, “Many Labs 2: Investigating Variation in Replicability Across Sample and Setting,” along with commentaries, is available online.

References:

Inbar, Y., Pizarro, D.A., Knobe, J., Bloom, P. (2009). Disgust sensitivity predicts intuitive disapproval of gays. Emotion, 9(3), 435-439, doi: 10.1037/a0015960

Klein, et al. (2018). Many Labs 2, Investigating variation in replicability across sample and setting. Advances in Methods and Practices in Psychological Science. Preprint, doi/full/10.1177/0956797618805750.

Tversky, A., Kahneman, D. (1981). The framing decisions and the psychology of choice. Science, Jan 30; 211(4481):453-458.

Zaval, L., Keenan, E.A., Johnson, E.J., Weber, E.U. (2014). How warm days increase belief in global warming. Nature Climate Change 4(2), 143-147.