Psychology’s Replication Crisis Has Made The Field Better

In 2012, psychologists Will Gervais and Ara Norenzayan published a paper in the journal Science reporting a series of experiments that suggested engaging in analytical thinking could reduce someone’s religious belief. It sounded vaguely plausible, but five years later, another group of researchers attempted to replicate the finding. They used a sample size about two and a half times larger and found no evidence that analytic thinking caused a decrease in religious belief.

“Is it fun to find out that a study you published in a high profile outlet back in the day does not hold up well to more rigorous scrutiny? Oh hell no,” Gervais wrote in a blog post. He recommended that other researchers avoid the experience by using rigorous methods upfront. “CHECK YOURSELF BEFORE YOU WRECK YOURSELF while being open to REVISING YOUR BELIEFS.”

Being open to revising beliefs in the face of new evidence is, of course, a central tenet of the scientific enterprise. But research is done by fallible human beings who don’t always live up to scientific principles. When psychology first entered a period of upheaval commonly referred to as the “replication crisis,” not everyone in the field shared Gervais’s openness to updating. But as the field has reckoned with replication issues, it has been forced to follow its own rules more closely.

The replication crisis arose from a series of events that began around 2011, the year that social scientists Uri Simonsohn, Leif Nelson and Joseph Simmons published a paper, “False-Positive Psychology,” that used then-standard methods to show that simply listening to the Beatles song “When I’m Sixty-Four” could make someone younger. It was an absurd finding, and that was the point. The paper highlighted the dangers of p-hacking — adjusting the parameters of an analysis until you get a statistically significant p-value (a difficult-to-understand number often misused to imply a finding couldn’t have happened by chance) — and other subtle or not-so-subtle ways that researchers could tip the scales to produce a favorable result. Around the same time, other researchers were reporting that some of psychology’s most famous findings, such as the idea that “priming” people by presenting them with stereotypes about elderly people made them walk at a slower pace, were not reproducible.

Read the whole story (subscription may be required): FiveThirtyEight