Psychology Is in Crisis Over Whether It’s in Crisis

Wired:

LAST SUMMER, THE field of psychology had a moment—possibly one of the most influential events in science last year. On August 27, 2015, a group called the Open Science Collaboration published the results of its Reproducibility Project, a three-year effort to re-do 100 psychology studies. Replication is, of course, one of the fundamental tenets of good science. The group wanted to see how many of the original effects they could replicate. The result: It only worked about 40 percent of the time.

That did not go over well. But now the psychology establishment is fighting back. Along with some colleagues,Dan Gilbert, a psychologist at Harvard University, has re-analyzed the paper about re-analyzing papers, and they say that it’s wrong. And in fact, the public’s conclusions about the paper—that psychology is in crisis—are even wronger. More wrong. “We’re arguing with virtually every journalist we know that wrote some version of ‘psychology’s in deep trouble,’” Gilbert says. The comment on the Reproducibility Project appears in Science today, an attempt to reinterpret the data and highlight what the researchers see as flaws. Their conclusion: Reproducibility in psychology is doing great.

…

Nosek and his colleagues respond to those criticisms in turn, chipping away at some of the statistical analysis in the comment. To the endorsement question, they countered that a scientist could decline to thumbs-up a replication for many reasons—not just, as implied, low confidence in the quality of the replication methodology. An original researcher could just as easily decline because they weren’t confident in their own original results.

…

“That’s what’s at issue here,” says Barrett. “How to we develop a generalizable science?” To do that, the field might have to change its understanding of what it means to find a meaningful effect. That could involve coping with the challenges of assembling larger sample sizes, and controlling for many more factors than it could possibly think are necessary. The field may have to think differently about how it thinks about itself.

Is psychology in the middle of a replication crisis? “No,” says Barrett. “But it’s in a crisis of philosophy of science.”

Read the whole story: Wired