Do Great Results in the Lab Hold Up in the Field?

It was good news in 1999, when Craig A. Anderson and his colleagues compared laboratory and field research on 38 topics in 21 meta-analyses and found a lot of agreement between the results. Greg Mitchell, a social psychologist at the University of Virginia School of Law, wanted to know if these findings would hold up in a bigger sample. In a paper published in Perspectives on Psychological Science, Mitchell replicated the Anderson study with 217 lab-field comparisons from 82 meta-analyses, in such areas as industrial-organizational (I-O), social, consumer, and developmental psychology.

The results: “On one level, there is good news: a high degree of correspondence between findings observed in the lab and those found in the field,” Mitchell says.  “But if you look more closely, there are major variations. I-O led the pack by a long shot, social psychology did worse, and most other sub-disciplines fell somewhere in between.”

If you extract the I-O stats from the batch, the overall correlation between lab and field results drops considerably.  And in 30 of the 217 comparisons, the results in the field were the opposite of those in the lab.  Of these reversals, the majority came from social psychology. Laboratory studies of gender differences fared particularly poorly when results were tested under more realistic conditions.

The lesson? “We need to be conducting more field studies,” says Mitchell, and scientists should attempt to recreate as much of the field as possible in the lab, he says. “Because there’s a nontrivial chance the lab will point us in the wrong direction.”