Great Results in the Psych Lab—But Do They Hold Up in the Field?

How well do findings in the psychology lab generalize to real life? This criterion—“external validity”—is probably the most important for experimental psychology. So it was good news when, in 1999, Craig A. Anderson and his colleagues compared laboratory and field research on 38 topics in 21 meta-analyses (or analyses of numerous other studies), and found a lot of agreement between the results of the two. Greg Mitchell, a social psychologist at the University of Virginia School of Law, wanted to know if these findings hold up in a bigger sample—and whether there were differences among different kinds of psychological research. So, in a new paper published in Perspectives on Psychological Science, a journal of the Association for Psychological Science, Mitchell replicated the Anderson study with 217 lab-field comparisons from 82 meta-analyses, in such areas as industrial-organizational (I-O), social, consumer, and developmental psychology.

The results: “On one level, there is good news: a high degree of correspondence between findings observed in the lab and those found in the field,” Mitchell says.  “But if you look more closely, there are major variations. I-O led the pack by a long shot, social psychology did worse, and most other sub-disciplines fell somewhere in between.”  If you extract the I-O stats from the batch, the overall correlation between lab and field results drops considerably.  And in a relatively small but concerning number of comparisons, 30 of the 217, the results in the field were the opposite of those in the lab.  Of these reversals, the majority came from social psychology.   Laboratory studies of gender differences fared particularly poorly when results were tested under more realistic conditions.

Concludes Mitchell: “It’s not really helpful to think in broad terms about external validity. It is a concept you have to take finding by finding, setting by setting.”

The problem—and the solution—may derive from two ways of creating experimental “realism,” Mitchell says. In I-O, which is concerned with workplace dynamics such as management and productivity, psychologists often aim for “mundane realism”: they “try to bring the field into the lab.” In social psychology and other sub-disciplines, experimenters are partial to “psychological realism”: “They create a world in the lab and try to activate the same feeling or thoughts inside the lab but not with the same stimuli as outside the lab.”

In lab experiments, moreover, it’s usually the simplest, most controlled conditions that are considered the most rigorous. The authors of such studies have the most confidence in the causality of the relationships they observe. But of course real life is anything but simple and controlled. “We may be oversimplifying things so much that we’ve lost some important variables in the translation,” says Mitchell.

The lesson? “We need to be conducting more field studies,” and in the lab, aiming for more mundane realism, he says. “Because there’s a nontrivial chance the lab will point us in the wrong direction.”