Weaknesses in Emotion-Expression Research Outlined in New Report

Software that purports to read emotions in faces is being deployed or tested for a variety of purposes, including surveillance, hiring, clinical diagnosis, and market research. But a new scientific report finds that facial movements are an inexact gauge of a person’s feelings, behaviors, and intentions.

“It is not possible to confidently infer happiness from a smile, anger from a scowl or sadness from a frown, as much of current technology tries to do when applying what are mistakenly believed to be the scientific facts,” a group of leading experts in psychological science, neuroscience, and computer science write in their comprehensive research review.

The report appears in Psychological Science in the Public Interest, and is authored by APS President Lisa Feldman Barrett of Northeastern University, Ralph Adolphs of the California Institute of Technology, Stacy Marsella of Northeastern University and the University of Glasgow, Aleix M. Martinez of The Ohio State University, and APS Fellow Seth D. Pollak of the University of Wisconsin-Madison.

The authors note that the general public and some scientists believe that there are unique facial expressions that reliably indicate six emotion categories: anger, sadness, happiness, disgust, fear, and surprise. But in reviewing more than 1,000 published findings about facial movements and emotions, they found that typical study designs don’t capture the real-life differences in the way people convey and interpret emotions on faces. A scowl or a smile can express more than one emotion depending on the situation, the individual, or the culture, they say.

“People scowl when angry, on average, approximately 25% of the time, but they move their faces in other meaningful ways when angry,” Feldman Barrett explains. “They might cry, or smile, or widen their eyes and gasp. And they also scowl when not angry, such as when they are concentrating or when they have a stomach ache. Similarly, most smiles don’t imply that a person is happy, and most of the time people who are happy do something other than smile.”

In a separate article in the journal, Alan Cowen and APS Fellow Dacher Keltner of the University of California, Berkeley, Disa Sauter of the University of Amsterdam, and APS Fellow Jessica L. Tracy of the University of British Columbia note that most scientists agree that facial expressions are meaningful, even if they don’t follow a one-to-one match with the six basic emotion categories. They propose a new model for studying emotion-related responses in all their complexity and variations. This approach would measure not only facial cues, but also body movements, voice fluctuations, head movements and other indicators to capture such nuanced responses as smiles of embarrassment or sympathetic vocalizations, they say.

“We thought this was an especially important issue to address because of the way so-called ‘facial expressions’ are being used in industry, educational and medical settings, and in national security,” Feldman Barrett and her coauthors say.

The report was featured in The Washington Post, NBC News, Forbes, The Verge, Fast Company, MIT Technology Review, and ACLU’s Free Future blog.