A Change of Face: New Approaches to Measuring Face Recognition and Identification

This feature is adapted from the article “Recognizing Faces,” appearing in the June issue of Current Directions in Psychological Science.

Many of us believe we’re rather good at recognizing faces. After all, few of us would fail to recognize Meryl Streep, Barack Obama, or Scarlett Johansson in a variety of photographs or video clips.

But the ease we have when identifying such familiar faces seems to have overshadowed our limitations in a different type of face perception: Our ability to identify relative strangers is generally rather weak.

Over the last several years, it’s become clear that unfamiliar-face recognition is a problem worthy of study in its own right, in large part because of its role in law enforcement and security settings. These situations include:

• photo-to-photo matching for an image of a suspect to a surveillance camera image from a crime scene;
• person-to-photo matching at passport control; and
• person-to-photo-ID matching to verify the age of people purchasing items such as alcohol, cigarettes, or guns.

Field studies have demonstrated that comparing a photo of a face to a person standing in front of us is not as easy as we imagine. In a 1997 psychological study conducted in a supermarket, for example, cashiers who were given an incentive to spot fake-photo credit cards challenged roughly 10% of confederate “shoppers” who presented valid cards and accepted 64% of the invalid cards if the customer looked similar to the person pictured.

Unfamiliar-face matching is also a problem for professionals who rely on this ability. This includes passport officers, who typically use photo identification when processing people’s applications for passports. In 2014, one of us (Burton) and colleagues showed that passport-issuing officers in Sydney had a 10% overall error rate when asked to verify photos of applicants. Despite extensive training, highly experienced officers performed no better, on average, than new recruits.

Lab studies have not always predicted such failure rates, and the reason may lie in the typical design of such experiments. Specifically, lab studies often use only one or two standardized photographs of each face throughout the experiment, failing to capture the way an individual’s appearance typically varies from picture to picture based on lighting, angle, pose, or even hairstyle.

Recognizing Pictures Rather Than Faces

Modern research on face recognition can be traced back to the seminal 1975 review by psychological scientist Hadyn Ellis, which brought together a wide range of studies of normal adults, infants, and children — as well as research on the effects of brain injury. Ellis noted remarkably high levels of correct recognition of faces in standard memory recognition tasks. In such experiments, researchers typically had participants study photographs of people they’d never seen before, then — in a test phase — had them pick out those photos when they were mixed with pictures of other unfamiliar faces. Studies that followed that review generated similar results, even when participants were able to study each face for only a few seconds.

But those lab results failed to generalize to real-world settings, particularly in the criminal justice system, where eyewitnesses were discovered to have misidentified perpetrators of various crimes despite insisting that they were correct.

APS Fellow Vicki Bruce of Newcastle University in the United Kingdom was among the first researchers to show why lab results were leading researchers to overestimate human performance with unfamiliar faces. In 1982, Bruce published the results of a study in which the faces that some of the participants viewed changed in expression, angle, or both between presentation and the actual testing phase. She found that participants were slower to recognize an unfamiliar face if it changed during the test; in other words, they seemed to be better at memorizing still pictures rather than changing faces. Subsequent studies have produced similar results.

A powerful insight into the nature of this problem derives from an experimental sorting task devised by our lab and researchers from the United Kingdom and Australia. Participants were given a set of 40 everyday photographs showing unfamiliar faces and asked to sort the pictures into separate piles for each individual. Only two faces were shown, but the participants sorted their pictures into anywhere from three to 16 different piles, with nine piles being the most common number. In other words, participants typically thought that the set of 40 photos depicted nine different individuals when there were actually only two.

This finding is particularly interesting because it runs counter to the widely accepted intuition that people mainly struggle to tell similar faces apart. Rather, participants are likely to see photos of unfamiliar faces as more diverse than they actually are, as the aforementioned study shows. The problem is as much one of seeing that very different images can represent the same unfamiliar identity as it is one of telling faces apart.

Importantly, real-life views of faces are highly variable, whether they’re seen in person, in videos, or in photographs. This variability might result from within-person variability (e.g., differences between various views of the same face) or between-person variability (e.g., differences between similar views of different faces). This variability does not cause problems if a face is familiar, but has a substantial impact for unfamiliar faces.

In these two pairs of images from the Glasgow Face Matching Test, participants are asked whether each pair shows the same person or different people. The top pair shows the same person, and the bottom row shows two different people. The individuals pictured have given written consent for use of their pictures. Results from numerous studies show a wide variation in study participants’ ability to match and distinguish faces correctly. © A. Mike Burton

Standard face-memory tests, such as the Recognition Memory Test and the Cambridge Face Memory Test, don’t specifically measure unfamiliar-face-matching ability. That’s what drove the development several years ago of a tool called the Glasgow Face Matching Test.

This psychometric test uses front-facing images of people taken with two different cameras. Subjects are shown photo pairs and asked to determine whether they represent two images of the same person or two different people. Typically, 15% to 20% of their matches are wrong, but there are substantial individual differences, and the measure also has been used to identify super-recognizers — people who show enormous accuracy in remembering faces.

The findings show that image variability can easily confuse the visual system when people look at unfamiliar faces. This seems curious given that we see so many faces in everyday life. But variability is, to some extent, idiosyncratic — that is, the ways in which Face A varies across different images need not be the same as the ways in which Face B varies across different images. Our lab analyzed the properties of face images both within and between identities. Consistent with previous findings, the largest variations, common to all the faces, involved physical differences such as pose and lighting direction. Once those common variations were removed, the differences among images of each individual were highly person-specific. Thus, learning a new face entails learning how that particular face varies.

In Good Light and in Bad

This idiosyncratic changeability of faces explains how we can be “expert” at recognizing one face but not another. Learning how the face of actor Brad Pitt can vary — through seeing him in many settings over the course of his career — allows us to recognize new photos of him even when he’s caught with an unusual expression or in poor lighting. This “expertise” may not generalize to another person’s face, however, because that person will vary in different ways across images. This is why it’s easy to match two photographs of a familiar person but highly difficult to match two photographs of an unfamiliar person, whose range of variability we don’t know.

It also explains why attempts to train people to become better at pairing images of unfamiliar faces are likely to fail. Our research has demonstrated that observers can be trained to recognize a particular new face very well, but that this training will not generalize to recognizing a different face.

There are other aspects of unfamiliar-face perception in which people do show expertise, however. For example, people can usually make a reasonably accurate estimate of a person’s age or sex from his or her face, and they can interpret subtle differences in facial expression and gaze direction. What characterizes such abilities is that, unlike face recognition, they involve cues that are highly consistent across many different faces.

It is clear that considering image variability is critical to understanding face recognition. We need to study naturally occurring, or “ambient,” images. Experimentally using standardized images in an attempt to minimize the impact of “nuisance” variation — such as lighting and camera differences — can limit our understanding of face recognition. By using the full range of ambient images of the type people encounter every day in newspapers, on television, and online, we can preserve the natural within-person variations that characterize our real-world experience of faces.

References and Further Reading

Bruce, V. (1982). Changing faces: Visual and non-visual coding processes in face recognition. British Journal of Psychology, 73, 105–116.

Burton, A. M. (2013). Why has research in face recognition progressed so slowly? The importance of variability. The Quarterly Journal of Experimental Psychology, 66, 1467–1485.

Burton, A. M., White, D., & McNeill, A. (2010). The Glasgow Face Matching Test. Behavior Research Methods, 42, 286–291. doi:10.3758/BRM.42.1.286

Ellis, H. D. (1975). Recognizing faces. British Journal of Psychology, 66, 409–426.

Jenkins, R., White, D., van Montfort, X., & Burton, A. M. (2011). Variability in photos of the same face. Cognition, 121, 313–323.

Kemp, R., Towell, N., & Pike, G. (1997). When seeing should not be believing: Photographs, credit cards and fraud. Applied Cognitive Psychology, 11, 211–222.

Ritchie, K. L., & Burton, A. M. (2017). Learning faces from variability. The Quarterly Journal of Experimental Psychology, 70, 897–905.

White, D., Kemp, R. I., Jenkins, R., Matheson, M., & Burton, A. M. (2014). Passport officers’ errors in face matching. PLoS ONE, 9, e103510. doi:10.1371/journal.pone.0103510

Young, A. W., & Burton, A. M. (2017). Recognizing faces. Current Directions in Psychological Science, 26, 212–217. doi:10.1177/0963721416688114