What Type of Templates Do We Use for Visual Processing? Caricatures Might Be the Answer

In this episode, Zekun Sun and Chaz Firestone from Johns Hopkins University join Under the Cortex to discuss their new paper in Psychological Science titled “Caricaturing Shapes in Visual Memory.” The conversation with APS’s Özge G. Fischer Baum reviews how our visual system uses templates and exaggerates the basic features of objects in memory. The authors describe a series of experiments that show how visual memory is more complex than previously thought and that visual details in memory are not always lost. 

Send us your thoughts and questions at [email protected].

Unedited transcript

[00:00:00.270] – APS’s Özge Gürcanlı Fischer Baum

This is Under the Cortex. Today, we explore the world of visual processing. How do you think our brains perceive and remember what we see in the real world? Do our minds exaggerate the most distinctive features of objects? How does mental caricaturing leave traces in our memory? I am Özge Gürcanlı Fischer Baum with the Association for Psychological Science. I’m joined by Zekun Sun and Chaz Firestone from Johns Hopkins University. They have a recent article on this topic published in Psychological Science. Together, we will explore how mental caricaturing works in visual processing. Zekun and Chaz, welcome to Under the Cortex.

[00:00:51.430] – Chaz Firestone

Thank you for having us.

[00:00:52.430] – Zekun Sun

Thank you.

[00:00:53.690] – APS’s Özge Gürcanlı Fischer Baum

Let me start with our first question. What type of psychologist are you?

[00:00:59.120] – Zekun Sun

I’ll go first. Psychologists are people who are curious about how minds work. For me, I share the general interests about the human mind, and meanwhile, I’m especially intrigued by the questions about human vision, like how do people see the world and how do they make sense of what they see?

[00:01:21.950] – APS’s Özge Gürcanlı Fischer Baum

Great. What about you, Chaz?

[00:01:24.150] – Chaz Firestone

I think Zekun is exactly right that different psychologists tend to get interested in different parts of the mind. For the For the two of us, that part is often the visual system. But zooming out a bit, a theme across the mental processes that I personally find most fascinating is that they’re often the ones that we tend to take for granted and almost find easy to carry out. When you make a complicated decision or you answer a difficult math question or you play chess or something like that, those are really interesting problems, but those are problems that not only are difficult, but also feel difficult when we do them. But then there are other processes that aren’t like that. When you see something, or you remember something, or you understand the words that I’m seeing right now, those things aren’t that hard to do. And yet we know that under the hood, there’s a ton of sophisticated processing that makes them possible. I think a lot of the most interesting questions in psychology are really about that. How do our minds make really difficult problems feel so easy to do? The psychologists that Zekun and I are the ones that are interested in those kinds of problems.

[00:02:25.710] – APS’s Özge Gürcanlı Fischer Baum

You are both visual scientists. Then let me ask you this, what initially got you interested in studying visual processing?

[00:02:34.410] – Zekun Sun

For me, talking about my professional journey, I completed my PhD training at Johns Hopkins with Professor Chaz Firestone, and I’m now trained as a postdoc at Yale with Professor Samuel McDougle. Before all this, I did a master in cognitive psychology in the Chinese Academy of Sciences, where I studied the experience of pain with a focus on the visual attention. I guess from there, I really developed my special interests in vision.

[00:03:08.430] – Chaz Firestone

To take what I said earlier a step farther, the visual system might well be the part of our mind that we use the most often. So anytime we’re awake, we’re basically seeing. And yet from the inside, we can’t really tell or feel how our visual processing works. So you don’t feel your retinal ganglion cells detect edges. You don’t feel your color vision system adjust for the lighting conditions that you’re in and so on. But we now have a really deep understanding of how those processes work. And so visual processing is just such a beautiful example of why the science of psychology is so necessary It’s an example of why we can’t just trust our own intuitions about how our minds work, not only because our intuitions often lead us astray, which is true in all domains of psychology, but also because just in many cases, we barely have intuitions about visual processes at all. That’s true not only a vision itself, but also its connection to other processes in the mind, including memory, which is, of course, the topic of Zekun’s project that we’re here to discuss today.

[00:04:10.510] – APS’s Özge Gürcanlı Fischer Baum

How did you get into the perceptual process of caricaturing? When I look at your study, I see that a lot of the research you cite about is not very recent.

[00:04:20.630] – Zekun Sun

The research of false memory and memory biases is not new. You’re right, we do cite classic papers memory biases and errors. For example, if you ask people to do a memory version of a telephone game, meaning that I show an ambiguous pattern to the first person and ask them to draw it from their memory, then I show this person’s drawing to the next person and let them also draw from their memory. Then repeating this process many times, you will see people start to something that makes more sense to them, such as an animal or an everyday object. This research is very interesting since it tells us when people see something, how do they really encode it into their mind? Do they, for example, add anything or remove anything, or do they have a model in their mind, they match it, too. In our case, we explore something even more basic. When we see novel and meaningless visual stimulus, such as a simple geometric shape, does our visual system distort it? If yes, does the memory really normalize or exaggerate it?

[00:05:48.430] – APS’s Özge Gürcanlı Fischer Baum

Yeah, great. Zekun, I would like to go back to what you said in pacing about basic shapes. Why did you choose to study caricaturing with basic shapes?

[00:06:00.910] – Zekun Sun

For us, basic shapes are more interesting than, let’s say, faces, animals, and everyday objects, because when you have an obvious prototype in your mind, your memory could be susceptible to those schemes. But what about a novel shape, like randomly generated by a computer? You cannot think of an obvious prototype for it, or you can say there could be a lot of possible schemes. Also, we can easily to really generate a whole bunch of such novel shapes and ask if there’s any memory bias across a wide range of shapes and visual features. So there were some really important reasons, both theoretically and practical. That led us to think basic geometric shapes could be a really interesting place to look for biases like this.

[00:06:55.180] – APS’s Özge Gürcanlı Fischer Baum

Yeah. Is there a reason why people might be prone to caricatur in their visual memories, not only faces, but something as basic as shapes?

[00:07:05.330] – Zekun Sun

I would say this could just be how the visual memory system works. One of our findings is that people don’t really have an explicit reason to do so. In fact, they didn’t even realize they kept remembering those shapes as a character version. So this suggests that this process is not like something deliberate and inferential. But more implicit and automatic. At the same time, this question can be still relevant if we ask whether the mind automatically encodes basic shapes in an exaggerated form. I think possible reasons could be, for example, by making visual features more silient, the mind may have a better chance to recognize it later. In other words, a character shape could be more distinguishable and thus to be more memorable. This could help people remember important information and perhaps better navigate their environment. But these are just possibilities that need to be tested in a future study.

[00:08:14.180] – Chaz Firestone

I might just add here that there’s this really interesting idea in the memory literature that your mind, in some sense, protects the individual memories you form by emphasizing what it is that makes them distinct from other memories you formed or might expect to form later. But most of the findings that are accrued in support of that theory come from cases in which the task that you give a subject is to place different memories in competition with one another, where the experimental paradigm makes it so that you actively have to avoid interference between one memory and another, or to encode one stimulus rather than another. And one thing that distinguishes our experiments from some of those approaches is that we really go out of our way to avoid those circumstances. So there’s very little competition at all in our studies, and yet we still find these effects, which is something that’s pretty intriguing.

[00:09:03.250] – APS’s Özge Gürcanlı Fischer Baum

Yeah, your study is really interesting. Let’s take a step back. What did you find in your experiments?

[00:09:09.420] – Zekun Sun

Our experiments basically ask people to view and memorize a series of basic random-looking shapes. For each shape, they first see and remember it in a second. After a very short delay, like 1.2 second, they reproduce the exact shape using a slide bar that adjusts the salience of its features. It sounds pretty simple, right? You just make this thing in front of you as same as the thing you just saw in a moment ago. However, across a few experiments, we found that people just kept making shapes to be more exaggerated than they truly are, or we can say they reproduce shapes in a captured form. Here, capture means the salience of shapes features such as the curvature of the contour, convex and concave angles, and other salient parts. If we showed you something curvy and round, the if you tended to reproduce was like, sharper and spikier.

[00:10:21.090] – APS’s Özge Gürcanlı Fischer Baum

How can you be sure your effects are due to caricaturing and not something else?

[00:10:26.110] – Zekun Sun

Right. That’s a question we, as scientists, We ask ourselves every time we have some new findings, right? So in this work, we made efforts to address alternative explanations. So even though we asked people to reproduce what they really saw, they might have other reasons to do it in a biased way. But think about this. If people just respond in a captured way instead of really remembering a captured shape, then you will also find similar biases in a non-memory task. This is what we did in one of our control experiments in which people reproduced the same set of shapes, but not from memory, but from perception. This time, they reproduced a shape when the target shape remains on the screen. Now the capture bias mostly goes away. Then we know that this bias is really driven by memory processing instead of some response preferences, just like some people may prefer seeing and making exaggerated things. This is not the only alternative explanation we dealt with in the paper. We also considered, for example, whether people have used any explicit strategy or if they are biased by the responding slide we used in experiment, you can find more details about those control experiments by reading our paper.

[00:12:00.360] – APS’s Özge Gürcanlı Fischer Baum

Yeah, thank you. I encourage our listeners to go to our website and look at your article under our journals. It is a great article. Let me ask you this, did anything in your study surprise you?

[00:12:16.920] – Zekun Sun

Yeah, to be honest, we didn’t expect a culture bias before collecting data. In fact, it is hard to make that prediction from previous literature because you see a lot of work showing that visual memory could lose details and get simpler over time. Our finding, in another sense, is showing that visual memory can be more complex than it is. The details are not lost but amplified. This is quite surprising to me.

[00:12:49.260] – APS’s Özge Gürcanlı Fischer Baum

Other studies show that people lose details in their memory over time. It is interesting here that you find the opposite. What do you think is different about perceptions on this level of encoding?

[00:13:02.020] – Zekun Sun

Yeah, just like what we all experienced in real life, we have some memories that fade away as time goes by. But there are also other memories that we feel as vivid and realistic as we first experience them. Even more strikingly, we sometimes remember an experience to be so exciting and marvelous, such as a sports game or a movie. But when When we revisit that game or that movie, guess what? It looks less great than we remembered. Chas has another paper about how the pictures in visual memory could have higher resolution than they really are which is actually inspired by our experience with video games. Think about a video game you played in your childhood. I bet if you go back and play it again, you may find it how much worse it is than what you memorized.

[00:14:06.210] – Chaz Firestone

To continue that thread a little bit, what Zekun is saying even matches lots of our intuitions, just as ordinary people, not just as scientists, So we tend to think that memories are some blurry or faded record of what we’ve seen. But there’s now lots of evidence that that’s just not how things work. Our memory systems are actually trying to rebuild what we’ve seen. And while sometimes what they rebuild is missing information that was originally there, there are other times that this rebuilding process actually adds detail that wasn’t present at the time you saw whatever it is that you’re recalling now. This is one of the situations. You’re remembering the image as being more informationally dense than it really was.

[00:14:50.940] – APS’s Özge Gürcanlı Fischer Baum

Does having a general bias towards exaggerating aspects of memories have implications for visual perception of more complex topics? What are the greater implications of your results?

[00:15:05.740] – Zekun Sun

Cetra bias in basic visual processing may imply an encoding mechanism underlying other types of memory distortions in more complex, sophisticated contexts. Information is first processed in the visual perceptual system and then in higher-level systems, such as long-term memory and other cognitive conceptual systems. In fact, we often think of memory biases and errors in those more complex and high-level system. So I want to mention one contribution we made with this work is to show that Character bias could even happen in a very short time scale and in basic visual processes. Encoding that character in mind could be more fundamental than we first think of.

[00:16:00.090] – APS’s Özge Gürcanlı Fischer Baum

Do you think anything in your studies can speak to why gossip stories get exaggerated and dramatically change over time?

[00:16:09.430] – Zekun Sun

Yeah, it feels like gossip and shapes the character share something similar in terms of exaggerating details of information. But I guess when gossip is spread in the crowd, sometimes people intentionally exaggerate the suppressing parts to make it even more like mind-blowing to capture attention or even achieve other goals. I feel suspicious that this behavior could be implicit and automatic in any sense, like what we showed with our work here. And also, it’s not clear whether gossiping really reveals how a person receives, remembers, or even understands information. Complex social motives could also involve here. Which I think might be less relevant to our work.

[00:17:04.600] – APS’s Özge Gürcanlı Fischer Baum

This was a great conversation. Thank you very much. This is really important research. Zekun and Chaz, thanks for joining Under the Cortex.

[00:17:13.270] – Chaz Firestone

Thanks for having us.

[00:17:14.220] – Zekun Sun

Yeah, thank you.

[00:17:15.670] – APS’s Özge Gürcanlı Fischer Baum

This is Özge Gürcanlı Fischer Baum with APS, and I have been speaking to Zekun Sun and Chaz Firestone from Johns Hopkins University. If you want to know more about this research, visit psychologicalscience.org. Would you like to reach us? Send us your thoughts and questions at [email protected].