Brain Reconciles Sight and Sound in Different Ways

A new study from psychology researchers at UCLA provides insights into how the brain combines sound and vision. The research suggests that there is not one sole mechanism in the brain that governs how much our senses work together to process information.

Among the implications of the study: It might not be as easy as many people had assumed to categorize the way in which we perceive and learn.

“We should be cautious not to make blanket statements about how we process information, like ‘I’m a visual learner,’” said Ladan Shams, an associate professor of psychology in the UCLA College and senior author of the research. “That’s not necessarily true across the board. For example, your brain may combine sights and sounds a lot in one task — watching TV, for example — but only a little in another task — such as playing the piano.”

The researchers found that people’s vision frequently influenced their hearing when they tried to identify the specific location of sounds and flashes of light, and that their hearing influenced vision when they counted the sounds and flashes.

In one part of the study, 59 participants, mostly UCLA undergraduates, were seated in front of a computer monitor with speakers on either side and asked to count the number of flashes of light on the screen and beeps played on the speakers. Sometimes they only saw flashes, sometimes they only heard beeps and sometimes they both saw flashes and heard beeps — in which case the numbers could vary, up to four of each. The researchers presented 360 combinations of beeps and flashes in a one-hour period.

“When people have to process different numbers of beeps and flashes, it’s really hard — the senses blend together,” said Brian Odegaard, a UCLA postdoctoral scholar who was the study’s lead author. “Most people, when presented with two beeps and one flash, mistakenly said they saw two flashes, while a few participants could accurately tease apart lights and sounds.”

Shams explained that most people are pretty good at identifying visual stimuli if no other senses are involved, but in the experiment, what participants saw was typically affected by what they heard.

“When there’s sound, many people get fooled into thinking that there are fewer or more flashes,” she said.

In the second part of the study, subjects were asked to sit facing a black screen, behind which were five speakers. A projector mounted overhead was used to flash bursts of light onto the screen, at the same spots where the speakers were located.

The researchers played brief bursts of sound and triggered flashes of light, in various combinations, and asked participants to identify where they originated. The participants used a computer mouse to point at the part of the screen where they thought the sounds and flashes occurred; each was presented with 525 trials.

When the flash and sound occurred in different locations from each other, most participants struggled to correctly identify where the sound occurred. The effect was similar to what happens when people watch a ventriloquist with a puppet or a magician using misdirection to pull off an illusion, Odegaard said.

Given that the two tasks both involved brief sights and sounds, the researchers figured that the people who were fooled less on one task would be fooled less on the other as well, but the researchers were surprised to find no correlation in how much subjects combined the two senses on the two tasks.

A week later, the participants were given both of the same tasks again. The results were the same—although they performed consistently on each individual task, their abilities to bind the two senses in the two different tests showed no correlation.

People vary in how much their hearing and vision interact, Shams explained. For example, some find it easier to understand someone who is speaking when they are looking at the speaker. Some are better able, when driving, to estimate how fast other vehicles are traveling using both visual and aural cues.

“This finding suggests that if you find a strategy of combining your senses that helps you perform a given task, stick with it,” she said.

Reference

Odegaard, B., & Shams, L. (2016). The brain’s tendency to bind audiovisual signals is stable but not general. Psychological Science. doi: 10.1177/0956797616628860