Sound-Shape Associations Depend on Early Visual Experiences

Data from individuals with different types of severe visual impairment suggest that the associations we make between sounds and shapes — a “smooth” b or a “spiky” k — may form during a sensitive period of visual development in early childhood. The findings are published in Psychological Science.

“In congenitally blind participants, as well as in participants with a history of congenital or childhood developmental cataracts, we found no evidence of reliable sound-shape associations,” says lead author Suddha Sourav of the University of Hamburg. “However, in late permanently blind individuals whose blindness began after 12 years of typical vision, we observed a typical sound-shape association for touched objects that was not significantly different from the typically sighted control group.”

“The results provide evidence that commonly found associations between shapes and sounds depend on an extended developmental period during which high visual capabilities are required,” Sourav explains. “However, once acquired, sound-shape associations seem to be robust and are not extinguished by blindness.”

Philosophers and researchers alike have long been fascinated by the cross-modal associations humans almost universally make, linking features across sensory modalities. One of the most common cross-modal associations is sound-shape association as shown in the bouba–kiki effect, whereby people reliably associate pseudowords such as bouba with round shapes and pseudowords such as kiki with angular shapes.

Researchers have found evidence for the effect across cultural and ethnic groups, including the Namibian Himba tribe that does not use a written language, which raises the question of whether these associations are somehow innate or depend on sensory experience. Some studies have shown that sound-shape associations such as the bouba–kiki effect emerge very early in childhood, while other research indicates that individuals who are congenitally blind do not form these associations. Together with the new findings, the available evidence suggests that these cross-modal associations may only emerge within a sensitive developmental period in childhood.

To find out whether such a sensitive period exists, Sourav and colleagues recruited participants with different visual abilities and histories from the LV Prasad Eye Institute in Hyderabad, India, and the local community in Hamburg, Germany. The sample included 15 participants who were congenitally and permanently blind, 12 participants who became blind after the age of 12 years, 30 participants born with dense cataracts in both eyes who later had their vision surgically restored, 24 participants whose cataracts emerged before age 12 and were later removed, and 70 participants with normal vision.

The researchers presented all participants with four object pairs to be explored by touch. The objects in each pair were shapes that differed in their edges or surface patterns. Participants received the objects in an opaque cloth bag and were tasked with retrieving the object that best matched the sound of the pseudoword for that trial. Participants with visual capabilities additionally saw a fifth object pair and indicated which shape best matched the pseudoword.

Participants who experienced visual deprivation early in life performed no differently from chance level, regardless of whether they later had their vision restored, indicating that they did not have a systematic sound-shape association. The same was found in the group with a transient phase of visual impairments before the age of 12 (developmental cataract group). Only participants with late-onset blindness (after the age of 12) and participants with normal vision showed evidence of reliable sound-shape associations.

“Our data demonstrate two sides of sensitive periods: On the one hand, experience during an extended period of development is a prerequisite to acquire multisensory representations. On the other hand, representations acquired during this sensitive phase are not lost later in life despite dramatic and long-lasting changes in the environment,” explains Sourav.

“Our data provide valuable information on the experience-dependent nature of multisensory processing, offering a starting point for asking questions about how our sensory modalities shape word formation and, ultimately, the languages we speak,” Sourav concludes.

Coauthors on the research include Ramesh Kekunnaya and Idris Shareef  of the LV Prasad Eye Institute; Seema Banerjee of the LV Prasad Eye Institute and Hong Kong Polytechnic University; Davide Bottari of the University of Hamburg and IMT School for Advanced Studies Lucca; and Brigitte Röder of the University of Hamburg.

This study was funded by grants from the European Research Council and the Deutsche Forschungsgemeinschaft