Breaking News
December 17, 2018 - In-school nutrition programs among students limit increases in BMI, finds study
December 17, 2018 - Risk for Hospitalization for Heart Failure Greater With Diabetes
December 17, 2018 - Food assistance may help older adults adhere to diabetes meds
December 17, 2018 - Supporting a family’s goals during a difficult pregnancy
December 17, 2018 - Neurons with Good Housekeeping Are Protected from Alzheimer’s
December 17, 2018 - CTF along with NTAP and Sage announce first-ever open data portal for neurofibromatosis
December 17, 2018 - Intimacy: The Elusive Fountain of Youth?
December 17, 2018 - Will saliva translate to a real diagnostic tool?
December 17, 2018 - DFG establishes nine new Research Units and one new Clinical Research Unit
December 17, 2018 - Assisted living’s breakneck growth leaves patient safety behind
December 17, 2018 - America’s teens report dramatic increase in their use of vaping devices in just one year
December 17, 2018 - Enlarged heart linked to a higher risk of dementia
December 17, 2018 - Prostate cancer detection using MRI now first-line investigation tool
December 17, 2018 - Loughborough academics part of new project investigating effectiveness of personalized breast cancer screening
December 17, 2018 - Adolescents who use cognitive reappraisal had better metabolic measures, shows study
December 17, 2018 - Probiotics may offer therapeutic benefits for biopolar patients
December 17, 2018 - Stealth BioTherapeutics Granted Fast Track Designation for Elamipretide for the Treatment of Dry Age-Related Macular Degeneration with Geographic Atrophy
December 17, 2018 - Studies reveal role of red meat in gut bacteria, heart disease development
December 17, 2018 - Eisai enters into agreement with Eurofarma for its anti-obesity agent lorcaserin
December 17, 2018 - Researchers use brain connectome to reassess neuroimaging findings of Alzheimer’s disease
December 17, 2018 - “Miracle” baby survives Ebola in Congo and rapid a new Ebola detection device
December 17, 2018 - Mechanisms behind neonatal diabetes uncovered
December 17, 2018 - AHF urges the WHO to expedite approval process for vaccine effective against Ebola
December 17, 2018 - Study finds misuse of benzodiazepines to be highest among young adults
December 17, 2018 - TGen receives PayPal grant to underwrite costs of genetic tests for children with rare disorders
December 17, 2018 - New research highlights why HIV-infected patients suffer higher rates of cancer
December 17, 2018 - Antibiotic-resistant bacteria could soon be targeted with Alzheimer’s drug
December 17, 2018 - Rutgers scientists take an important step in making diseased hearts heal themselves
December 17, 2018 - Tailored Feedback at CRC Screen Improves Lifestyle Behaviors
December 17, 2018 - Loss of two genes drives a deadly form of colorectal cancer, reveals a potential treatment
December 17, 2018 - How the Mediterranean Diet Can Help Women’s Hearts
December 17, 2018 - Sustained connections associated with symptoms of autism
December 17, 2018 - Concussion rates among young football players were higher than previously reported
December 17, 2018 - Cresco Labs granted approval to operate marijuana dispensary in Ohio
December 17, 2018 - Study provides insight into health risks facing new mothers
December 17, 2018 - AMSBIO expands Wnt signaling pathway product range to aid research
December 16, 2018 - Surgical treatment unnecessary for many prostate cancer patients
December 16, 2018 - Excess weight responsible for cancers globally finds report
December 16, 2018 - Regular sex associated with greater enjoyment of life in seniors
December 16, 2018 - Social stigma contributes to poor mental health in the autistic community
December 16, 2018 - Multidisciplinary team successfully performs complex surgery on patient suffering from enlarged skull
December 16, 2018 - Experts analyze data that can guide antidepressant discontinuation
December 16, 2018 - Menlo Therapeutics’ Successful Phase 2 Clinical Trial of Serlopitant Demonstrates Reduction of Pruritus Associated with Psoriasis
December 16, 2018 - Siblings of children with autism or ADHD are at elevated risk for both disorders
December 16, 2018 - New project aims to understand why and how metabolic disorders develop in patients
December 16, 2018 - Diets containing GM maize have no harmful effects on health or metabolism of rats
December 16, 2018 - Are doctors and teachers confusing immaturity and attention deficit?
December 16, 2018 - Hearing loss linked with increased risk for premature death
December 16, 2018 - Chromatrap buffer reagents for lysing cells offer many benefits
December 16, 2018 - Young Breast Cancer Patients Face Higher Risk for Osteoporosis
December 16, 2018 - 3-D printing offers helping hand to people with arthritis
December 16, 2018 - Community Health Choice helps manage complex and chronic care conditions
December 16, 2018 - Regular trips out could dramatically reduce depression in older age
December 16, 2018 - CWRU to use VivaLNK’s Vital Scout device for stress study in student athletes
December 16, 2018 - ‘Easy Way Out’? Stigma May Keep Many From Weight Loss Surgery
December 16, 2018 - Gout drug may protect against chronic kidney disease
December 16, 2018 - Talking about memories enhances the wellbeing of older and younger people
December 16, 2018 - Occupational exposure to pesticides increases risk for cardiovascular disease among Latinos
December 16, 2018 - A biomarker in the brain’s circulation system may be Alzheimer’s earliest warning
December 16, 2018 - Magnesium may play important role in optimizing vitamin D levels, study shows
December 16, 2018 - The effect of probiotics on intestinal flora of premature babies
December 16, 2018 - Parents spend more time talking with kids about mechanics of using mobile devices
December 16, 2018 - Biohaven Announces Positive Results from Ongoing Rimegepant Long-Term Safety Study
December 16, 2018 - Arterial stiffness may predict dementia risk
December 16, 2018 - Study explores link between work stress and increased cancer risk
December 16, 2018 - Sex work criminalization linked to incidences of violence finds study
December 16, 2018 - Johns Hopkins researchers discover swarming behavior in fish-dwelling parasite
December 16, 2018 - Schistosomiasis prevention and treatment could help control HIV
December 16, 2018 - Early postpartum opioids linked with persistent usage
December 16, 2018 - Johns Hopkins researchers identify molecular causes of necrotizing enterocolitis in preemies
December 16, 2018 - Advanced illumination expands capabilities of light-sheet microscopy
December 16, 2018 - Alzheimer’s could possibly be spread via contaminated neurosurgery
December 16, 2018 - Unraveling the complexity of cancer biology can prompt new avenues for drug development
December 16, 2018 - Inflammatory Bowel Disease, Prostate Cancer Linked
December 16, 2018 - Cannabis youth prevention strategy should target mental wellbeing
December 15, 2018 - Recent developments and challenges in hMAT inhibitors
December 15, 2018 - Sewage bacteria found lurking in Hudson River sediments
December 15, 2018 - CDC selects UMass Amherst biostatistician model that helps predict influenza outbreaks
December 15, 2018 - Researchers reveal brain mechanism that drives itch-evoked scratching behavior
December 15, 2018 - New computer model helps predict course of the disease in prostate cancer patients
Study reveals vision’s role in vowel perception

Study reveals vision’s role in vowel perception

image_pdfDownload PDFimage_print
In a study based at Brown University, researchers found that the motion and configuration of a speaker’s lips are key components of the information people gather when distinguishing vowels in speech. In this case the difference is the position of the lips when making the ‘oo’ sound, as in goose, in English and in French. Credit: Masapollo et. al.

For all talkers, except perhaps the very best ventriloquists, the production of speech is accompanied by visible facial movements. Because speech is more than just sound, researchers set out to ascertain the exact visual information people seek when distinguishing vowel sounds.

“An important and highly debated issue in our field concerns what is it that we are attending to in speech—what’s the object of perception?” said lead author Matthew Masapollo, who conducted the research as a postdoctoral scholar at Brown University and is a now at Boston University. “Another question that’s debated is whether speech processing is special and distinct from other kinds of auditory processing since it is not purely an acoustic signal.”

Resolving these questions would improve the scientific understanding of how we perceive speech, Masapollo said. That, in turn, could apply to the design of more intelligible online avatars and physical robots, and could even improve computer recognition of human speech and enhance communication devices for the hearing impaired.

While scads of studies have investigated which audible features of speech are important, Masapollo said, far fewer have looked at which visual components are essential, despite evidence from phenomena as intuitive as lip reading that the sights of speech matter, too.

Through a series of experiments at Brown and McGill University in Montreal reported in the Journal of Experimental Psychology: Human Perception and Performance, Masapollo and colleagues found that when people perceive speech, they closely watch the form and motion of the lips. If either of those cues is missing, their ability to make subtle distinctions between vowel sounds suffers measurably.

“The findings demonstrate that adults are sensitive to the observable shape and movement patterns that occur when a person talks,” said Masapollo, who did the work as a researcher in the lab of senior author James Morgan, a Brown professor of cognitive, linguistic and psychological sciences.

Exploiting differences in speech perception

Earlier this year, Masapollo set the table for the new study when he and co-authors Linda Polka and Lucie Ménard showed in the journal Cognition that people exhibit the same “directional asymmetry” in visually perceiving vowels that they do when hearing vowels: They are better at distinguishing between two versions of the “oo” sound, as in the word “loose,” if the less extremely articulated version occurs first and then the more extreme version second. If the order is switched, they are much less likely to discriminate them—by sight or sound. While these directional effects may seem like a quirky instinct, they reflect a universal bias favoring vowels produced with extreme articulatory maneuvers. Current research is focused on uncovering what salient features or properties of extreme vowels give rise to these perceptual asymmetries.

Researchers employed many visual representations of lip motion to see which essential features really mattered. Masapollo et. al. Credit: Brown University

It turns out that this asymmetry plays out between French and English, being manifest in the bilingual speech of many Canadians. When speaking French, their articulation of “oo” is produced with more visible lip protrusion and tongue positioning than when making the same vowel sound in English.

For the new study, Masapollo realized that this asymmetry in vowel production and perception provided a great opportunity to determine which visual features matter in distinguishing subtle speech differences. He devised and led five experiments to ferret out exactly what visual information was pertinent to this asymmetry.

In the first, with help from Brown graduate student and co-author Lauren Franklin, he employed eye-tracking technology to measure where Brown student volunteers looked when watching videos of a bilingual Canadian woman make “oo” sounds in both French and English. Definitively, people watched the mouth, far more, for instance than the eyes.

But what about the mouth mattered? To determine if motion, rather than simply a particular position, was important, the next experiment presented students with a still frame rather than video. In experiment two, volunteers at McGill tried to distinguish “oo” speech using just still images of the same speaker. Without the cue of motion, the results showed, the asymmetry of French-English or English-French ordering no longer occurred, suggesting that motion is a key component in this instinct of vowel perception.

In the next three experiments, the team continued to investigate which visual aspects of speech perception mattered among groups of Brown or McGill student volunteers. In experiment three, the subjects saw not a face, but an array of four dots in a diamond pattern that moved just like the speaker’s lips did. When the speaker pursed her lips to make the “oo,” the dots moved closer together, for example. Masapollo’s hypothesis was that position and motion might matter together, even if the face isn’t actually represented. In this experiment, people returned to showing the asymmetry suggesting that he was on the right track.

Experiment four was exactly the same but the dot pattern was rotated 45 degrees clockwise, showing more of a square than a diamond. Here the asymmetry didn’t occur, suggesting that the orientation of the dots to represent a speech-making mouth matter. In experiment five, the motion was represented by a sideways figure eight that would move in a manner analogous to the speaker’s lips. There, too, without even an essential form of a mouth, people didn’t show their instinctual asymmetry of vowel perception. Mere motion, without the form and position of a mouth, was not enough.

“Overall, the picture that emerges is that perceptual asymmetries appear to be elicited by optical stimuli that depict both lip motion and configural information,” the authors wrote.

To Masapollo, the results demonstrate that vision makes specific contributions to perceiving speech.

“The findings of the present research suggest that the information we are attending to in speech is multimodal, and perhaps gestural, in nature,” Masapollo said. “Our perceptual system appears to treat auditory and visual speech information similarly.”


Explore further:
When your eyes override your ears: New insights into the McGurk effect

More information:
Matthew Masapollo et al, Asymmetries in unimodal visual vowel perception: The roles of oral-facial kinematics, orientation, and configuration., Journal of Experimental Psychology: Human Perception and Performance (2018). DOI: 10.1037/xhp0000518

Journal reference:
Cognition

Provided by:
Brown University

Tagged with:

About author

Related Articles