Breaking News
July 21, 2018 - Whole-brain LIPUS therapy improves cognitive dysfunction in mice simulating dementia, Alzheimer’s
July 21, 2018 - Digital media use raising risk of ADHD symptoms among the young
July 21, 2018 - Phase 3 study of tanezumab in patients with osteoarthritis pain meets all three co-primary endpoints
July 21, 2018 - Restoring mitochondrial function to reverse aging-related skin wrinkles, hair loss in mice
July 21, 2018 - SP PennTech introduces RW-500 rotary vial washer for biotech, pharmaceutical applications
July 21, 2018 - Researchers to study molecular mechanisms behind susceptibility of males to autism
July 21, 2018 - Fourth Published Clinical Trial Confirms Long-Term Safety of Niagen Supplementation at High Doses and Shows Potential for Improvement in Liver Health
July 21, 2018 - Parents say intense gun violence in PG-13 movies appropriate for teens 15 and older
July 21, 2018 - FAU researchers find possible cause of Parkinson’s disease in the patients’ immune system
July 21, 2018 - Protective qualities of ‘good cholesterol’ reduce after menopause
July 21, 2018 - Researchers develop new way to uncover hidden breast cancer tumors
July 21, 2018 - FDA approves first drug for treatment of adult AML patients with specific genetic mutation
July 21, 2018 - Top AI companies join hands to discover novel drugs for DMD
July 21, 2018 - Ferring announces FDA approval of ZOMACTON for injection in four new pediatric indications
July 20, 2018 - Researchers design proteins that can self-assemble into complex structures
July 20, 2018 - AVITA Medical expands management team to support launch of RECELL device to treat burns
July 20, 2018 - FDA Approves Tibsovo (ivosidenib) for Relapsed or Refractory Acute Myeloid Leukemia with an IDH1 Mutation
July 20, 2018 - Developmental screening and surveillance rates remain low, new study suggests
July 20, 2018 - TGen opens tissue donation portal to advance DIPG research
July 20, 2018 - Health impact of highly processed summertime staples
July 20, 2018 - Exergaming can improve health in overweight and obese children, study shows
July 20, 2018 - Postmenopausal factors may impact heart-protective qualities of ‘good cholesterol’
July 20, 2018 - MRI and blood test combination results in improved prostate cancer diagnosis
July 20, 2018 - Update Health Professional and Consumer on Recent Recalled Products
July 20, 2018 - Researchers trace Parkinson’s damage in the heart
July 20, 2018 - Wearable device designed to measure cortisol in sweat
July 20, 2018 - Scientists demonstrate a new regulation mechanism for skeletal muscles
July 20, 2018 - Exposure to mobile phone radiation may negatively impact memory performance in adolescents
July 20, 2018 - SUSU scientists find alternative method to treat post-traumatic stress disorder syndrome
July 20, 2018 - Gestational diabetes may increase offspring’s heart disease risk
July 20, 2018 - New vaccine could protect unborn babies from Zika virus
July 20, 2018 - Researchers find high mercury and methylmercury concentrations in traditional Tibetan medicine
July 20, 2018 - Brief Safety Plan Intervention in ER Can Cut Suicidal Behavior
July 20, 2018 - Toward a better understanding of Parkinson’s disease
July 20, 2018 - Med school communications office wins four national awards | News Center
July 20, 2018 - Professional baseball players with faster hand-eye coordination may have better batting performance
July 20, 2018 - Study looks into mechanisms that control sleep and wakefulness
July 20, 2018 - Scientists identify melanoma biomarkers that could help tailor immunotherapy treatments
July 20, 2018 - Research reveals long-term efficacy of drug used to treat common cause of kidney failure
July 20, 2018 - Timing of dinner associated with breast and prostate cancer risks
July 20, 2018 - Health Tip: Performing the Heimlich Maneuver
July 20, 2018 - Nearly all adolescents have eating, activity or weight-related issues
July 20, 2018 - Sage launches new web-based tool that helps explore curated genomic analyses of Alzheimer’s
July 20, 2018 - High-performance porous polymeric material for chromatography applications
July 20, 2018 - New molecule shows great promise for future treatment of many cancers
July 20, 2018 - New research project investigates alternative treatments for eye infections
July 20, 2018 - Immune T cells are built to react as fast as possible, shows study
July 20, 2018 - ZHX2 protein could offer a new treatment strategy for kidney cancer
July 20, 2018 - EKF’s Quo-Lab POC HbA1c analyzer meets international quality targets for diabetes testing
July 20, 2018 - Health burdens of very high risk drinking are potentially large, study reveals
July 20, 2018 - Using miniature drug-filled nanocarriers to target headaches and tumors
July 20, 2018 - Researchers uncover cause for progression of prostate cancer to incurable stage
July 20, 2018 - Studies highlight issues regarding black lung, opioid overdose, police violence and more
July 20, 2018 - AbbVie submits supplemental NDA to FDA for venetoclax to treat acute myeloid leukemia
July 20, 2018 - Researchers are one step closer to developing eye drops to treat age-related macular degeneration (AMD)
July 20, 2018 - Patients maintain muscle mass five years after surgically induced weight loss
July 20, 2018 - AMSBIO introduces new, powerful CRISPR/Cas9 gene editing kits
July 20, 2018 - PureTech Health collaborates with Roche to advance oral administration of antisense oligonucleotides
July 20, 2018 - Analysis reveals disparities in cancer death rates among minority groups
July 20, 2018 - Dr Maddy Parsons receives Royal Microscopical Society Life Science Medal
July 20, 2018 - Study finds link between DNA methylation and non-alcoholic fatty liver disease
July 20, 2018 - Military personnel with head trauma and football players with suspected CTE show similar brain changes
July 20, 2018 - Vidac Pharma Announces Initiation of Phase 2b Clinical Trial of VDA-1102 Ointment in Patients with Actinic Keratosis
July 20, 2018 - KKR is buying Envision Healthcare in a nearly $10B deal
July 20, 2018 - Older people with broken bones face higher risk of death for up to 10 years
July 20, 2018 - A simple pill for meth addicts on the cards
July 20, 2018 - UA researchers to repurpose ketamine to reduce side effects in Parkinson’s patients
July 20, 2018 - Child psychiatrist available on call to help assess separated immigrant children
July 20, 2018 - High bitter-taste sensitivity linked to increased risk of cancer
July 20, 2018 - Falling temperatures may lead to rise in numbers of deaths from stroke
July 20, 2018 - Supplemental oxygen prevents rise in morning blood pressure in OSA patients
July 20, 2018 - High fruit and vegetable intake linked to reduced risk of breast cancer
July 20, 2018 - Careful patient selection may help achieve good outcomes for vaginal mesh surgery
July 20, 2018 - Researchers raise viability of cloned mice using somatic cell nuclear transfer method
July 20, 2018 - 3HP for Latent TB Infection Treatment | 2018 | Newsroom | NCHHSTP
July 20, 2018 - An orange a day keeps macular degeneration away: 15-year study
July 20, 2018 - Researchers elucidate how the brain drives trial-by-trial adaptation to compensate for errors
July 20, 2018 - Understanding triple-negative breast cancer to develop better treatments
July 20, 2018 - Study compares outpatient antibiotic prescribing with traditional medical, retail clinic settings
July 20, 2018 - Immediate Monitoring With ECG Patch Ups A-Fib Diagnosis Rate
Study reveals vision’s role in vowel perception

Study reveals vision’s role in vowel perception

image_pdfDownload PDFimage_print
In a study based at Brown University, researchers found that the motion and configuration of a speaker’s lips are key components of the information people gather when distinguishing vowels in speech. In this case the difference is the position of the lips when making the ‘oo’ sound, as in goose, in English and in French. Credit: Masapollo et. al.

For all talkers, except perhaps the very best ventriloquists, the production of speech is accompanied by visible facial movements. Because speech is more than just sound, researchers set out to ascertain the exact visual information people seek when distinguishing vowel sounds.

“An important and highly debated issue in our field concerns what is it that we are attending to in speech—what’s the object of perception?” said lead author Matthew Masapollo, who conducted the research as a postdoctoral scholar at Brown University and is a now at Boston University. “Another question that’s debated is whether speech processing is special and distinct from other kinds of auditory processing since it is not purely an acoustic signal.”

Resolving these questions would improve the scientific understanding of how we perceive speech, Masapollo said. That, in turn, could apply to the design of more intelligible online avatars and physical robots, and could even improve computer recognition of human speech and enhance communication devices for the hearing impaired.

While scads of studies have investigated which audible features of speech are important, Masapollo said, far fewer have looked at which visual components are essential, despite evidence from phenomena as intuitive as lip reading that the sights of speech matter, too.

Through a series of experiments at Brown and McGill University in Montreal reported in the Journal of Experimental Psychology: Human Perception and Performance, Masapollo and colleagues found that when people perceive speech, they closely watch the form and motion of the lips. If either of those cues is missing, their ability to make subtle distinctions between vowel sounds suffers measurably.

“The findings demonstrate that adults are sensitive to the observable shape and movement patterns that occur when a person talks,” said Masapollo, who did the work as a researcher in the lab of senior author James Morgan, a Brown professor of cognitive, linguistic and psychological sciences.

Exploiting differences in speech perception

Earlier this year, Masapollo set the table for the new study when he and co-authors Linda Polka and Lucie Ménard showed in the journal Cognition that people exhibit the same “directional asymmetry” in visually perceiving vowels that they do when hearing vowels: They are better at distinguishing between two versions of the “oo” sound, as in the word “loose,” if the less extremely articulated version occurs first and then the more extreme version second. If the order is switched, they are much less likely to discriminate them—by sight or sound. While these directional effects may seem like a quirky instinct, they reflect a universal bias favoring vowels produced with extreme articulatory maneuvers. Current research is focused on uncovering what salient features or properties of extreme vowels give rise to these perceptual asymmetries.

Researchers employed many visual representations of lip motion to see which essential features really mattered. Masapollo et. al. Credit: Brown University

It turns out that this asymmetry plays out between French and English, being manifest in the bilingual speech of many Canadians. When speaking French, their articulation of “oo” is produced with more visible lip protrusion and tongue positioning than when making the same vowel sound in English.

For the new study, Masapollo realized that this asymmetry in vowel production and perception provided a great opportunity to determine which visual features matter in distinguishing subtle speech differences. He devised and led five experiments to ferret out exactly what visual information was pertinent to this asymmetry.

In the first, with help from Brown graduate student and co-author Lauren Franklin, he employed eye-tracking technology to measure where Brown student volunteers looked when watching videos of a bilingual Canadian woman make “oo” sounds in both French and English. Definitively, people watched the mouth, far more, for instance than the eyes.

But what about the mouth mattered? To determine if motion, rather than simply a particular position, was important, the next experiment presented students with a still frame rather than video. In experiment two, volunteers at McGill tried to distinguish “oo” speech using just still images of the same speaker. Without the cue of motion, the results showed, the asymmetry of French-English or English-French ordering no longer occurred, suggesting that motion is a key component in this instinct of vowel perception.

In the next three experiments, the team continued to investigate which visual aspects of speech perception mattered among groups of Brown or McGill student volunteers. In experiment three, the subjects saw not a face, but an array of four dots in a diamond pattern that moved just like the speaker’s lips did. When the speaker pursed her lips to make the “oo,” the dots moved closer together, for example. Masapollo’s hypothesis was that position and motion might matter together, even if the face isn’t actually represented. In this experiment, people returned to showing the asymmetry suggesting that he was on the right track.

Experiment four was exactly the same but the dot pattern was rotated 45 degrees clockwise, showing more of a square than a diamond. Here the asymmetry didn’t occur, suggesting that the orientation of the dots to represent a speech-making mouth matter. In experiment five, the motion was represented by a sideways figure eight that would move in a manner analogous to the speaker’s lips. There, too, without even an essential form of a mouth, people didn’t show their instinctual asymmetry of vowel perception. Mere motion, without the form and position of a mouth, was not enough.

“Overall, the picture that emerges is that perceptual asymmetries appear to be elicited by optical stimuli that depict both lip motion and configural information,” the authors wrote.

To Masapollo, the results demonstrate that vision makes specific contributions to perceiving speech.

“The findings of the present research suggest that the information we are attending to in speech is multimodal, and perhaps gestural, in nature,” Masapollo said. “Our perceptual system appears to treat auditory and visual speech information similarly.”


Explore further:
When your eyes override your ears: New insights into the McGurk effect

More information:
Matthew Masapollo et al, Asymmetries in unimodal visual vowel perception: The roles of oral-facial kinematics, orientation, and configuration., Journal of Experimental Psychology: Human Perception and Performance (2018). DOI: 10.1037/xhp0000518

Journal reference:
Cognition

Provided by:
Brown University

Tagged with:

About author

Related Articles