Breaking News
April 23, 2018 - CDC seeking $400 million to replace lab for deadliest germs
April 23, 2018 - Sensirion to present single-use liquid flow sensor at COMPAMED 2017
April 23, 2018 - FDA approves contact lenses that shade the sun
April 22, 2018 - Concussion recovery and symptom severity found to vary between men and women
April 22, 2018 - C. Difficile Risk Higher With Stoma Reversal Versus Colectomy
April 22, 2018 - Repeated ranibizumab doesn’t impair macular perfusion
April 22, 2018 - New microscope reveals how cells behave in 3D and real time inside living organisms
April 22, 2018 - Study shows clinical benefit and monetary gains of weight-loss surgery
April 22, 2018 - GNA Biosolutions launches world’s first Laser PCR platform at Medica Trade Fair
April 22, 2018 - Researchers present simulation model to investigate hospital responsiveness to mass casualty incidents
April 22, 2018 - Does Pot Really Dull a Teen’s Brain?
April 22, 2018 - Controversial pregnancy test drug shows deformities in zebrafish embryos within hours of exposure
April 22, 2018 - New research partnership makes childbirth safer in Mozambique
April 22, 2018 - Brief bedside visual art intervention reduces pain, anxiety in cancer patients
April 22, 2018 - The memory part of the brain may also hold clues for anxiety and depression
April 22, 2018 - AYOXXA to develop multiplex immunoassay to support treatment of sepsis patients
April 22, 2018 - New Drug Combo Ups Survival in HER2/neu Uterine Serous Cancer
April 22, 2018 - Researchers chart a new way to look at concussion
April 22, 2018 - Every parent needs to know fundamental red flags for autism
April 22, 2018 - Anatotemp expands anatomic dental implant healing abutments with 4Side anti-rotational connection
April 22, 2018 - Gene Twist Can Make Your Blood Pressure Spike From Salt
April 22, 2018 - Study suggests failed osteoarthritis drug could help treat opioid addiction
April 22, 2018 - More Americans aware of growing problem of opioid addiction
April 22, 2018 - KHN’s ‘What the Health?’ Nothing in health care ever goes away
April 22, 2018 - BGS to promote high-quality sterilization services at Health GB in Manchester
April 22, 2018 - New integrated POC tool detects biomarkers of heart failure rapidly and precisely
April 22, 2018 - Direct electrical current can be delivered to nerves for blocking pain signals
April 22, 2018 - Newly Published Phase 2 Study Found Esketamine Demonstrated Significantly Rapid Improvements in Depressive Symptoms and Suicidality
April 22, 2018 - Healthy red blood cells owe their shape to muscle-like structures
April 22, 2018 - Researchers present case study of management of rheumatic mitral regurgitation in woman contemplating pregnancy
April 22, 2018 - New black Porvair Krystal UV Quartz microplates for Circular Dichroism measurements
April 22, 2018 - Advanced flow chemistry modules enhance control of nanoprecipitation
April 22, 2018 - Look! Down in the petri dish! It’s a superplatelet!
April 22, 2018 - Research reveals why people with tetraplegia more likely to suffer from sleep apnea
April 21, 2018 - New non-invasive nerve stimulation may offer relief for people with hand tremor
April 21, 2018 - Smartphone App May Up Medication Adherence in HTN
April 21, 2018 - Western diet depletes artery-protecting immune cells
April 21, 2018 - Excelitas Technologies launches new powerful LED light source for fluorescence microscopy
April 21, 2018 - Academia and high tech companies join forces to increase production capacity for microfluidic systems
April 21, 2018 - Developing cooking skills as young adult may have long-term health benefits
April 21, 2018 - Study compares survival outcomes of different drugs for type 2 diabetes
April 21, 2018 - More Than 40 Percent of Americans Breathe Dirty Air: Report
April 21, 2018 - Obstructive sleep apnea – Genetics Home Reference
April 21, 2018 - More evidence shows exposure to traffic and outdoor air pollution increases risk of asthma
April 21, 2018 - Novel gold nanoparticle technology could guide cancer treatment in real-time
April 21, 2018 - News coverage of Ebola impacted public’s perception on disease and survivors
April 21, 2018 - S.Africa’s DIY battle against HIV
April 21, 2018 - Children with autism have gastrointestinal and immune system deregulation, research finds
April 21, 2018 - Human brain processes sight and sound in the same way, shows study
April 21, 2018 - Evolutionary history of tumor helps predict severity of prostate cancer
April 21, 2018 - Pepper plant metabolizes antibiotic in personal care products
April 21, 2018 - Tradeshow Talks with Integra
April 21, 2018 - EPFL becomes part of Chan Zuckerberg’s project to develop Human Cell Atlas
April 21, 2018 - Pfizer Announces Positive Topline Results From Phase 3 ATTR-ACT Study Of Tafamidis In Patients With Transthyretin Cardiomyopathy
April 21, 2018 - Breaking through the HIV vaccine ‘logjam’
April 21, 2018 - IntelliCyt introduces new QSol buffer to enable robust, consistent sampling
April 21, 2018 - Scientists publish comprehensive lineage tree of whole adult animal in Science journal
April 21, 2018 - Innovative method based on FluidFM technology could revolutionize biological research
April 21, 2018 - Americans world’s biggest TV addicts, watching four hours a day
April 21, 2018 - Investigational drug may help increase protein levels in babies with spinal muscular atrophy
April 21, 2018 - Study shows distinctions between age groups in predicting and responding to stress at home
April 21, 2018 - Aziyo Biologics, BIOTRONIK enter into US co-distribution agreement
April 21, 2018 - Opiate Use Linked to Early Mortality in IBD Patients
April 21, 2018 - Online ads help pregnant smokers quit
April 21, 2018 - Opioid pain medications may not be safe for hemodialysis patients
April 21, 2018 - Rare variants in non-coding DNA inherited from parents heighten autism risk
April 21, 2018 - A needleless glucose monitor for diabetes patients
April 21, 2018 - BD introduces new informatics and automation solutions for clinical laboratories
April 21, 2018 - Turn Chores Into a Fitness Routine
April 21, 2018 - DNA methylation plays key role in stem cell differentiation
April 21, 2018 - Scientists find link between soil metals and cancer mortality
April 21, 2018 - Experts discuss implications of low calcium intake in global population
April 21, 2018 - GNA Biosolutions to display Pharos V8 Laser PCR instrument at Analytica trade fair
April 21, 2018 - People with vitamin D deficiency may be at greater risk of diabetes
April 21, 2018 - Study findings could open new possibilities for treating cancer with adenovirus
April 21, 2018 - People who use medical marijuana have higher rates of prescription drug use, study finds
April 21, 2018 - Study debunks ‘myth’ that strenuous exercise dampens immunity
April 21, 2018 - FDA approves marijuana based medication for epilepsy treatment
April 21, 2018 - Researchers find novel genes for longevity in mammals
April 21, 2018 - GNA Biosolutions and project partners launch new research project to develop TB diagnostic platform for POC applications
Study reveals vision’s role in vowel perception

Study reveals vision’s role in vowel perception

image_pdfDownload PDFimage_print
In a study based at Brown University, researchers found that the motion and configuration of a speaker’s lips are key components of the information people gather when distinguishing vowels in speech. In this case the difference is the position of the lips when making the ‘oo’ sound, as in goose, in English and in French. Credit: Masapollo et. al.

For all talkers, except perhaps the very best ventriloquists, the production of speech is accompanied by visible facial movements. Because speech is more than just sound, researchers set out to ascertain the exact visual information people seek when distinguishing vowel sounds.

“An important and highly debated issue in our field concerns what is it that we are attending to in speech—what’s the object of perception?” said lead author Matthew Masapollo, who conducted the research as a postdoctoral scholar at Brown University and is a now at Boston University. “Another question that’s debated is whether speech processing is special and distinct from other kinds of auditory processing since it is not purely an acoustic signal.”

Resolving these questions would improve the scientific understanding of how we perceive speech, Masapollo said. That, in turn, could apply to the design of more intelligible online avatars and physical robots, and could even improve computer recognition of human speech and enhance communication devices for the hearing impaired.

While scads of studies have investigated which audible features of speech are important, Masapollo said, far fewer have looked at which visual components are essential, despite evidence from phenomena as intuitive as lip reading that the sights of speech matter, too.

Through a series of experiments at Brown and McGill University in Montreal reported in the Journal of Experimental Psychology: Human Perception and Performance, Masapollo and colleagues found that when people perceive speech, they closely watch the form and motion of the lips. If either of those cues is missing, their ability to make subtle distinctions between vowel sounds suffers measurably.

“The findings demonstrate that adults are sensitive to the observable shape and movement patterns that occur when a person talks,” said Masapollo, who did the work as a researcher in the lab of senior author James Morgan, a Brown professor of cognitive, linguistic and psychological sciences.

Exploiting differences in speech perception

Earlier this year, Masapollo set the table for the new study when he and co-authors Linda Polka and Lucie Ménard showed in the journal Cognition that people exhibit the same “directional asymmetry” in visually perceiving vowels that they do when hearing vowels: They are better at distinguishing between two versions of the “oo” sound, as in the word “loose,” if the less extremely articulated version occurs first and then the more extreme version second. If the order is switched, they are much less likely to discriminate them—by sight or sound. While these directional effects may seem like a quirky instinct, they reflect a universal bias favoring vowels produced with extreme articulatory maneuvers. Current research is focused on uncovering what salient features or properties of extreme vowels give rise to these perceptual asymmetries.

Researchers employed many visual representations of lip motion to see which essential features really mattered. Masapollo et. al. Credit: Brown University

It turns out that this asymmetry plays out between French and English, being manifest in the bilingual speech of many Canadians. When speaking French, their articulation of “oo” is produced with more visible lip protrusion and tongue positioning than when making the same vowel sound in English.

For the new study, Masapollo realized that this asymmetry in vowel production and perception provided a great opportunity to determine which visual features matter in distinguishing subtle speech differences. He devised and led five experiments to ferret out exactly what visual information was pertinent to this asymmetry.

In the first, with help from Brown graduate student and co-author Lauren Franklin, he employed eye-tracking technology to measure where Brown student volunteers looked when watching videos of a bilingual Canadian woman make “oo” sounds in both French and English. Definitively, people watched the mouth, far more, for instance than the eyes.

But what about the mouth mattered? To determine if motion, rather than simply a particular position, was important, the next experiment presented students with a still frame rather than video. In experiment two, volunteers at McGill tried to distinguish “oo” speech using just still images of the same speaker. Without the cue of motion, the results showed, the asymmetry of French-English or English-French ordering no longer occurred, suggesting that motion is a key component in this instinct of vowel perception.

In the next three experiments, the team continued to investigate which visual aspects of speech perception mattered among groups of Brown or McGill student volunteers. In experiment three, the subjects saw not a face, but an array of four dots in a diamond pattern that moved just like the speaker’s lips did. When the speaker pursed her lips to make the “oo,” the dots moved closer together, for example. Masapollo’s hypothesis was that position and motion might matter together, even if the face isn’t actually represented. In this experiment, people returned to showing the asymmetry suggesting that he was on the right track.

Experiment four was exactly the same but the dot pattern was rotated 45 degrees clockwise, showing more of a square than a diamond. Here the asymmetry didn’t occur, suggesting that the orientation of the dots to represent a speech-making mouth matter. In experiment five, the motion was represented by a sideways figure eight that would move in a manner analogous to the speaker’s lips. There, too, without even an essential form of a mouth, people didn’t show their instinctual asymmetry of vowel perception. Mere motion, without the form and position of a mouth, was not enough.

“Overall, the picture that emerges is that perceptual asymmetries appear to be elicited by optical stimuli that depict both lip motion and configural information,” the authors wrote.

To Masapollo, the results demonstrate that vision makes specific contributions to perceiving speech.

“The findings of the present research suggest that the information we are attending to in speech is multimodal, and perhaps gestural, in nature,” Masapollo said. “Our perceptual system appears to treat auditory and visual speech information similarly.”


Explore further:
When your eyes override your ears: New insights into the McGurk effect

More information:
Matthew Masapollo et al, Asymmetries in unimodal visual vowel perception: The roles of oral-facial kinematics, orientation, and configuration., Journal of Experimental Psychology: Human Perception and Performance (2018). DOI: 10.1037/xhp0000518

Journal reference:
Cognition

Provided by:
Brown University

Tagged with:

About author

Related Articles