paper

The Lipreading Gap: Do VSR Models Perceive Visual Speech Like Human Lipreaders?

paperactiveprovisionalthe-lipreading-gap-do-vsr-models-perceive-visual-speech-like-human-lipreaders--f846addf·1 events·first seen 9d ago

Aliases: The Lipreading Gap: Do VSR Models Perceive Visual Speech Like Human Lipreaders?

Co-occurring entities

MaFI

More like this (12)

Vision-Language Models Cross-Modal Masking for Robust Silent Speech Synthesis Using sEMG and Lipreading LabVLA: Grounding Vision-Language-Action Models in Scientific Laboratories Gaze Heads: How VLMs Look at What They Describe Gaze Heads: How VLMs Look at What They Describe visual language model Modeling Complex Behaviors: Multi-Personality Composition and Dynamic Switching in Vision-Language Models Vision-Language-Action models TempoVLA: Learning Speed-Controllable Vision-Language-Action Policies Audio Interaction Model Vision-Language-Action model SigLIP2

Recent events (1)

5arXiv · cs.CL·9d ago·source ↗

VSR models outperform humans on lipreading benchmarks but rely on language cues, not visual perception

A new arXiv paper compares three visual speech recognition (VSR) systems against human lipreaders on the MaFI dataset using word, character, phoneme, and viseme-level metrics. Despite higher overall accuracy, VSR models succeed and fail on different words than humans, and their errors are better explained by training word frequency than visual informativeness. A text-only n-gram baseline given minimal phoneme input rivals human performance, suggesting VSR systems primarily exploit language priors rather than genuine visual speech perception. The findings raise questions about whether benchmark-beating performance reflects the capability it purports to measure.

Evaluation and Benchmarking Multimodal Progress MaFI The Lipreading Gap: Do VSR Models Perceive Visual Speech Like Human Lipreaders?