3arXiv cs.CL (Computation and Language)·
Benchmarking human vs. ASR systems on diverse Dutch speech (children, elderly, Flemish)
A preprint from arXiv compares state-of-the-art ASR systems against Dutch native listeners on 'diverse' speech including child speech, older adult speech, and Flemish regional accents. Google Telephony outperformed other ASR systems, and ASR systems matched or exceeded human listener performance in several conditions. Performance gaps remain for aging-related acoustic variability and regional accents, and the choice of test set materially affects benchmark conclusions.
Related guides (1)
Related events (8)
Case study compares human and ASR system performance on Dutch dysarthric speech recognition
A new arXiv preprint compares human listeners against three off-the-shelf ASR systems (Whisper-large-V3, Google Chirp 3, and Omnilingual) on recognizing continuous Dutch speech from a single speaker with severe dysarthria. Both humans and ASR systems exceeded 70% WER on average, confirming the extreme difficulty of dysarthric speech recognition. Fine-tuning on dysarthric speech substantially reduced WER, with personalized models outperforming human listeners, though WER remained above 23%. The study highlights the need for personalized ASR approaches for dysarthric speakers.
ServiceNow AI benchmarks frontier ASR systems on code-switched bilingual speech
ServiceNow AI published a benchmarking study evaluating frontier automatic speech recognition (ASR) systems on code-switched speech, where speakers alternate between two languages mid-conversation. The work targets a practical gap in voice agent deployments serving bilingual customer populations. Results assess how well current ASR models handle this linguistically complex scenario, with implications for enterprise voice AI reliability.
SPEARBench: Benchmark for naturalness evaluation in streaming speech-to-speech language models
Researchers introduce SPEARBench, a benchmark for evaluating conversational naturalness in streaming speech-to-speech language models, covering dimensions such as response latency, turn-taking, prosody, dialect consistency, emotional adaptation, and interpersonal stance. The benchmark constructs controlled dialogue prompts from the Seamless Interaction corpus and evaluates multiple contemporary models against human reference answers. Results reveal that current models achieve high signal-level quality and low ASR error rates but still diverge from human conversational behavior on latency, overlap, dialect preservation, and stance dynamics. The work addresses a gap in existing speech and text benchmarks that fail to capture conversational naturalness.
ParaPairAudioBench: Pairwise benchmark reveals large gaps in LALM paralinguistic judgment
Researchers introduce ParaPairAudioBench, a pairwise audio benchmark of 5,175 audio pairs spanning five paralinguistic dimensions (Style, Rate, Emphasis, Age, Gender) designed to evaluate Large Audio-Language Models as judges. Experiments show current LALMs lag human judgment by 32 percentage points on average and exhibit severe calibration failures, especially in ambiguous 'Tie' cases. The benchmark includes same-transcript and cross-transcript conditions to disentangle lexical from acoustic reliance, enabling more rigorous assessment of LALM reliability for speech evaluation.
VSR models outperform humans on lipreading benchmarks but rely on language cues, not visual perception
A new arXiv paper compares three visual speech recognition (VSR) systems against human lipreaders on the MaFI dataset using word, character, phoneme, and viseme-level metrics. Despite higher overall accuracy, VSR models succeed and fail on different words than humans, and their errors are better explained by training word frequency than visual informativeness. A text-only n-gram baseline given minimal phoneme input rivals human performance, suggesting VSR systems primarily exploit language priors rather than genuine visual speech perception. The findings raise questions about whether benchmark-beating performance reflects the capability it purports to measure.
FunASR: Industrial-Grade Speech Recognition Toolkit with 170x Realtime Performance
FunASR is an open-source speech recognition toolkit from ModelScope supporting 50+ languages, speaker diarization, emotion detection, and streaming inference at 170x realtime speed. It exposes an OpenAI-compatible API, positioning it as a drop-in alternative for production ASR workloads. The repository has accumulated 16,317 stars with modest daily momentum (+42 today).
Audit reveals protocol-level shortcuts in large audio-language model judges for speech evaluation
Researchers audit 'protocol-level shortcuts' in large audio-language models (LALMs) used as automatic judges for speech evaluation, testing across three deployment protocols: feature-blueprint judging, reference-conditioned judging, and pairwise A/B comparison. Across six judges and four attributes, several LALMs are found to rely on shortcuts rather than actual audio content — for example, incorrect specialist labels collapse emotion accuracy to 0.10 or below for five judges, and Qwen3-Omni-Thinking shows position bias in A/B comparisons. The findings indicate that high human-agreement scores can overstate judge validity, and that each model-protocol pair requires its own shortcut probe to be trustworthy.
Study compares silence/turn-taking gaps in human sitcoms vs. Google NotebookLM synthetic podcasts
A preprint investigates silence gaps and turn-taking patterns in two corpora: thirty US sitcoms and fifty-one synthetic podcasts generated by Google NotebookLM. The study compares gap durations across speaker gender (estimated via fundamental frequency in Praat) and production settings. The work provides empirical data on how AI-generated conversational audio differs from human discourse in prosodic timing.