3arXiv cs.CL (Computation and Language)·
Iterative pseudo-labeling approach improves Mandarin-English code-switching ASR
A new arXiv preprint introduces a three-phase iterative pseudo-labeling framework for code-switching automatic speech recognition (ASR), applied here to Mandarin-English mixing. The method generates pseudo-labels from unlabeled corpora, trains a bilingual model in two stages, and iteratively refines it, achieving Mix Error Rate reductions of 6.35% and 8.29% on the SEAME benchmark's devman and devsge subsets. This is the first application of iterative pseudo-labeling to code-switching ASR, addressing the chronic data scarcity problem in this domain.
Related guides (1)
Related events (8)
Audio-LLM-based data filtering for speech-to-speech translation via Rank-to-Distill
A new arXiv paper proposes using audio large language models to filter noisy training data for end-to-end speech-to-speech translation (S2ST). The authors introduce a two-stage Rank-to-Distill strategy: a lightweight ranker generates pseudo-labels from noisy speech pairs, which then supervise an audio-LLM to make keep/drop decisions directly from raw audio. Experiments on CVSS-C and SpeechMatrix benchmarks show up to +1.4 ASR-BLEU improvement over unfiltered baselines.
REDDIT framework corrects timestamp drift in autoregressive ASR without catastrophic forgetting
A new arXiv paper introduces REDDIT (Replay-based Distribution EDITing), a two-stage post-training framework that corrects timestamp drift in autoregressive ASR systems like Whisper across long non-speech spans. The method updates only 1.6% of model parameters and constructs correction supervision without human annotations, using VAD-trimmed speech with inserted non-speech gaps. On Whisper-tiny, long-gap mIoU improves from 38.7% to 95.0% and out-of-domain alignment error drops from 2752 ms to 223 ms, while preserving transcription quality that ordinary SFT decoder tuning catastrophically degrades.
RL-based alignment improves interactivity in full-duplex spoken dialogue models
Researchers propose a post-training alignment method using reinforcement learning to improve interactivity in full-duplex spoken dialogue models, which can listen and speak simultaneously. The method addresses four canonical axes of interactivity—pause handling, turn-taking, backchanneling, and user interruption—each with axis-specific reward functions, plus an LLM-based reward to prevent semantic degradation. The approach is applied to two open-source models, Moshi and PersonaPlex, showing consistent improvements in both offline and real-time multi-turn evaluation.
Continual learning approach for disfluency-aware ASR with explicit disfluency tokens
A new arXiv preprint addresses the challenge of transcribing disfluent speech (hesitations, repetitions, fillers) in ASR systems, which typically omit such markers causing information loss. The authors introduce explicit disfluency tokens into a pretrained ASR model and apply continual learning to adapt across datasets with varying disfluency distributions while mitigating catastrophic forgetting. The work identifies a trade-off between disfluency marker learning and general ASR performance, and finds a consistent cross-attention head mechanism shared across continual learning methods.
ServiceNow AI benchmarks frontier ASR systems on code-switched bilingual speech
ServiceNow AI published a benchmarking study evaluating frontier automatic speech recognition (ASR) systems on code-switched speech, where speakers alternate between two languages mid-conversation. The work targets a practical gap in voice agent deployments serving bilingual customer populations. Results assess how well current ASR models handle this linguistically complex scenario, with implications for enterprise voice AI reliability.
Synthetic LLM-generated conversations improve ASR training for low-resource languages
Researchers propose a pipeline that uses LLMs to generate scenario-level dialogues and TTS to synthesize multi-speaker audio, creating simulated conversational training data for ASR systems. Evaluated on the Hungarian BEA-Dialogue benchmark, a model trained on 67 hours of real plus 636 hours of synthetic data outperforms a zero-shot model trained on 2,700 hours of real Hungarian speech. The study tests five LLM families under multiple budget and mixing configurations using a FastConformer-Large backbone, finding that generator choice and data composition significantly affect gains.
HERMES: Hierarchical multi-granularity labeling substrate for pre-training data mixture design
HERMES is a data-derived labeling substrate that annotates each document once into a coarse-to-fine hierarchical code using a Learned Semantic Transform and 3-stage residual vector quantization, supporting up to ~130k cells at the finest granularity. The key contribution is reframing data mixture design from selecting among fixed label sets to navigating a reusable granularity hierarchy. In 1B-parameter, 25B-token pre-training experiments, the hierarchy reveals granularity-dependent interactions: a combined coverage-quality rule lifts a 16-task capability macro-average by +0.0253 at one prefix length but loses its edge at the next finer level as candidate pools shrink ~5x. The work argues the bottleneck in data mixing is the label system rather than the mixer itself.
Cross-modal masking framework improves silent speech synthesis from sEMG and lipreading
Researchers propose a masked multimodal speech synthesis framework that jointly trains on surface electromyography (sEMG) and video-based lipreading signals using modality masking to improve robustness to sensor failure or degradation. In multispeaker settings, the approach reduces word error rate by up to 14 absolute percentage points over the strongest unimodal baseline. Masking strategies outperform degradation-specific data augmentation for handling missing modalities, with phone-level analysis revealing complementary contributions across vowels and consonant groups.