4arXiv cs.CL (Computation and Language)·
MEUSLI: open-source multilingual projector linking Whisper to LLMs for 28-language ASR
MEUSLI is a new open-science multilingual projector family that connects a Whisper speech encoder to open-source multilingual LLMs, enabling end-to-end ASR across 28 European languages. The system extends prior monolingual pipelines using continual learning techniques to support additional unseen languages and generalizes beyond ASR to speech translation and topic identification with minimal task-specific supervision. The work positions itself as a foundation for scalable, inclusive open-source SpeechLLM development.
Related guides (3)
Related events (8)
AlignAtt4LLM adapts simultaneous speech translation policy to decoder-only LLMs for IWSLT 2026
Researchers present AlignAtt4LLM, a simultaneous speech translation system for IWSLT 2026 covering English to German, Italian, and Chinese. The system cascades Qwen3-ASR for incremental transcription with Gemma-4 E4B-it for translation, applying a novel AlignAtt policy adapted for decoder-only LLMs that lack encoder-decoder cross-attention. Key contributions include explicit source span prompting, offline alignment head selection, and query/key capture to recover a usable attention-based read/write policy. The system outperforms IWSLT 2026 baselines for European language pairs in both low- and high-latency regimes.
Introducing Whisper
OpenAI introduced Whisper, an open-source automatic speech recognition (ASR) system trained on 680,000 hours of multilingual and multitask supervised data collected from the web. The model demonstrates strong robustness to accents, background noise, and technical language, approaching human-level accuracy in English transcription. Whisper supports transcription in multiple languages as well as translation to English, and the weights and inference code were released publicly.
Synthetic LLM-generated conversations improve ASR training for low-resource languages
Researchers propose a pipeline that uses LLMs to generate scenario-level dialogues and TTS to synthesize multi-speaker audio, creating simulated conversational training data for ASR systems. Evaluated on the Hungarian BEA-Dialogue benchmark, a model trained on 67 hours of real plus 636 hours of synthetic data outperforms a zero-shot model trained on 2,700 hours of real Hungarian speech. The study tests five LLM families under multiple budget and mixing configurations using a FastConformer-Large backbone, finding that generator choice and data composition significantly affect gains.
First systematic study of federated learning for SpeechLLM-based end-to-end ASR
A new arXiv preprint presents the first systematic study of federated training applied to speech language model (SpeechLLM) architectures for end-to-end automatic speech recognition. The authors design a communication-efficient federated optimization strategy addressing gradient overhead and distributed computational constraints, evaluated on English and Italian monolingual ASR tasks. Results show competitive word error rates with reduced communication costs compared to centralized baselines, with ablation studies covering speech encoder architecture choices.
AuRA: Distilling audio understanding into LLMs via LoRA adaptation
AuRA is a new method for integrating speech understanding into LLMs by distilling audio encoding capability directly into LoRA-adapted model weights, bypassing cascaded ASR-LLM pipelines. A lightweight audio embedding layer feeds speech to both an ASR encoder (teacher) and a LoRA-adapted LLM (student), with layer-wise distillation aligning hidden states. The approach claims to outperform cascaded systems, bridge-based adaptation baselines, and large-scale multimodal models on multiple speech-language benchmarks while enabling parallel end-to-end inference without large-scale multimodal training.
UniAudio-Token: Semantic Speech Tokenizer with General Audio Perception for Audio-LLMs
UniAudio-Token is a framework from Tencent that extends semantic speech tokenizers—commonly used as interfaces for Audio-LLMs—to support general audio perception without sacrificing speech quality. It introduces two mechanisms: Semantic-Acoustic Primitives (SAP) for structured supervision decomposing audio into linguistic, vocal, and auditory-scene components, and Semantic-Acoustic Equilibrium (SAE), a content-aware gating mechanism that restores fine-grained acoustic details from shallow layers. Evaluations show it outperforms all single-codebook baseline tokenizers on both understanding and generation tasks when integrated with downstream LLMs. Code, training/inference scripts, and model checkpoints are publicly released.
Nemotron-Labs Audex-30B: Unified audio-text MoE LLM with no text regression
Nvidia's Nemotron Labs introduces Audex-30B-A3B, a 30B-parameter mixture-of-experts audio-text LLM built on the Nemotron-Cascade-2 text backbone. The model handles audio understanding, ASR, translation, TTS, and speech-to-speech generation within a single Transformer decoder by projecting audio into the text embedding space. Training used 157.4B audio tokens and 320.5B text tokens with multi-stage supervised learning, RL, and on-policy distillation. Model checkpoints are publicly released, and the authors report state-of-the-art audio performance with minimal regression on text reasoning and agentic capabilities.
BayLing-Duplex: Native full-duplex speech dialogue using a single autoregressive LLM
Researchers introduce BayLing-Duplex, a speech language model that achieves native full-duplex interaction — simultaneous listening and speaking — using a single autoregressive LLM with no auxiliary VAD or turn-taking module. Built by fine-tuning GLM-4-Voice on 400K samples plus a lightweight DPO stage, it reaches 92% turn-taking success and 100% interruption success on InstructS2S-Eval, and improves speech-response quality substantially over Moshi. The approach adds only special tokens to the standard vocabulary, making it portable across LLM architectures without architectural changes.