4arXiv cs.CL (Computation and Language)·
First systematic study of federated learning for SpeechLLM-based end-to-end ASR
A new arXiv preprint presents the first systematic study of federated training applied to speech language model (SpeechLLM) architectures for end-to-end automatic speech recognition. The authors design a communication-efficient federated optimization strategy addressing gradient overhead and distributed computational constraints, evaluated on English and Italian monolingual ASR tasks. Results show competitive word error rates with reduced communication costs compared to centralized baselines, with ablation studies covering speech encoder architecture choices.
Related guides (2)
Related events (8)
Audio-LLM-based data filtering for speech-to-speech translation via Rank-to-Distill
A new arXiv paper proposes using audio large language models to filter noisy training data for end-to-end speech-to-speech translation (S2ST). The authors introduce a two-stage Rank-to-Distill strategy: a lightweight ranker generates pseudo-labels from noisy speech pairs, which then supervise an audio-LLM to make keep/drop decisions directly from raw audio. Experiments on CVSS-C and SpeechMatrix benchmarks show up to +1.4 ASR-BLEU improvement over unfiltered baselines.
Synthetic LLM-generated conversations improve ASR training for low-resource languages
Researchers propose a pipeline that uses LLMs to generate scenario-level dialogues and TTS to synthesize multi-speaker audio, creating simulated conversational training data for ASR systems. Evaluated on the Hungarian BEA-Dialogue benchmark, a model trained on 67 hours of real plus 636 hours of synthetic data outperforms a zero-shot model trained on 2,700 hours of real Hungarian speech. The study tests five LLM families under multiple budget and mixing configurations using a FastConformer-Large backbone, finding that generator choice and data composition significantly affect gains.
AlignAtt4LLM adapts simultaneous speech translation policy to decoder-only LLMs for IWSLT 2026
Researchers present AlignAtt4LLM, a simultaneous speech translation system for IWSLT 2026 covering English to German, Italian, and Chinese. The system cascades Qwen3-ASR for incremental transcription with Gemma-4 E4B-it for translation, applying a novel AlignAtt policy adapted for decoder-only LLMs that lack encoder-decoder cross-attention. Key contributions include explicit source span prompting, offline alignment head selection, and query/key capture to recover a usable attention-based read/write policy. The system outperforms IWSLT 2026 baselines for European language pairs in both low- and high-latency regimes.
FedLAB: Traceable semantic codebooks for federated multimodal graph foundation learning
FedLAB is a new federated learning framework for multimodal graph foundation models that organizes knowledge into typed hierarchical codebooks covering modality evidence, node semantics, and topology context. The system enables semantic traceability under strict data isolation, addressing a gap where existing methods exchange knowledge through parameters or embeddings without exposing how evidence jointly supports predictions. Evaluated on 10 benchmarks and 6 downstream tasks, FedLAB improves over state-of-the-art baselines by up to 7.53% while keeping raw data local.
MEUSLI: open-source multilingual projector linking Whisper to LLMs for 28-language ASR
MEUSLI is a new open-science multilingual projector family that connects a Whisper speech encoder to open-source multilingual LLMs, enabling end-to-end ASR across 28 European languages. The system extends prior monolingual pipelines using continual learning techniques to support additional unseen languages and generalizes beyond ASR to speech translation and topic identification with minimal task-specific supervision. The work positions itself as a foundation for scalable, inclusive open-source SpeechLLM development.
AuRA: Distilling audio understanding into LLMs via LoRA adaptation
AuRA is a new method for integrating speech understanding into LLMs by distilling audio encoding capability directly into LoRA-adapted model weights, bypassing cascaded ASR-LLM pipelines. A lightweight audio embedding layer feeds speech to both an ASR encoder (teacher) and a LoRA-adapted LLM (student), with layer-wise distillation aligning hidden states. The approach claims to outperform cascaded systems, bridge-based adaptation baselines, and large-scale multimodal models on multiple speech-language benchmarks while enabling parallel end-to-end inference without large-scale multimodal training.
Lychee-FD: Hierarchical parameter separation tackles modality interference in full-duplex spoken language models
Researchers introduce Lychee-FD, a native end-to-end full-duplex Spoken Language Model (SLM) framework that addresses modality interference between acoustic and semantic processing. The paper identifies gradient conflicts arising from shared deep parameter spaces as the root cause of performance degradation in full-duplex SLMs, and proposes a hierarchical parameter separation strategy with a dedicated semantic alignment channel. Experiments show +7.4% improvement on Spoken QA and +28.5% on FullDuplexBench 1.5 without inference efficiency loss.
RL-based alignment improves interactivity in full-duplex spoken dialogue models
Researchers propose a post-training alignment method using reinforcement learning to improve interactivity in full-duplex spoken dialogue models, which can listen and speak simultaneously. The method addresses four canonical axes of interactivity—pause handling, turn-taking, backchanneling, and user interruption—each with axis-specific reward functions, plus an LLM-based reward to prevent semantic degradation. The approach is applied to two open-source models, Moshi and PersonaPlex, showing consistent improvements in both offline and real-time multi-turn evaluation.