7arXiv cs.CL (Computation and Language)·
Nemotron-Labs Audex-30B: Unified audio-text MoE LLM with no text regression
Nvidia's Nemotron Labs introduces Audex-30B-A3B, a 30B-parameter mixture-of-experts audio-text LLM built on the Nemotron-Cascade-2 text backbone. The model handles audio understanding, ASR, translation, TTS, and speech-to-speech generation within a single Transformer decoder by projecting audio into the text embedding space. Training used 157.4B audio tokens and 320.5B text tokens with multi-stage supervised learning, RL, and on-policy distillation. Model checkpoints are publicly released, and the authors report state-of-the-art audio performance with minimal regression on text reasoning and agentic capabilities.
Related guides (4)
Related events (8)
UniAudio-Token: Semantic Speech Tokenizer with General Audio Perception for Audio-LLMs
UniAudio-Token is a framework from Tencent that extends semantic speech tokenizers—commonly used as interfaces for Audio-LLMs—to support general audio perception without sacrificing speech quality. It introduces two mechanisms: Semantic-Acoustic Primitives (SAP) for structured supervision decomposing audio into linguistic, vocal, and auditory-scene components, and Semantic-Acoustic Equilibrium (SAE), a content-aware gating mechanism that restores fine-grained acoustic details from shallow layers. Evaluations show it outperforms all single-codebook baseline tokenizers on both understanding and generation tasks when integrated with downstream LLMs. Code, training/inference scripts, and model checkpoints are publicly released.
Introducing NVIDIA Nemotron 3 Nano Omni: Long-Context Multimodal Intelligence for Documents, Audio and Video Agents
NVIDIA has released Nemotron 3 Nano Omni, a multimodal model targeting long-context understanding across documents, audio, and video modalities. The model is positioned for agentic use cases requiring cross-modal reasoning. It is published via the Hugging Face blog as part of NVIDIA's Nemotron model family. No detailed technical specifications or benchmark results are provided in the available body text.
Thinking Machines' TML-Interaction-Small 276B-A12B Advances SOTA Realtime Voice and VAD
Thinking Machines has released TML-Interaction-Small, a 276B-A12B mixture-of-experts model targeting native interaction capabilities including realtime voice. The model is reported to advance state-of-the-art in realtime voice interaction and supersedes standard voice activity detection (VAD) approaches. The item is a brief AINews digest entry from Latent Space with minimal technical detail beyond the headline claims.
Mistral AI Releases Voxtral: Open-Weight Speech Understanding Models in 24B and 3B Sizes
Mistral AI has released Voxtral, a family of two open-weight speech understanding models (Voxtral Small at 24B and Voxtral Mini at 3B) under the Apache 2.0 license. Both models support long-form audio up to 30-40 minutes, native multilingual transcription, built-in Q&A and summarization, and function-calling directly from voice, built on the Mistral Small 3.1 language model backbone. Benchmarks show Voxtral outperforms Whisper large-v3 across all tasks and is competitive with GPT-4o mini and Gemini 2.5 Flash on audio understanding, while pricing starts at $0.001/minute via API. Models are available on Hugging Face and through Mistral's API, with a transcription-optimized variant (Voxtral Mini Transcribe) also offered.
Mistral Releases Voxtral TTS: 4B-Parameter Multilingual Text-to-Speech Model
Mistral AI has launched Voxtral TTS, its first text-to-speech model, built on a 4B-parameter transformer-based autoregressive flow-matching architecture derived from Ministral 3B. The model supports 9 languages with zero-shot voice adaptation from as little as 3 seconds of reference audio, achieving 70ms latency for typical inputs and a real-time factor of ~9.7x. Human evaluations claim superior naturalness compared to ElevenLabs Flash v2.5 and parity with ElevenLabs v3. The model is available via Mistral Studio and API, targeting enterprise voice agent workflows.
AudioDER: Deduplication-enhanced reasoning dataset for post-training large audio-language models
Researchers introduce AudioDER, a ~191k-sample post-training dataset for Large Audio-Language Models (LALMs) built via an acoustic similarity-based deduplication pipeline to reduce redundancy and improve corpus diversity. Each sample pairs an audio clip with a multiple-choice question, answer candidates, a caption, and a chain-of-thought rationale generated by Qwen3-30B. Post-training Qwen2-Audio-7B-Instruct on AudioDER yields consistent gains on audio reasoning benchmarks including MMAU-mini, MMSU, and MMAR. The work addresses a data quality gap in audio-language training rather than proposing a new model architecture.
OpenAI Updates Audio Models That Reason, Transcribe, and Translate
OpenAI introduced three new audio models in its Realtime API: GPT-Realtime-2 (speech-to-speech with five configurable reasoning effort levels), GPT-Realtime-Translate (70+ input languages), and GPT-Realtime-Whisper (transcription). GPT-Realtime-2 operates as an end-to-end audio model including reasoning, with latency ranging from 1.12 seconds at minimal effort to 2.33 seconds at high effort. Benchmark results are mixed: it leads Scale AI's Audio MultiChallenge and Artificial Analysis Conversational Dynamics but trails Step-Audio R1.1 Realtime and Grok Voice Think Fast 1.0 on speech reasoning and agentic tasks. The configurable reasoning-latency tradeoff is positioned as a key differentiator for voice agent applications.
AuRA: Distilling audio understanding into LLMs via LoRA adaptation
AuRA is a new method for integrating speech understanding into LLMs by distilling audio encoding capability directly into LoRA-adapted model weights, bypassing cascaded ASR-LLM pipelines. A lightweight audio embedding layer feeds speech to both an ASR encoder (teacher) and a LoRA-adapted LLM (student), with layer-wise distillation aligning hidden states. The approach claims to outperform cascaded systems, bridge-based adaptation baselines, and large-scale multimodal models on multiple speech-language benchmarks while enabling parallel end-to-end inference without large-scale multimodal training.