3arXiv cs.CL (Computation and Language)·
Multimodal Voice Activity Projection framework for turn-taking prediction in social robots
Researchers introduce MM-VAP, a Multimodal Voice Activity Projection framework that extends audio-only turn-taking prediction to synchronized audio-visual inputs for use in social robots acting as conversation mediators. The approach uses pretrained audio-visual backbones adapted via Low-Rank Adaptation, an inter-speaker attention stage, and a semantic consistency loss to regularize the output space. Experiments on the NoXi, NoXi+J, and Haru EDR corpora show improvements over baselines on several turn-taking events. The work targets human-robot interaction scenarios where anticipating conversational dynamics is required rather than simply reacting to pauses.
Related guides (1)
Related events (8)
ModeratorLM: Role-conditioned turn-taking for multi-party voice agents with 40%+ precision gains
Researchers introduce ModeratorLM, a voice agent system that conditions turn-taking behavior on an explicitly assigned conversational role in multi-party settings, built on a streaming speech LLM. A reasoning-augmented variant adds chain-of-thought over conversational context. Evaluated on real-world meeting data and the new RolePlayConv synthetic dataset, the system achieves over 40% improvement in turn-taking precision and 70% in recall while reducing false-positive interruptions versus non-role-conditioned baselines.
Cross-modal masking framework improves silent speech synthesis from sEMG and lipreading
Researchers propose a masked multimodal speech synthesis framework that jointly trains on surface electromyography (sEMG) and video-based lipreading signals using modality masking to improve robustness to sensor failure or degradation. In multispeaker settings, the approach reduces word error rate by up to 14 absolute percentage points over the strongest unimodal baseline. Masking strategies outperform degradation-specific data augmentation for handling missing modalities, with phone-level analysis revealing complementary contributions across vowels and consonant groups.
OmniAgent: POMDP-based active perception agent for long video understanding with test-time scaling
Researchers introduce OmniAgent, a multimodal agent that reformulates long video understanding as a POMDP-based iterative Observation-Thought-Action cycle, selectively distilling audio-visual cues into persistent textual memory rather than processing all frames uniformly. The system uses Agentic Supervised Fine-Tuning and a novel reinforcement learning method (TAURA) with turn-level entropy for credit assignment. OmniAgent demonstrates positive test-time scaling and achieves state-of-the-art open-source results across ten benchmarks, with its 7B model outperforming Qwen2.5-VL-72B on LVBench (50.5% vs. 47.3%).
RL-based alignment improves interactivity in full-duplex spoken dialogue models
Researchers propose a post-training alignment method using reinforcement learning to improve interactivity in full-duplex spoken dialogue models, which can listen and speak simultaneously. The method addresses four canonical axes of interactivity—pause handling, turn-taking, backchanneling, and user interruption—each with axis-specific reward functions, plus an LLM-based reward to prevent semantic degradation. The approach is applied to two open-source models, Moshi and PersonaPlex, showing consistent improvements in both offline and real-time multi-turn evaluation.
WordVoice: Word-level acoustic control framework for LLM-based TTS with 4.7k-hour bilingual dataset
Researchers propose WordVoice, a framework enabling explicit, decoupled word-level control over acoustic attributes (duration, boundary, energy, pitch, tone) in LLM-based text-to-speech systems. The work introduces a bound-token mechanism for acoustic planning within the LLM and a fine-grained modulation module to align discrete tokens with continuous waveforms. Accompanying the model is WordVoice-5A, a 4.7k-hour bilingual annotated dataset with five-dimensional word-level labels. The system targets high-precision applications such as audiobook narration and video dubbing where implicit TTS generation falls short.
Audio Interaction Model: Unified Streaming LALM with Always-On Perceive-Decide-Respond Loop
Researchers introduce the Audio Interaction Model framework and a concrete implementation called Audio-Interaction, a unified streaming Large Audio Language Model that handles both offline tasks and real-time audio interaction through a continuous perceive-decide-respond loop. The system is built on SoundFlow, a framework covering data construction, training, and asynchronous low-latency inference. The authors also release StreamAudio-2M, a 2.6M-item streaming corpus spanning 28 sub-tasks, and Proactive-Sound-Bench for evaluating proactive audio intervention. Evaluated across 8 benchmarks, the model preserves competitive offline performance while enabling real-time ASR, streaming instruction following, and proactive response capabilities not available in prior offline LALMs.
Thinking Machines' TML-Interaction-Small 276B-A12B Advances SOTA Realtime Voice and VAD
Thinking Machines has released TML-Interaction-Small, a 276B-A12B mixture-of-experts model targeting native interaction capabilities including realtime voice. The model is reported to advance state-of-the-art in realtime voice interaction and supersedes standard voice activity detection (VAD) approaches. The item is a brief AINews digest entry from Latent Space with minimal technical detail beyond the headline claims.
HAT-4D: Agentic framework for 4D multi-object interaction reconstruction from monocular video
HAT-4D is a new agentic framework that reconstructs 3D geometry, temporal dynamics, and physical interactions of multiple objects from single monocular videos, targeting scalable data collection for Embodied AI and Vision-Language-Action (VLA) model training. The system integrates VLMs with a multi-level human-in-the-loop feedback mechanism to resolve depth ambiguities and occlusions without expensive multi-camera rigs. The authors also introduce MVOIK-4D, an open-world benchmark for monocular 4D interaction reconstruction with a novel evaluation protocol focused on physical plausibility and temporal consistency. Experiments show state-of-the-art performance on most metrics, and HAT-4D-generated data improves downstream model fine-tuning.