4arXiv cs.AI (Artificial Intelligence)·
StoryTeller: Training-free framework for narrative-coherent long-form audio description of films
StoryTeller is a training-free framework that enables video-language models to produce story-aware audio descriptions for blind and low-vision audiences by maintaining a verified narrative memory across scenes. The system requires only raw video and a movie title, using semantic filtering and VLM verification to accept only video-supported facts, with no fine-tuning or precomputed character banks needed. The authors also introduce StoryAD-QA, a QA-based benchmark for evaluating narrative coherence in generated audio descriptions. Experiments show improvements over baselines on standard AD benchmarks and diverse long-form videos.
Related guides (3)
Related events (8)
WordVoice: Word-level acoustic control framework for LLM-based TTS with 4.7k-hour bilingual dataset
Researchers propose WordVoice, a framework enabling explicit, decoupled word-level control over acoustic attributes (duration, boundary, energy, pitch, tone) in LLM-based text-to-speech systems. The work introduces a bound-token mechanism for acoustic planning within the LLM and a fine-grained modulation module to align discrete tokens with continuous waveforms. Accompanying the model is WordVoice-5A, a 4.7k-hour bilingual annotated dataset with five-dimensional word-level labels. The system targets high-precision applications such as audiobook narration and video dubbing where implicit TTS generation falls short.
STORM: Internalized Spatial-Temporal Reasoning for Video-Language Models via Latent Trajectories
STORMS is a two-stage training framework that teaches large vision-language models to perform spatial-temporal video reasoning through bounded continuous latent trajectories rather than explicit textual chain-of-thought, keyframe selection, or external tool use. In Stage I, latent tokens are aligned with thought-video representations derived from generated videos; in Stage II, answer-only supervision internalizes the reasoning process. At inference time, no video regeneration or frame reinsertion is required, reducing latency and engineering complexity. Evaluations on VideoMME, MVBench, TempCompass, and MMVU show improved accuracy with substantially lower inference overhead versus tool-based pipelines.
Data2Story: Multi-agent framework for end-to-end data journalism with verifiable claims
Researchers introduce Data2Story (Data Journalist Agent), a multi-agent framework that orchestrates specialized roles to transform raw data into multimodal news articles. A key innovation is an Inspector module that grounds every claim back to data, code, or external references, enabling verifiability. The system also generates interactive multimodal outputs (maps, audio) rather than static text and charts. Evaluation across 18 articles with 53 human participants shows competitive quality versus expert-written pieces, with particular strength in transparency and auditability, though human journalists retain an edge in editorial angle and creative design.
DramaSR-LRM: Reasoning LLM with multimodal tool-use for speaker recognition in TV dramas
Researchers introduce DramaSR-532K, a large-scale benchmark of 532K annotated dialogue lines across 900+ characters from long-form TV dramas, targeting multimodal speaker recognition. They also propose DramaSR-LRM, a system built on a large reasoning model that uses multimodal tool-use to aggregate auditory, linguistic, and visual cues for speaker attribution. The approach significantly outperforms baselines, especially on short utterances where acoustic biometrics alone are unreliable. Data and code are to be publicly released.
LeVo 2: Hybrid LLM-Diffusion framework for stable full-length song generation with hierarchical modeling
LeVo 2 is a new hybrid LLM-Diffusion system for controllable full-length song generation that addresses the coherence-vs-acoustics trade-off through hierarchical token prediction: a language model handles semantic planning via mixed tokens, then predicts vocal and accompaniment tracks in parallel, while a diffusion-based codec reconstructs waveforms. A key contribution is an aesthetics-guided progressive post-training schedule combining SFT, offline DPO, and semi-online DPO to separately optimize quality, controllability, and musicality. Expert listening tests show LeVo 2 outperforms open-source baselines across six subjective dimensions and approaches leading commercial systems on several metrics.
Audio Interaction Model: Unified Streaming LALM with Always-On Perceive-Decide-Respond Loop
Researchers introduce the Audio Interaction Model framework and a concrete implementation called Audio-Interaction, a unified streaming Large Audio Language Model that handles both offline tasks and real-time audio interaction through a continuous perceive-decide-respond loop. The system is built on SoundFlow, a framework covering data construction, training, and asynchronous low-latency inference. The authors also release StreamAudio-2M, a 2.6M-item streaming corpus spanning 28 sub-tasks, and Proactive-Sound-Bench for evaluating proactive audio intervention. Evaluated across 8 benchmarks, the model preserves competitive offline performance while enabling real-time ASR, streaming instruction following, and proactive response capabilities not available in prior offline LALMs.
OmniAgent: POMDP-based active perception agent for long video understanding with test-time scaling
Researchers introduce OmniAgent, a multimodal agent that reformulates long video understanding as a POMDP-based iterative Observation-Thought-Action cycle, selectively distilling audio-visual cues into persistent textual memory rather than processing all frames uniformly. The system uses Agentic Supervised Fine-Tuning and a novel reinforcement learning method (TAURA) with turn-level entropy for credit assignment. OmniAgent demonstrates positive test-time scaling and achieves state-of-the-art open-source results across ten benchmarks, with its 7B model outperforming Qwen2.5-VL-72B on LVBench (50.5% vs. 47.3%).
Explainability pipeline reveals divergent cues used by deepfake speech detectors
Researchers propose an audio-native explainability pipeline using Integrated Gradients on time-aligned self-supervised representations to localize decision evidence in deepfake speech detectors. Applied to three WavLM-based detectors (AASIST, CA-MHFA, SLS) on the ASVspoof 5 benchmark, the method reveals that despite similar performance, each detector relies on fundamentally different cues: environmental noise, phoneme artifacts, and word boundaries respectively. Findings are validated via causal masking experiments that confirm performance degrades when primary cues are removed. The work advances interpretability of audio deepfake detection, relevant to AI safety and media authenticity.