4arXiv cs.CL (Computation and Language)·
FacialTalker: Facial-expression-aware conversational speech synthesis with DualDPO and VSDD-1K dataset
Researchers introduce FacialTalker, a conversational speech synthesis framework built on a large language model backbone that incorporates facial expression signals via AUTokenizer, a visual tokenizer trained on facial Action Units. A dual direct preference optimization (DualDPO) strategy jointly constrains visual and speech token sequences to improve multimodal alignment. The authors also release VSDD-1K, a 1,033-hour multimodal dialogue dataset of synchronized speaker video and speech collected from real-world internet conversations. Experiments show FacialTalker outperforms baselines on expression perception and speech naturalness.
Related guides (3)
Related events (8)
Multimodal Voice Activity Projection framework for turn-taking prediction in social robots
Researchers introduce MM-VAP, a Multimodal Voice Activity Projection framework that extends audio-only turn-taking prediction to synchronized audio-visual inputs for use in social robots acting as conversation mediators. The approach uses pretrained audio-visual backbones adapted via Low-Rank Adaptation, an inter-speaker attention stage, and a semantic consistency loss to regularize the output space. Experiments on the NoXi, NoXi+J, and Haru EDR corpora show improvements over baselines on several turn-taking events. The work targets human-robot interaction scenarios where anticipating conversational dynamics is required rather than simply reacting to pauses.
VideoFDB: First Benchmark for Full-Duplex Audio-Visual Conversational Agent Evaluation
VideoFDB is introduced as the first benchmark targeting full-duplex audio-visual-to-audio-visual (AV2AV) conversational agents, filling a gap where existing full-duplex benchmarks evaluate only speech. It provides 237 dyadic video-call clips covering 11 nonverbal conversational dynamics, a perception/generation taxonomy, and an LM-as-judge rubric framework. Evaluation across open- and closed-source vision-speech agents reveals systematic failure modes including captioning collapse and visual-stream ignorance, and shows current systems cannot perform the streaming joint audiovisual grounding required for natural conversation. Cascaded speech-to-avatar architectures are found to be architecturally incapable of producing full-duplex nonverbal cues.
WordVoice: Word-level acoustic control framework for LLM-based TTS with 4.7k-hour bilingual dataset
Researchers propose WordVoice, a framework enabling explicit, decoupled word-level control over acoustic attributes (duration, boundary, energy, pitch, tone) in LLM-based text-to-speech systems. The work introduces a bound-token mechanism for acoustic planning within the LLM and a fine-grained modulation module to align discrete tokens with continuous waveforms. Accompanying the model is WordVoice-5A, a 4.7k-hour bilingual annotated dataset with five-dimensional word-level labels. The system targets high-precision applications such as audiobook narration and video dubbing where implicit TTS generation falls short.
Cross-modal masking framework improves silent speech synthesis from sEMG and lipreading
Researchers propose a masked multimodal speech synthesis framework that jointly trains on surface electromyography (sEMG) and video-based lipreading signals using modality masking to improve robustness to sensor failure or degradation. In multispeaker settings, the approach reduces word error rate by up to 14 absolute percentage points over the strongest unimodal baseline. Masking strategies outperform degradation-specific data augmentation for handling missing modalities, with phone-level analysis revealing complementary contributions across vowels and consonant groups.
BayLing-Duplex: Native full-duplex speech dialogue using a single autoregressive LLM
Researchers introduce BayLing-Duplex, a speech language model that achieves native full-duplex interaction — simultaneous listening and speaking — using a single autoregressive LLM with no auxiliary VAD or turn-taking module. Built by fine-tuning GLM-4-Voice on 400K samples plus a lightweight DPO stage, it reaches 92% turn-taking success and 100% interruption success on InstructS2S-Eval, and improves speech-response quality substantially over Moshi. The approach adds only special tokens to the standard vocabulary, making it portable across LLM architectures without architectural changes.
RL-based alignment improves interactivity in full-duplex spoken dialogue models
Researchers propose a post-training alignment method using reinforcement learning to improve interactivity in full-duplex spoken dialogue models, which can listen and speak simultaneously. The method addresses four canonical axes of interactivity—pause handling, turn-taking, backchanneling, and user interruption—each with axis-specific reward functions, plus an LLM-based reward to prevent semantic degradation. The approach is applied to two open-source models, Moshi and PersonaPlex, showing consistent improvements in both offline and real-time multi-turn evaluation.
DramaSR-LRM: Reasoning LLM with multimodal tool-use for speaker recognition in TV dramas
Researchers introduce DramaSR-532K, a large-scale benchmark of 532K annotated dialogue lines across 900+ characters from long-form TV dramas, targeting multimodal speaker recognition. They also propose DramaSR-LRM, a system built on a large reasoning model that uses multimodal tool-use to aggregate auditory, linguistic, and visual cues for speaker attribution. The approach significantly outperforms baselines, especially on short utterances where acoustic biometrics alone are unreliable. Data and code are to be publicly released.
End-to-end speech-to-speech conversational system for Algerian Dialect using modular NLP pipeline
Researchers present Dziri Voicebot, a modular speech-to-speech conversational system targeting Algerian Dialect, a low-resource language with codeswitching and orthographic challenges. The pipeline integrates Whisper-based ASR, transformer NLU, retrieval-augmented generation, and neural TTS, with dedicated datasets constructed for the telecom domain. The system reports low word error rate, high intent classification scores, and stable TTS quality, offering a reproducible baseline for low-resource dialectal conversational AI.