VSDD-1K

FacialTalker: Facial-expression-aware conversational speech synthesis with DualDPO and VSDD-1K dataset

Researchers introduce FacialTalker, a conversational speech synthesis framework built on a large language model backbone that incorporates facial expression signals via AUTokenizer, a visual tokenizer trained on facial Action Units. A dual direct preference optimization (DualDPO) strategy jointly constrains visual and speech token sequences to improve multimodal alignment. The authors also release VSDD-1K, a 1,033-hour multimodal dialogue dataset of synchronized speaker video and speech collected from real-world internet conversations. Experiments show FacialTalker outperforms baselines on expression perception and speech naturalness.