4arXiv cs.CL (Computation and Language)·
SeRIn: Decomposed multimodal fusion architecture achieves state-of-the-art on sentiment analysis benchmarks
Researchers propose SeRIn (Segregate, Refine, Integrate), a multimodal language model fusion scheme that separates modality-specific refinement from cross-modal interaction via distinct architectural pathways. The design defers full cross-modal interaction to a final prediction step, with ablations showing structured interaction rather than added capacity drives performance gains. SeRIn achieves state-of-the-art results on CH-SIMS and CMU-MOSEI benchmarks, and exhibits emergent modality reweighting under visual corruption without explicit supervision.
Related guides (2)
Related events (8)
RDS Fusion: Hybrid neuro-symbolic gating with compressed CoT for zero-shot irony detection
Researchers introduce the Robust Dual-Signal (RDS) Fusion framework, a hybrid neuro-symbolic architecture that compresses Chain-of-Thought reasoning without supervised fine-tuning for irony and sarcasm detection in social media text. Evaluated on TweetEval (N=734) and iSarcasm, the zero-shot system matches fine-tuned BERTweet performance and outperforms supervised SemEval transformer ensembles on the imbalanced iSarcasm dataset. A statistical ablation shows that only the full concurrent fusion of all three signals yields a validated improvement, with individual components providing no significant standalone gain.
Cross-modal masking framework improves silent speech synthesis from sEMG and lipreading
Researchers propose a masked multimodal speech synthesis framework that jointly trains on surface electromyography (sEMG) and video-based lipreading signals using modality masking to improve robustness to sensor failure or degradation. In multispeaker settings, the approach reduces word error rate by up to 14 absolute percentage points over the strongest unimodal baseline. Masking strategies outperform degradation-specific data augmentation for handling missing modalities, with phone-level analysis revealing complementary contributions across vowels and consonant groups.
Multimodal Embedding & Reranker Models with Sentence Transformers
Hugging Face's Sentence Transformers library has added support for multimodal embedding and reranking models, enabling joint text-image (and potentially other modality) representations within a unified framework. The update extends the library's existing text-focused embedding capabilities to handle cross-modal retrieval and reranking tasks. This lowers the barrier for practitioners building multimodal search and RAG pipelines using open-weights models.
Training and Finetuning Multimodal Embedding & Reranker Models with Sentence Transformers
Hugging Face published a blog post detailing how to train and finetune multimodal embedding and reranker models using the Sentence Transformers library. The post covers techniques for building models that can jointly embed text and images for retrieval and reranking tasks. This represents an extension of the Sentence Transformers ecosystem into multimodal territory, enabling practitioners to build cross-modal search and ranking systems.
LoMo: Local Modality Substitution for Deeper Vision-Language Fusion
This paper identifies a 'carrier sensitivity' problem in Vision-Language Models (VLMs), where replacing textual queries with rendered-image equivalents causes significant performance degradation due to asymmetric roles of text and images in training data. The authors propose Local Modality Substitution (LoMo), a data curation paradigm that reformulates single-modality prompts into interleaved multimodal sequences by dynamically rendering text spans as images, enforcing cross-modal representational invariance. Evaluated across 13 multimodal benchmarks, LoMo improves over standard supervised fine-tuning by 2.67 points on LLaVA-OneVision-1.5-8B and 2.82 points on Qwen3.5-9B. The approach is architecture-agnostic and lightweight, requiring no changes to model architecture.
FusionRS: Large-scale RGB-infrared-text dataset for dual-modal remote sensing vision-language models
Researchers introduce FusionRS, the first large-scale dataset pairing RGB and infrared remote sensing images with both conventional and IR-aware text captions, designed to support dual-modal vision-language learning. The dataset is constructed by translating public RGB remote sensing images into infrared-style counterparts using image translation. Using FusionRS, the authors train CLIP-style alignment models and fine-tune generative VLMs, demonstrating improvements in RGB-IR alignment, infrared-to-text retrieval, and dual-modal captioning over RGB-only baselines. The work addresses a gap in multimodal remote sensing foundation models by providing modality-specific textual supervision for infrared imagery.
Light-MER: Sub-1B parameter multimodal emotion recognition via knowledge distillation
Researchers from GAIR Lab propose Light-MER, a lightweight framework for multimodal emotion recognition (MER) that uses knowledge distillation to transfer capabilities from large teacher models (7B+) to sub-1B student models. Two novel optimization strategies are introduced: an optimal transport loss combining Sliced Wasserstein Distance with hidden-state alignment, and a multi-reward optimization strategy based on GRPO. Experiments across nine benchmark datasets show Light-MER achieves state-of-the-art performance with substantially improved inference efficiency, challenging the assumption that large models are necessary for high-quality MER.
Steerable Model Merging (ST-Merge) improves multilingual reasoning via adaptive gated cross-attention
Researchers propose ST-Merge, a framework for adaptively merging a multilingual model and a reasoning model using a gated cross-attention mechanism that weights each source model's contribution based on input characteristics. The approach addresses the limitation of static one-size-fits-all merging strategies that fail to resolve conflicts between source models. Experiments across 21 languages on four multilingual reasoning benchmarks show consistent improvements over strong baselines.