5arXiv cs.AI (Artificial Intelligence)·
SceneBind: Omni-modal scene representation combining semantic and 3D spatial understanding across vision, audio, and language
SceneBind is a new multimodal representation framework that jointly encodes semantic and 3D spatial structure across vision, audio, and language modalities. It represents scenes as semantic-spatial entities using global embeddings plus object-centric slots, addressing a gap in existing omni-modal encoders that handle 'what' but not 'where'. The authors introduce a binaural audio-visual dataset with spatial annotations and a matching scheme for cross-modal retrieval and object grounding, achieving state-of-the-art on scene/spatial retrieval with strong zero-shot transfer to audio-visual localization.
Related guides (2)
Related events (8)
SCENT framework uses language guidance to bridge vision and olfactory sensor representations
Researchers introduce SCENT, a multimodal learning framework that uses Vision-Language Models to generate semantic scene descriptors as a bridge between electronic-nose (e-nose) signals and visual/textual representations. A smell encoder is trained to map e-nose signals into a shared embedding space, with a language-guided latent decomposition separating object-specific odors from ambient environmental contributions. Evaluated on the New York Smells dataset, SCENT achieves state-of-the-art cross-modal retrieval on smell-to-image and smell-to-text tasks, outperforming vision-only baselines. The work extends multimodal learning into olfaction, a largely unexplored sensory modality.
Information-theoretic formalization of the binding problem in Vision Transformers
Researchers introduce a formal information-theoretic framework for the binding problem — the challenge of associating features (color, shape) with the correct objects in multi-object scenes. They develop a probing method to measure binding information in model representations and apply it to several pre-trained Vision Transformers, examining components like the [CLS] token and spatial tokens across datasets with feature sharing, occlusion, and natural features. Results position binding information as a key factor in visual recognition and reasoning quality, and suggest current ViT architectures have limited binding capability, consistent with known failure modes.
OneCanvas achieves state-of-the-art 3D scene understanding via panoramic reprojection in VLMs
OneCanvas is a new method for 3D scene understanding in Vision-Language Models that aggregates multi-view patch features onto a single equirectangular panoramic canvas using depth and camera pose, avoiding complex geometry encoders or large training budgets. A 3D position embedding restores metric depth information lost during angular projection, and a spatial pretraining curriculum generates on-the-fly supervision for spatial reasoning tasks. The approach achieves state-of-the-art results on SQA3D and VSI-Bench benchmarks while using an order of magnitude less training compute than competing methods, and supports situated reasoning relevant to robotics and embodied AI.
OmniAgent: POMDP-based active perception agent for long video understanding with test-time scaling
Researchers introduce OmniAgent, a multimodal agent that reformulates long video understanding as a POMDP-based iterative Observation-Thought-Action cycle, selectively distilling audio-visual cues into persistent textual memory rather than processing all frames uniformly. The system uses Agentic Supervised Fine-Tuning and a novel reinforcement learning method (TAURA) with turn-level entropy for credit assignment. OmniAgent demonstrates positive test-time scaling and achieves state-of-the-art open-source results across ten benchmarks, with its 7B model outperforming Qwen2.5-VL-72B on LVBench (50.5% vs. 47.3%).
SeRIn: Decomposed multimodal fusion architecture achieves state-of-the-art on sentiment analysis benchmarks
Researchers propose SeRIn (Segregate, Refine, Integrate), a multimodal language model fusion scheme that separates modality-specific refinement from cross-modal interaction via distinct architectural pathways. The design defers full cross-modal interaction to a final prediction step, with ablations showing structured interaction rather than added capacity drives performance gains. SeRIn achieves state-of-the-art results on CH-SIMS and CMU-MOSEI benchmarks, and exhibits emergent modality reweighting under visual corruption without explicit supervision.
CSB dataset benchmarks a decade of VLM progress on complex social scene understanding
Researchers introduce the Complex Social Behavior (CSB) dataset of 100 images depicting complex social interactions, used to evaluate nine vision-language models spanning 2017–2025 against human descriptions and a gold standard. MLLMs have largely closed the accuracy gap with top-ranked human descriptions and nearly eliminated most error types (object detection, recognition, hallucination, scene understanding), with spatial dependence errors being the notable remaining failure mode. The study also finds that MLLMs have eliminated the accuracy gap between simple MS-COCO scenes and complex social scenes, a gap that pre-MLLM models struggled with significantly.
LoMo: Local Modality Substitution for Deeper Vision-Language Fusion
This paper identifies a 'carrier sensitivity' problem in Vision-Language Models (VLMs), where replacing textual queries with rendered-image equivalents causes significant performance degradation due to asymmetric roles of text and images in training data. The authors propose Local Modality Substitution (LoMo), a data curation paradigm that reformulates single-modality prompts into interleaved multimodal sequences by dynamically rendering text spans as images, enforcing cross-modal representational invariance. Evaluated across 13 multimodal benchmarks, LoMo improves over standard supervised fine-tuning by 2.67 points on LLaVA-OneVision-1.5-8B and 2.82 points on Qwen3.5-9B. The approach is architecture-agnostic and lightweight, requiring no changes to model architecture.
Introducing ConTextual: Benchmark for Joint Text-Image Reasoning in Text-Rich Scenes
Hugging Face introduces ConTextual, a new benchmark evaluating multimodal models on their ability to jointly reason over text and images in text-rich scenes. The benchmark targets a specific capability gap where models must integrate visual and textual information simultaneously rather than treating them independently. A leaderboard accompanies the benchmark to track model progress on this task.