6arXiv cs.AI (Artificial Intelligence)·
MODUS: Decoder-only architecture for any-to-any multimodal modeling without modality-specific heads
Researchers at EPFL introduce Modus, a decoder-only model that treats all modalities symmetrically for any-to-any prediction without modality-specific heads, losses, or task pipelines. Unlike prior any-to-any models that use encoder-decoder or diffusion architectures trained from scratch, Modus leverages pre-trained decoder-only models as a prior. The model supports chained generation through intermediate modalities and cross-modal self-verification, achieving competitive performance with specialist and multitask baselines across benchmarks. All materials are open-sourced.
Related guides (2)
Related events (8)
SeRIn: Decomposed multimodal fusion architecture achieves state-of-the-art on sentiment analysis benchmarks
Researchers propose SeRIn (Segregate, Refine, Integrate), a multimodal language model fusion scheme that separates modality-specific refinement from cross-modal interaction via distinct architectural pathways. The design defers full cross-modal interaction to a final prediction step, with ablations showing structured interaction rather than added capacity drives performance gains. SeRIn achieves state-of-the-art results on CH-SIMS and CMU-MOSEI benchmarks, and exhibits emergent modality reweighting under visual corruption without explicit supervision.
Co-learning framework for multi-modal classification with arbitrary missing modalities
A new arXiv preprint introduces a co-learning framework for multi-modal classification that handles arbitrary missing modalities at inference time, without assuming predefined missing patterns. Two complementary methods are proposed — one operating at the feature level and one at the decision level — evaluated on two multi-modal benchmarks. The approach prioritizes inter-modal collaboration over fusion, showing robustness gains across both minimal and extreme missing-modality conditions.
Efficient MultiModal Data Pipeline (MMDP) from Hugging Face
Hugging Face published a blog post describing an efficient multimodal data pipeline (MMDP) for processing and preparing multimodal training data at scale. The post covers architectural choices and tooling for handling diverse data modalities in ML workflows. As a tier-2 source with default commentary depth, the technical substance is likely focused on practical data engineering patterns for multimodal model training.
MOJO framework combines masked autoencoding with supervised learning for generalizable neural population decoding
Researchers introduce MOJO (Masked autOencoder-based JOint training), a training framework that augments spike-tokenizing neural decoding models with self-supervised learning via masked autoencoding alongside supervised objectives. Evaluated on spiking datasets from monkey motor cortex and multi-regional mouse recordings, MOJO outperforms purely supervised models, with especially strong gains in few-shot and label-limited settings. The framework also generalizes to human electrocorticography during speech, achieving performance comparable to neuro-foundation models designed for continuous signals. The work advances a path toward more scalable and flexible training of neural foundation models using unlabelled data.
AdaCodec: Predictive Visual Coding for Efficient Video MLLMs
AdaCodec introduces a predictive visual code interface for video multimodal large language models that exploits temporal redundancy in video. Instead of encoding every sampled frame as an independent RGB image, it sends full visual tokens only for reference frames with high conditional predictive cost, and encodes inter-frame changes as compact P-tokens. Evaluated against a Qwen3-VL-8B per-frame baseline across eleven benchmarks, AdaCodec at 1/7 the token budget (32k vs 224k tokens) surpasses the baseline on all long-video benchmarks while reducing time-to-first-token from 9.26s to 1.62s.
OFA: Towards Building a One-For-All Unified Multimodal Pretrained Model
Alibaba's Qwen team introduces OFA (One-For-All), a unified multimodal pretrained model designed to handle both understanding and generation tasks across multiple modalities within a single framework. The model is pretrained using instruction-based multitask pretraining to endow it with diverse capabilities. This work was published in late 2022 as part of the broader wave of generalist multimodal models. It represents an early effort toward a single model architecture capable of spanning vision, language, and cross-modal tasks.
Latent World Recovery: multimodal learning framework for missing modalities in bioscience
A new arXiv preprint introduces Latent World Recovery (LWR), a framework for multimodal learning when some modalities are unavailable at training or inference time. LWR aligns modality-specific embeddings in a shared latent space and fuses only available modalities, avoiding explicit reconstruction of missing ones. The approach is evaluated on incomplete multi-omics benchmarks for cancer phenotype classification and survival prediction, demonstrating robustness under partial observation.
BrainJanus: Unified autoregressive model for brain encoding/decoding across vision and language
BrainJanus is a unified model that integrates brain neural activity, vision, and language within a single autoregressive framework using next-token prediction. The system introduces a Unified Brain Tokenizer to quantize neural dynamics into discrete tokens aligned with visual and linguistic representations in a shared embedding space. It supports any-to-any generation including image-to-brain, text-to-brain, brain-to-image, and brain-to-text tasks, with reported zero-shot generalization and interpretable biological topography. The work positions itself as a general-purpose brain modeling paradigm at the intersection of neuroscience and multimodal AI.