4arXiv cs.CL (Computation and Language)·
Light-MER: Sub-1B parameter multimodal emotion recognition via knowledge distillation
Researchers from GAIR Lab propose Light-MER, a lightweight framework for multimodal emotion recognition (MER) that uses knowledge distillation to transfer capabilities from large teacher models (7B+) to sub-1B student models. Two novel optimization strategies are introduced: an optimal transport loss combining Sliced Wasserstein Distance with hidden-state alignment, and a multi-reward optimization strategy based on GRPO. Experiments across nine benchmark datasets show Light-MER achieves state-of-the-art performance with substantially improved inference efficiency, challenging the assumption that large models are necessary for high-quality MER.
Related guides (3)
Related events (8)
HumP-KD: Uncertainty-aware multi-stage knowledge distillation for efficient fire classification
Researchers propose HumP-KD, a knowledge distillation framework that compresses two heterogeneous transformer teachers (Swin-Tiny and ViT-Base) into a lightweight MobileViT-S student for real-time fire classification. The student model achieves 0.9876 mean F1 on a 31K-image dataset while retaining only 4.94M parameters—a 5.7× reduction over Swin-Tiny—and runs at 37.72 CPU FPS. The framework combines hierarchical feature alignment, spatial attention masking, and progressive multi-stage distillation to maintain accuracy under degraded visual conditions.
MOPD: Multi-Teacher On-Policy Distillation for integrating multiple RL-trained capabilities in LLMs
Researchers propose Multi-teacher On-Policy Distillation (MOPD), a post-training paradigm that first trains domain-specialized RL teacher models, then distills them into a student model using on-policy rollouts to eliminate exposure bias. Evaluated on Qwen3-30B-A3B, MOPD outperforms Mix-RL, Cascade RL, Off-Policy Finetune, and Param-Merge baselines while preserving nearly all per-domain capability. The method has been deployed in production for MiMo-V2-Flash, an industrial-scale frontier model, validating its practical applicability. The approach also enables parallel, decoupled development of domain teachers, reducing cross-domain interference in multi-capability post-training.
SeRIn: Decomposed multimodal fusion architecture achieves state-of-the-art on sentiment analysis benchmarks
Researchers propose SeRIn (Segregate, Refine, Integrate), a multimodal language model fusion scheme that separates modality-specific refinement from cross-modal interaction via distinct architectural pathways. The design defers full cross-modal interaction to a final prediction step, with ablations showing structured interaction rather than added capacity drives performance gains. SeRIn achieves state-of-the-art results on CH-SIMS and CMU-MOSEI benchmarks, and exhibits emergent modality reweighting under visual corruption without explicit supervision.
MobileMoE: Scaling Mixture-of-Experts for Sub-Billion Parameter On-Device Deployment
MobileMoE introduces a family of on-device MoE language models with 0.3–0.9B active parameters and 1.3–5.3B total parameters, targeting mobile deployment under memory and compute constraints. The authors derive an on-device MoE scaling law identifying a sweet spot of moderate sparsity with fine-grained and shared experts, then train models through a four-stage recipe including quantization-aware training on open-source data. Across 14 benchmarks, MobileMoE matches or exceeds leading dense on-device LLMs with 2–4× fewer inference FLOPs, and delivers 1.8–3.8× faster prefill and 2.2–3.4× faster decode than dense baselines on commodity smartphones at comparable INT4 memory.
Vision-OPD: On-Policy Self-Distillation for Fine-Grained Visual Understanding in MLLMs
Vision-OPD addresses a 'regional-to-global perception gap' in multimodal LLMs, where models answer fine-grained visual questions more accurately when given cropped evidence regions than full images. The method instantiates a crop-conditioned teacher and full-image-conditioned student from the same MLLM, minimizing token-level divergence along on-policy rollouts to transfer regional perception to the full-image policy. This self-distillation requires no external teacher models, ground-truth labels, reward verifiers, or inference-time tools. Benchmarks show competitive or superior performance against larger open-source, closed-source, and agentic 'Thinking-with-Images' models.
LightMem-Ego: Lightweight streaming multimodal memory system for egocentric AI assistants
Researchers from Zhejiang University NLP Lab introduce LightMem-Ego, a lightweight multimodal memory system designed for mobile and wearable AI assistants. The system continuously ingests egocentric visual and audio streams, organizes them into a hierarchical three-tier memory (current, short-term, long-term), and routes retrieval dynamically based on user queries. It targets practical deployment on smartphones and AI glasses for tasks like object finding, conversation recall, and routine discovery, with code released publicly.
CHERRY: Compressed Hierarchical Experts with Recurrent Representational Yield — three compute-efficient LM training techniques
A preprint from arXiv introduces CHERRY, a suite of three complementary techniques for compute-efficient language model training: Selective Ground Truth Token Training (SGT) that concentrates supervision on ~15% of semantically loaded tokens while recovering ~67% of full-sequence loss reduction; depth compression that shrinks a 48-layer 1B-parameter model to 6 layers (227M) via layer averaging and recurrent unrolling, matching a 566M dense model's loss; and a Mixture of Efficient Experts (MoEE) assembly that outperforms individual compressed models at comparable active parameters. The techniques are validated on CHERRY-1.8B, a Korean-language foundation model trained entirely from scratch using these methods. Authors are transparent about scope limitations: one model family, Korean data, and loss-based metrics only.
Mach-Mind-4-Flash: 35B MoE agentic model matching 100B-class performance via post-training optimization
Mach-Mind-4-Flash is a 35B-parameter Mixture-of-Experts model with only 3B activated parameters that achieves performance comparable to 100B-class models through post-training techniques alone. The pipeline combines a unified RL/OPD training infrastructure with multi-teacher scheduling, parallel domain-specific RL experts fused via Multi-Teacher On-Policy Distillation (MOPD), and Hybrid Median-length Policy Optimization (HMPO) which compresses reasoning chains 19-46% with minimal accuracy loss. Benchmark results include 92.70 on AIME'26, 82.82 on IFBench, and 75.80 on BFCL-v4, claiming to lead or match models 10-30x its activated size at a fraction of inference cost. The work is notable for demonstrating that post-training optimization can close large gaps in activated parameter count for agentic tasks.