4arXiv cs.CL (Computation and Language)·
MoE²-LoRA: MoE-style low-rank adaptation for Mixture-of-Experts language models
Researchers introduce MoE²-LoRA, a parameter-efficient fine-tuning method specifically designed for Mixture-of-Experts (MoE) language models. The approach uses a dual-channel Routing-Conditioned Projection module that reuses base router activations to guide LoRA adapter routing, and introduces a single global LoRA expert pool shared across all layers. Evaluated across multiple MoE backbones at varying scales, MoE²-LoRA claims state-of-the-art downstream accuracy while better preserving general capabilities compared to existing PEFT methods.
Related guides (3)
Related events (8)
Localized LoRA-MoE: Block-wise and cell-wise low-rank expert routing for multi-task PEFT
A new arXiv preprint introduces Localized LoRA-MoE, a parameter-efficient fine-tuning framework that combines spatial blocking with dynamic context-conditioned routing to address gradient interference in multi-task LoRA adapters. The authors propose two architectures: Block-Wise LoRA-MoE with centralized macro-routing and Cell-Wise LoRA-MoE with decentralized per-cell expert gating. Empirical results across SVD matrix simulations, tabular tasks, and spatial vision benchmarks show that decentralized cell-level gating matches a global coordinator while providing fault isolation against sensor degradation and task-switching instability.
Expert Tying reduces MoE LLM memory footprint by ~2x with minimal quality loss
Researchers introduce Expert Tying, an architectural modification for Mixture-of-Experts LLMs that shares expert parameters across consecutive transformer layers while keeping routing and attention layer-independent. Evaluated on OLMoE, Qwen3, and DeepSeek-style MoE architectures, the method achieves nearly 2x memory reduction with negligible perplexity or downstream quality degradation. The approach exploits parameter redundancy in MoE pathways to improve the compute-to-memory trade-off for training and inference.
LoRA mixture-of-experts variants for continual learning in motion-language agents
A new arXiv preprint investigates continual learning for bidirectional motion-language agents that must both understand and generate human motion without catastrophic forgetting. The authors propose LoRA-based mixture-of-experts architectures with an autoencoder-based router for task-specific expert selection at inference time, requiring no task labels. Evaluated on a five-task benchmark derived from HumanML3D, the approach achieves near-zero forgetting across motion-to-text and text-to-motion directions. A key finding is that hard expert selection outperforms soft blending, and that token-level accuracy can diverge from downstream generation quality.
SARA framework aligns MoE routing distributions to improve low-resource multilingual performance
Researchers introduce SARA (Semantically Anchored Routing Alignment), a framework that addresses cross-lingual routing divergence in sparse Mixture-of-Experts LLMs by aligning the internal routing distributions of low-resource language tokens to match those of high-resource semantic anchors via symmetric Jensen-Shannon divergence constraints. Unlike logit-level distillation, SARA operates directly on MoE routing layers to encourage mechanistic consistency in expert selection across languages. Experiments on Qwen3-30B-A3B and Phi-3.5-MoE-instruct across 5 low-resource languages show modest but consistent gains (up to +1.2%) on Global-MMLU over standard instruction tuning.
Global-batch Load Balancing for MoE LLM Training from Qwen
Qwen Research introduces a global-batch load balancing technique for Mixture-of-Experts (MoE) LLM training, claiming it is nearly a 'free lunch' improvement. The method addresses expert load imbalance across training batches, a known efficiency and quality bottleneck in MoE architectures. The approach targets the router and expert activation dynamics in transformer-based MoE layers.
MobileMoE: Scaling Mixture-of-Experts for Sub-Billion Parameter On-Device Deployment
MobileMoE introduces a family of on-device MoE language models with 0.3–0.9B active parameters and 1.3–5.3B total parameters, targeting mobile deployment under memory and compute constraints. The authors derive an on-device MoE scaling law identifying a sweet spot of moderate sparsity with fine-grained and shared experts, then train models through a four-stage recipe including quantization-aware training on open-source data. Across 14 benchmarks, MobileMoE matches or exceeds leading dense on-device LLMs with 2–4× fewer inference FLOPs, and delivers 1.8–3.8× faster prefill and 2.2–3.4× faster decode than dense baselines on commodity smartphones at comparable INT4 memory.
SMoA: Spectrum Modulation Adapter for Parameter-Efficient Fine-Tuning
SMoA is a new parameter-efficient fine-tuning method that addresses LoRA's trade-off between rank size and parameter budget. It partitions model layers into spectral blocks and applies Hadamard-modulated low-rank branches to each diagonal block, enabling broader coverage of pretrained spectral directions without proportionally increasing trainable parameters. Theoretical analysis and empirical results on multiple tasks show SMoA outperforms LoRA and competitive LoRA-style baselines in lower-budget settings.
MAESTRO: Markov-chain-based structured pruning framework for Mixture-of-Experts language models
Researchers introduce MAESTRO, a structured pruning method for sparse Mixture-of-Experts (MoE) language models that models expert activation trajectories as Ergodic Markov chains to capture cross-layer routing dependencies. Unlike existing pruning methods adapted from dense transformers, MAESTRO uses stationary distributions of these chains as globally-aware importance heuristics. Under a 50% compression regime evaluated across five domains including safety and ethics, MAESTRO outperforms state-of-the-art baselines by up to 10.61% in average performance retention with lower cross-task variance.