4arXiv cs.CL (Computation and Language)·
Localized LoRA-MoE: Block-wise and cell-wise low-rank expert routing for multi-task PEFT
A new arXiv preprint introduces Localized LoRA-MoE, a parameter-efficient fine-tuning framework that combines spatial blocking with dynamic context-conditioned routing to address gradient interference in multi-task LoRA adapters. The authors propose two architectures: Block-Wise LoRA-MoE with centralized macro-routing and Cell-Wise LoRA-MoE with decentralized per-cell expert gating. Empirical results across SVD matrix simulations, tabular tasks, and spatial vision benchmarks show that decentralized cell-level gating matches a global coordinator while providing fault isolation against sensor degradation and task-switching instability.
Related guides (3)
Related events (8)
SMoA: Spectrum Modulation Adapter for Parameter-Efficient Fine-Tuning
SMoA is a new parameter-efficient fine-tuning method that addresses LoRA's trade-off between rank size and parameter budget. It partitions model layers into spectral blocks and applies Hadamard-modulated low-rank branches to each diagonal block, enabling broader coverage of pretrained spectral directions without proportionally increasing trainable parameters. Theoretical analysis and empirical results on multiple tasks show SMoA outperforms LoRA and competitive LoRA-style baselines in lower-budget settings.
SARA framework aligns MoE routing distributions to improve low-resource multilingual performance
Researchers introduce SARA (Semantically Anchored Routing Alignment), a framework that addresses cross-lingual routing divergence in sparse Mixture-of-Experts LLMs by aligning the internal routing distributions of low-resource language tokens to match those of high-resource semantic anchors via symmetric Jensen-Shannon divergence constraints. Unlike logit-level distillation, SARA operates directly on MoE routing layers to encourage mechanistic consistency in expert selection across languages. Experiments on Qwen3-30B-A3B and Phi-3.5-MoE-instruct across 5 low-resource languages show modest but consistent gains (up to +1.2%) on Global-MMLU over standard instruction tuning.
Using LoRA for Efficient Stable Diffusion Fine-Tuning
This Hugging Face blog post explains how Low-Rank Adaptation (LoRA) can be applied to fine-tune Stable Diffusion models efficiently. LoRA reduces the number of trainable parameters by decomposing weight updates into low-rank matrices, enabling fine-tuning on consumer hardware with significantly less memory. The post covers practical implementation details using the diffusers library.
Expert Tying reduces MoE LLM memory footprint by ~2x with minimal quality loss
Researchers introduce Expert Tying, an architectural modification for Mixture-of-Experts LLMs that shares expert parameters across consecutive transformer layers while keeping routing and attention layer-independent. Evaluated on OLMoE, Qwen3, and DeepSeek-style MoE architectures, the method achieves nearly 2x memory reduction with negligible perplexity or downstream quality degradation. The approach exploits parameter redundancy in MoE pathways to improve the compute-to-memory trade-off for training and inference.
Global-batch Load Balancing for MoE LLM Training from Qwen
Qwen Research introduces a global-batch load balancing technique for Mixture-of-Experts (MoE) LLM training, claiming it is nearly a 'free lunch' improvement. The method addresses expert load imbalance across training batches, a known efficiency and quality bottleneck in MoE architectures. The approach targets the router and expert activation dynamics in transformer-based MoE layers.
Hugging Face blog compares fine-tuning techniques beyond LoRA
A Hugging Face blog post examines whether alternative parameter-efficient fine-tuning (PEFT) methods can outperform LoRA, currently the dominant fine-tuning technique. The post likely benchmarks or analyzes competing approaches such as DoRA, IA3, or other PEFT variants against LoRA baselines. This is relevant for practitioners choosing fine-tuning strategies for LLMs.
MobileMoE: Scaling Mixture-of-Experts for Sub-Billion Parameter On-Device Deployment
MobileMoE introduces a family of on-device MoE language models with 0.3–0.9B active parameters and 1.3–5.3B total parameters, targeting mobile deployment under memory and compute constraints. The authors derive an on-device MoE scaling law identifying a sweet spot of moderate sparsity with fine-grained and shared experts, then train models through a four-stage recipe including quantization-aware training on open-source data. Across 14 benchmarks, MobileMoE matches or exceeds leading dense on-device LLMs with 2–4× fewer inference FLOPs, and delivers 1.8–3.8× faster prefill and 2.2–3.4× faster decode than dense baselines on commodity smartphones at comparable INT4 memory.
LoRA mixture-of-experts variants for continual learning in motion-language agents
A new arXiv preprint investigates continual learning for bidirectional motion-language agents that must both understand and generate human motion without catastrophic forgetting. The authors propose LoRA-based mixture-of-experts architectures with an autoencoder-based router for task-specific expert selection at inference time, requiring no task labels. Evaluated on a five-task benchmark derived from HumanML3D, the approach achieves near-zero forgetting across motion-to-text and text-to-motion directions. A key finding is that hard expert selection outperforms soft blending, and that token-level accuracy can diverge from downstream generation quality.