Localized LoRA-MoE

Localized LoRA-MoE: Block-wise and cell-wise low-rank expert routing for multi-task PEFT

A new arXiv preprint introduces Localized LoRA-MoE, a parameter-efficient fine-tuning framework that combines spatial blocking with dynamic context-conditioned routing to address gradient interference in multi-task LoRA adapters. The authors propose two architectures: Block-Wise LoRA-MoE with centralized macro-routing and Cell-Wise LoRA-MoE with decentralized per-cell expert gating. Empirical results across SVD matrix simulations, tabular tasks, and spatial vision benchmarks show that decentralized cell-level gating matches a global coordinator while providing fault isolation against sensor degradation and task-switching instability.