MoE²-LoRA

MoE²-LoRA: MoE-style low-rank adaptation for Mixture-of-Experts language models

Researchers introduce MoE²-LoRA, a parameter-efficient fine-tuning method specifically designed for Mixture-of-Experts (MoE) language models. The approach uses a dual-channel Routing-Conditioned Projection module that reuses base router activations to guide LoRA adapter routing, and introduces a single global LoRA expert pool shared across all layers. Evaluated across multiple MoE backbones at varying scales, MoE²-LoRA claims state-of-the-art downstream accuracy while better preserving general capabilities compared to existing PEFT methods.