5arXiv cs.CL (Computation and Language)·
MaLoRA and MaRA: Selective state-space adapters improve multi-hop reasoning over LoRA
A new arXiv preprint proposes two adapter families — MaLoRA (token-level dynamic scaling via Mamba recurrence) and MaRA (context-level segment retrieval via cross-segment state tracking) — as improvements over standard LoRA for language model reasoning. Evaluated on three frozen backbones (Qwen-2.5-7B, Llama-3.1-8B, Gemma-2-9B) and two multi-hop QA benchmarks (MuSiQue, 2WikiMultihopQA), the methods yield average gains of +6.8 F1 (+10.5% relative) over LoRA, with up to +18.2% relative improvement on the hardest configuration. Token-level gains also transfer to RULER QA-2 under length-stress conditions.
Related guides (2)
Related events (8)
AuRA: Distilling audio understanding into LLMs via LoRA adaptation
AuRA is a new method for integrating speech understanding into LLMs by distilling audio encoding capability directly into LoRA-adapted model weights, bypassing cascaded ASR-LLM pipelines. A lightweight audio embedding layer feeds speech to both an ASR encoder (teacher) and a LoRA-adapted LLM (student), with layer-wise distillation aligning hidden states. The approach claims to outperform cascaded systems, bridge-based adaptation baselines, and large-scale multimodal models on multiple speech-language benchmarks while enabling parallel end-to-end inference without large-scale multimodal training.
MoE²-LoRA: MoE-style low-rank adaptation for Mixture-of-Experts language models
Researchers introduce MoE²-LoRA, a parameter-efficient fine-tuning method specifically designed for Mixture-of-Experts (MoE) language models. The approach uses a dual-channel Routing-Conditioned Projection module that reuses base router activations to guide LoRA adapter routing, and introduces a single global LoRA expert pool shared across all layers. Evaluated across multiple MoE backbones at varying scales, MoE²-LoRA claims state-of-the-art downstream accuracy while better preserving general capabilities compared to existing PEFT methods.
Localized LoRA-MoE: Block-wise and cell-wise low-rank expert routing for multi-task PEFT
A new arXiv preprint introduces Localized LoRA-MoE, a parameter-efficient fine-tuning framework that combines spatial blocking with dynamic context-conditioned routing to address gradient interference in multi-task LoRA adapters. The authors propose two architectures: Block-Wise LoRA-MoE with centralized macro-routing and Cell-Wise LoRA-MoE with decentralized per-cell expert gating. Empirical results across SVD matrix simulations, tabular tasks, and spatial vision benchmarks show that decentralized cell-level gating matches a global coordinator while providing fault isolation against sensor degradation and task-switching instability.
SARA framework aligns MoE routing distributions to improve low-resource multilingual performance
Researchers introduce SARA (Semantically Anchored Routing Alignment), a framework that addresses cross-lingual routing divergence in sparse Mixture-of-Experts LLMs by aligning the internal routing distributions of low-resource language tokens to match those of high-resource semantic anchors via symmetric Jensen-Shannon divergence constraints. Unlike logit-level distillation, SARA operates directly on MoE routing layers to encourage mechanistic consistency in expert selection across languages. Experiments on Qwen3-30B-A3B and Phi-3.5-MoE-instruct across 5 low-resource languages show modest but consistent gains (up to +1.2%) on Global-MMLU over standard instruction tuning.
MADA-RL: Multi-agent debate with counterfactual critic advantage for parameter-efficient reasoning in compact models
MADA-RL is a post-training framework that specializes compact LLMs (≤4B parameters) into generator and critic roles, training them with a debate-aware RL signal using LoRA adapters on only a small fraction of parameters. The central contribution is a counterfactual critic advantage that conditions the critic's reward on the generator ensemble's per-instance accuracy, explicitly incentivizing critics to correct generator errors rather than reproduce correct answers. Applied to DeepSeek-R1-Distill-Qwen-1.5B, the method achieves +2.0 percentage points on mathematical reasoning benchmarks using 16× fewer trainable parameters than fully fine-tuned baselines, placing it on the accuracy-vs-trainable-parameter Pareto front. The approach does not surpass the strongest baselines (DeepScaleR, STILL-3) trained on larger datasets, and the paper analyzes this gap directly.
Code2LoRA: Hypernetwork generates repository-specific LoRA adapters for code models with zero token overhead
Code2LoRA is a hypernetwork framework that generates repository-specific LoRA adapters for code language models, eliminating the inference-time token overhead of RAG or long-context injection. It supports both static repository snapshots and evolving codebases via a GRU-backed adapter updated per code diff. The authors introduce RepoPeftBench, a new benchmark of 604 Python repositories with static and evolution tracks, on which Code2LoRA-Static matches per-repository LoRA fine-tuning upper bounds and Code2LoRA-Evo outperforms a shared LoRA by 5.2 percentage points.
κ-LoRA uses condition numbers to selectively update LoRA matrices, cutting fine-tuning cost
A new arXiv preprint introduces κ-LoRA, a method that uses matrix condition numbers to identify which LoRA weight matrices are most worth updating during fine-tuning. By restricting updates to the top 50% of matrices ranked by condition number, κ-LoRA halves the trainable parameter count, reduces fine-tuning time by 16.2% on average, and cuts memory cost by 4.5% while matching standard LoRA accuracy. The core finding — that matrices with larger condition numbers drive most adaptation gains — is presented as a novel theoretical contribution motivating the selective approach. The method targets resource-constrained settings including edge and on-device fine-tuning.
RASER: Recoverability-Aware Selective Escalation Router for Multi-Hop Question Answering
RASER introduces a family of lightweight routers that decide whether to escalate retrieval complexity for multi-hop QA without making additional LLM calls. Built on top of one-shot RAG using six derived features, RASER-2 and RASER-3 route queries to progressively more expensive retrieval strategies (PRUNE, IRCoT) only when needed. Across six LLMs and three benchmarks, the routers match SOTA F1 while consuming only 41-49% of the tokens required by always-escalating baselines.