Entity · paper

Spend Experts Where You Are Unsure: Confidence-Adaptive Routing for Mixture-of-Experts LoRA

paperactiveprovisionalspend-experts-where-you-are-unsure-confidence-adaptive-routing-for-mixture-of-experts-lora-86cbd046·1 events·first seen 2d ago

Aliases: Spend Experts Where You Are Unsure: Confidence-Adaptive Routing for Mixture-of-Experts LoRA

Co-occurring entities

Qwen2.5-7B CARE (Confidence-Adaptive Routing of Experts)Llama-3.1-8B

More like this (12)

CARE (Confidence-Adaptive Routing of Experts)Redesign Mixture-of-Experts Routers with Manifold Power Iteration Mixture of Efficient Experts Sparse Mixture-of-Experts Multi-Expert Routing for Multi-Domain Low-Resource OCR: A Manchu Case Study Mixture of Experts Toward Calibrated Mixture-of-Experts Under Distribution Shift Tying the Loop -- Tied Expert Layers in Mixture-of-Experts Language Models Localized LoRA-MoE From Self-Supervised Speech Models to Mixture-of-Experts for Robust Anti-Spoofing Cross-Layer Sparse Attention with Shared Routing Multi-LoRA serving

Recent events (1)

5arXiv · cs.LG·2d ago·source ↗

CARE: Confidence-Adaptive Routing for Mixture-of-Experts LoRA adjusts expert count per token

Researchers introduce CARE (Confidence-Adaptive Routing of Experts), a drop-in routing rule for MoE-LoRA that dynamically adjusts the number of active experts per token based on router output uncertainty rather than using a fixed top-k. The method uses nucleus-style cumulative mass thresholding with a budget thermostat to hit any target average expert count. Evaluated on LLaMA-3.1-8B and Qwen2.5-7B across commonsense, math, code, and knowledge benchmarks, CARE matches or outperforms fixed top-k baselines at equal compute while also improving out-of-distribution detection.

Open Weights Progress Inference Economics Qwen2.5-7B CARE (Confidence-Adaptive Routing of Experts)Spend Experts Where You Are Unsure: Confidence-Adaptive Routing for Mixture-of-Experts LoRA +1 more