paper
Expert-Aware Causal Tracing of Factual Recall in Sparse MoE Language Models
paperactiveprovisional
expert-aware-causal-tracing-of-factual-recall-in-sparse-moe-language-models-255ffff8·1 events·first seen 13d agoAliases: Expert-Aware Causal Tracing of Factual Recall in Sparse MoE Language Models
Co-occurring entities
More like this (12)
Attention Amnesia in Hybrid LLMs: When CoT Fine-Tuning Breaks Long-Range Recall, and How to Fix ItTying the Loop -- Tied Expert Layers in Mixture-of-Experts Language ModelsRecalling Too Well: Sycophancy Evaluation and Mitigation in Memory-Augmented ModelsReasoning Language ModelsFrom Observation to Intervention: A Causal Audit of Expert Importance in Mixture-of-Experts ModelsLanguage Modeling LossProvenanceGuard: Source-Aware Factuality Verification for MCP-Based LLM Agentsencoder-only language modelsSparse Subspace-to-Expert Sharing for Task-Agnostic Continual LearningDecomposing Factual Sycophancy in Language Models: How Size and Instruction Tuning Shape RobustnessCausally Evaluating the Learnability of Formal Language TasksLatentMoE
Recent events (1)
Expert-aware causal tracing of factual recall in sparse MoE language models
A new arXiv preprint extends causal tracing methodology to sparse mixture-of-experts (MoE) language models, asking which routed experts mediate factual recall rather than just which layers or feed-forward modules. Using CounterFact facts, the authors apply noise-corruption and clean-patch interventions to Qwen3-30B-A3B-Base and Mixtral-8x7B-v0.1, finding that expert-level localization is possible in the former (a single expert at layer 44) but requires multi-expert coalition recovery in the latter. The results indicate that factual localization in MoE models is model- and protocol-dependent rather than universal.