3arXiv cs.CL (Computation and Language)·
FSE: Fast-Slow Experts architecture for continual learning in Named Entity Recognition
A new arXiv preprint introduces FSE, a span-based NER model using a dual-expert architecture to address catastrophic forgetting in continual learning settings. A shared 'fast expert' filters unlikely spans at the token level, while task-specific 'slow experts' handle span classification, reducing per-task learning burden. The method also introduces a length-decay negative sampling strategy to handle span imbalance. Experiments on OntoNotes and FewNERD synthetic datasets show state-of-the-art results in continual NER scenarios.
Related guides (1)
Related events (8)
SETA: Sparse Subspace-to-Expert Sharing for Continual Learning in LLMs
Researchers introduce SETA (Mixture of Sparse Experts for Task Agnostic Continual Learning), a framework addressing catastrophic forgetting in LLMs via adaptive sparse subspace decomposition into task-specific and shared expert modules. The approach uses adaptive elastic anchoring and routing-aware regularization to protect shared knowledge at both weight and routing levels. Experiments on LLaMA-2 7B and Qwen3-4B show competitive or superior performance versus continual learning baselines, with strong retention of early-task knowledge.
Self-Generated Replay Nearly Eliminates Catastrophic Forgetting in Language Models
This paper investigates catastrophic forgetting in language models during continual learning, finding that models can use self-generated samples from their own training distribution as effective replay data, nearly eliminating forgetting without requiring stored exemplars. The authors identify two key conditions where forgetting persists: when models are pretrained near capacity saturation (leaving no room for new knowledge), and when low learning rates are used to reduce forgetting at the cost of requiring far more training steps. Self-generated replay breaks this learning-rate/forgetting tradeoff, enabling fast high-learning-rate finetuning without degradation on prior tasks.
CHERRY: Compressed Hierarchical Experts with Recurrent Representational Yield — three compute-efficient LM training techniques
A preprint from arXiv introduces CHERRY, a suite of three complementary techniques for compute-efficient language model training: Selective Ground Truth Token Training (SGT) that concentrates supervision on ~15% of semantically loaded tokens while recovering ~67% of full-sequence loss reduction; depth compression that shrinks a 48-layer 1B-parameter model to 6 layers (227M) via layer averaging and recurrent unrolling, matching a 566M dense model's loss; and a Mixture of Efficient Experts (MoEE) assembly that outperforms individual compressed models at comparable active parameters. The techniques are validated on CHERRY-1.8B, a Korean-language foundation model trained entirely from scratch using these methods. Authors are transparent about scope limitations: one model family, Korean data, and loss-based metrics only.
LoRA mixture-of-experts variants for continual learning in motion-language agents
A new arXiv preprint investigates continual learning for bidirectional motion-language agents that must both understand and generate human motion without catastrophic forgetting. The authors propose LoRA-based mixture-of-experts architectures with an autoencoder-based router for task-specific expert selection at inference time, requiring no task labels. Evaluated on a five-task benchmark derived from HumanML3D, the approach achieves near-zero forgetting across motion-to-text and text-to-motion directions. A key finding is that hard expert selection outperforms soft blending, and that token-level accuracy can diverge from downstream generation quality.
Temporal fusion strategies for NER in historical texts favor late fusion mechanisms
A new arXiv preprint systematically evaluates how temporal metadata can be embedded into transformer-based NER models for historical texts, comparing absolute vs. relative temporal representations and early vs. late fusion mechanisms including cross-attention, adapters, and concatenation. Experiments on French and German historical datasets show that late fusion strategies yield more robust and temporally generalizable performance, especially on early and noisy text periods. The work addresses a narrow but underexplored challenge of diachronic NLP where entity surface forms drift across time.
Expert-aware causal tracing of factual recall in sparse MoE language models
A new arXiv preprint extends causal tracing methodology to sparse mixture-of-experts (MoE) language models, asking which routed experts mediate factual recall rather than just which layers or feed-forward modules. Using CounterFact facts, the authors apply noise-corruption and clean-patch interventions to Qwen3-30B-A3B-Base and Mixtral-8x7B-v0.1, finding that expert-level localization is possible in the former (a single expert at layer 44) but requires multi-expert coalition recovery in the latter. The results indicate that factual localization in MoE models is model- and protocol-dependent rather than universal.
Complete-muE: Optimal Hyperparameter Transfer and Scaling for MoE Models
Complete-muE is a framework for transferring hyperparameters across dense FFN and Mixture-of-Experts (MoE) transformer architectures, addressing limitations of existing tools like μP and SDE that cannot handle simultaneous architecture and token-per-expert changes. It uses a two-bridge system: Bridge I maps dense FFN to Dense MoE via active-width μP with normalized router scale, and Bridge II maps Dense MoE to sparse MoE via activated-expert scaling with a first-order SDE correction. The practical outcome is a 'tune dense once, transfer to all' recipe that enables near-optimal hyperparameter reuse across MoE configurations without costly re-tuning. Experiments on language model and diffusion model pretraining confirm stable hyperparameter optima across architectures and parameter counts.
Continual learning approach for disfluency-aware ASR with explicit disfluency tokens
A new arXiv preprint addresses the challenge of transcribing disfluent speech (hesitations, repetitions, fillers) in ASR systems, which typically omit such markers causing information loss. The authors introduce explicit disfluency tokens into a pretrained ASR model and apply continual learning to adapt across datasets with varying disfluency distributions while mitigating catastrophic forgetting. The work identifies a trade-off between disfluency marker learning and general ASR performance, and finds a consistent cross-attention head mechanism shared across continual learning methods.