4arXiv cs.CL (Computation and Language)·
Taxonomy paper proposes unified framework for curriculum learning in NLP
A new arXiv preprint proposes a fine-grained taxonomy for curriculum learning (CL) in NLP, separating difficulty evaluation from training scheduling to enable systematic comparison of CL strategies. The authors identify a 'systematic incomparability problem' in prior work, where different notions of difficulty and scheduling are conflated under the same CL label. The paper provides the first formal treatment of CL schedulers in terms of expected training contribution, introducing retention regimes and monotonicity properties. The work is primarily a conceptual and analytical contribution aimed at improving rigor in CL research rather than reporting new empirical results.
Related guides (1)
Related events (8)
AgentCL: A Rigorous Evaluation Framework for Continual Learning in Language Agents
AgentCL is a new benchmark and evaluation framework designed to rigorously assess continual learning in language agents, addressing gaps in existing benchmarks that focus on retrieval over long-context documents or use naive task streams with limited cross-task analysis. The framework constructs compositional task streams where earlier sub-solutions, evidence, or workflows are intentionally reusable in later tasks, contrasting them with naive streams to measure transfer gains. The authors also introduce MemProbe, a probing method that stores interactions, insights, and skills while filtering unreliable experiences during consolidation. Empirical results across coding, deep research, and language understanding tasks show that controlled streams better distinguish memory design quality, and that naive streams can mask memory-induced degradation.
Multilayer taxonomy of 14 capability domains and 91 subskills for LLM evaluation
Researchers introduce a structured taxonomy organizing LLM capabilities into 14 domains and 91 subskills across Primitive, Constructed, and Integrative layers, grounded in human cognitive science rather than model architecture. To validate the framework, they screened 31,505 papers from ACL, AAAI, ICML, and NeurIPS (2023–2025) and mapped 15,934 LLM-focused papers through multi-model annotation. The analysis reveals heavy concentration of research attention on Language-Semantic Competence and Reasoning, with six domains receiving fewer than 2% of papers, highlighting systematic coverage gaps. The taxonomy is intended to support cross-study comparison, coverage audits, and hypothesis generation for training and transfer.
Causal evaluation framework for learnability of formal language tasks in LMs
A new arXiv preprint proposes a causal framework for evaluating how much task-specific data language models need to learn a given task. The authors use formal languages generated by probabilistic finite automata as a controlled testbed, introducing the 'binning semiring' algebraic object to control property frequency in training corpora. Experiments show that standard correlational evaluation practices produce incorrect learnability conclusions due to confounders, with implications for how natural-language task learning is studied.
Fodor and Pylyshyn's systematicity challenge to neural networks remains unmet, paper argues
A new arXiv preprint argues that recent claims that neural networks have met Fodor and Pylyshyn's systematicity challenge are premature. The authors specifically target Lake and Baroni's meta-learning for compositionality (MLC) protocol, showing it struggles with out-of-distribution rules and behaves unsystematically on many within-distribution problems. The paper concludes that the classical cognitive science challenge — that neural networks cannot explain systematic biconditional dependencies in language and thought — remains unresolved.
Structure-Aware Code Change Labeling with LLMs via Two-Stage Taxonomy Pipeline
This paper presents a systematic study of using LLMs for taxonomy-based labeling of code diff hunks, going beyond summarization to assign structured labels capturing semantic attributes like renames, moves, and logic modifications. The authors introduce a two-stage pipeline combining diff-hunk labeling with structural refinement, using few-shot prompting to remain language-agnostic. Evaluated across four LLMs on a curated benchmark of natural and synthetic patches, the best configuration achieves 84% recall and 81% precision. Results suggest LLM-based structured labeling can complement static analysis tools in code review workflows.
Continual-learning evaluation on Terminal-Bench 2.0 tests whether agent optimizer gains compound across tasks
A new arXiv paper introduces a two-phase continual-learning evaluation framework built on Terminal-Bench 2.0 to test whether agent-optimization gains persist and compound when new tasks arrive over time. Three agent-harness optimization methods — GEPA, Meta Harness, and RELAI-VCL — are compared under identical budgets; all improve in static single-phase settings but diverge sharply under continual optimization. RELAI-VCL is the only method that both transfers positively to unseen tasks and continues improving, reaching a 76.4% lifelong average pass rate versus 58.7% for the unoptimized baseline. The key finding is that compounding gains require regression control built into the optimization loop to prevent shortcut solutions.
BabyCL: Continual multimodal learning from egocentric child video in a single chronological pass
Researchers introduce BabyCL, a continual learning framework that processes the SAYCam egocentric child video dataset in a single chronological pass rather than shuffled multi-epoch training, more closely mimicking how children actually encounter their environment. The system combines streaming visual representation learning with image-text contrastive objectives, a multi-stage temporal segmentation, and a dual replay buffer managing visual and multimodal histories. BabyCL outperforms streaming baselines on the SAYCam Labeled-S 4AFC benchmark under matched compute budgets, substantially closing the gap to offline training upper bounds. The work advances understanding of whether neural networks can acquire word-referent mappings under biologically plausible training conditions.
Language-Critique Imitation Learning uses natural language feedback to learn from suboptimal demonstrations
A new arXiv preprint proposes a language-critique framework for imitation learning from suboptimal demonstrations, replacing scalar supervision signals (confidence scores, discriminator outputs) with structured natural language labels that describe task progress, failure modes, and corrective actions. The method instantiates a language-critique loss for both behavior cloning (LC-BC) and diffusion policies (LC-DP), with a theoretical bound on the expert performance gap. Empirical evaluation across navigation, manipulation, and gameplay tasks shows consistent outperformance over imitation learning and offline RL baselines.