5arXiv cs.CL (Computation and Language)·
Multilayer taxonomy of 14 capability domains and 91 subskills for LLM evaluation
Researchers introduce a structured taxonomy organizing LLM capabilities into 14 domains and 91 subskills across Primitive, Constructed, and Integrative layers, grounded in human cognitive science rather than model architecture. To validate the framework, they screened 31,505 papers from ACL, AAAI, ICML, and NeurIPS (2023–2025) and mapped 15,934 LLM-focused papers through multi-model annotation. The analysis reveals heavy concentration of research attention on Language-Semantic Competence and Reasoning, with six domains receiving fewer than 2% of papers, highlighting systematic coverage gaps. The taxonomy is intended to support cross-study comparison, coverage audits, and hypothesis generation for training and transfer.
Related guides (1)
Related events (8)
Comprehensive survey of metacognition in LLMs: foundations, methods, and open questions
A preprint from Yale NLP presents the first comprehensive survey of metacognition in large language models, covering how LLMs can exhibit or be endowed with self-monitoring, self-evaluation, and self-regulation abilities. The paper taxonomizes the field, reviews benchmarks and evaluation methods, and surveys techniques to elicit and improve metacognitive capabilities. The authors argue metacognition is increasingly recognized as central to capable, transparent, and reliable AI systems, and identify open challenges and future directions.
Survey chapter on LLM mechanisms, emergent capabilities, and cognition debates
A new arXiv preprint surveys current understanding of large language models, covering the Transformer architecture, emergent capabilities resembling human cognition (symbolic reasoning, theory of mind, deception), and explainability approaches from neuron activation analysis to circuit tracing. The chapter also engages the debate over whether LLMs genuinely understand or merely pattern-match, arguing against reductive anti-anthropomorphism while acknowledging human-LLM differences. It is framed as a book chapter synthesizing recent empirical findings and theoretical positions.
Paper challenges LLM expert-level claims by measuring variance and error magnitude in code-based data analysis tasks
A new arXiv paper argues that standard LLM benchmarks overstate model capabilities by focusing on average performance on training-data-adjacent tasks while ignoring response variance and error magnitude. The authors introduce a novel benchmark requiring frontier LLMs to write code for data analysis tasks, comparing results against human expert submissions. Human experts outperformed the frontier LLM on average across multiple metrics and showed lower performance variability. The findings challenge the prevailing narrative that LLMs perform at human-expert level on knowledge economy tasks.
LLawCo framework teaches embodied multi-agent LLMs to derive and follow cooperation laws
Researchers from MERL propose LLawCo (Learning Laws of Cooperation), a framework that enables embodied LLM-based agents to autonomously align with partners and task objectives in decentralized, partially observable environments. Agents reflect on past failures to extract misaligned behavioral patterns and derive high-level behavioral laws (e.g., 'Talk when necessary', 'Wait for partner'), which are incorporated into reasoning via supervised fine-tuning. The authors also introduce PARTNR-Dialog, a new large-scale multi-agent communicative planning benchmark, and report average success rate improvements of 4.5% on PARTNR-Dialog and 6.8% on TDW-MAT over state-of-the-art open-source communicative agent frameworks across four backbone LLMs.
Survey and recommendations for LLM-as-a-Judge evaluation in multilingual and low-resource language settings
A new arXiv paper surveys 650 ACL Anthology papers that use LLM-as-a-Judge evaluation, finding only 33 address multilingual or low-resource language settings. Analysis of those 33 papers reveals inconsistent outcomes, overtrust in LLM judgments, and over-reliance on single judge models. The authors provide recommendations for improving evaluation practice in these underserved settings.
LLMs can extract and reuse experiential abstractions to improve reasoning, study finds
Researchers investigate whether LLMs can distill solution traces into reusable natural-language abstractions (strategies, reminders) and apply them to future problems, analogous to human experiential learning. Using the MATH training set, abstractions are extracted by either a stronger teacher model or the LLMs themselves into a retrievable library, then applied via inference-time retrieval or RL with abstraction-augmented prompts. Both modes improve performance on mathematical and logical reasoning benchmarks, with self-extracted abstractions matching teacher-extracted ones. The framework generalizes across datasets and models, suggesting a promising direction for continual self-improvement in LLMs.
Mechanism-oriented taxonomy of indirect linguistic encoding improves LLM-based coded language detection
Researchers propose a comprehensive taxonomy of indirect linguistic expressions (ILE) — covering algospeak, euphemisms, and adversarial obfuscation — organized by underlying encoding mechanisms rather than communicative intent. The taxonomy is evaluated by injecting it into LLM prompts and benchmarked against four existing taxonomies and a no-taxonomy baseline on 2,000 manually annotated TikTok and Bluesky posts. The proposed taxonomy achieves 4.7% accuracy and 5.4% F1 improvements over the best competing approach across three LLMs, suggesting structured linguistic scaffolding meaningfully aids content moderation tasks.
Study finds LLM-generated research ideas cluster around synthesis and bridging, diverging from human distribution
A new arXiv paper introduces a large-scale evaluation framework for comparing LLM-generated research ideas against human-authored ones, using reverse-engineered prior-work sets as prompts. The authors develop a two-axis taxonomy of research taste (opportunity pattern and research paradigm) and find a consistent distributional gap: LLMs over-index on bridge-like opportunities and synthesis methods, while human researchers spread more broadly across framing and contribution types. The result suggests current LLMs produce reasonable but systematically narrower and shifted ideation relative to human researchers.