Entity · paper

Causally Evaluating the Learnability of Formal Language Tasks

paperactivecausally-evaluating-the-learnability-of-formal-language-tasks-79df3a3b·1 events·first seen Jun 9, 2026

Aliases: Causally Evaluating the Learnability of Formal Language Tasks

Co-occurring entities

Kullback-Leibler divergence binning semiring

More like this (12)

Language-Critique Imitation Learning from Suboptimal Demonstrations Can LLMs Judge Better Than They Generate? Evaluating Task Asymmetry, Mechanistic Interpretability and Transferability for In-Context QA On the Limits of Prompt-Conditioned Language Models as General-Purpose Learners Knowledge-Less Language Models Reasoning Language Models Invariant Learning Dynamics of Transformers in Inductive Reasoning Tasks Invariant Learning Dynamics of Transformers in Inductive Reasoning Tasks Evaluation Awareness Is Not One Capability: Evidence from Open Language Models LMs as Task-Specific Knowledge Bases: An Interpretability Analysis Knowledge Knows, Verbalization Tells: Disentangling Latent Directions for Mathematical Solvability in LLMs Exposure is Optional: Learning Unlike Coordination in Language Models Scaling LLM Reasoning from Minimal Labels: A Semi-Supervised Framework with a Lightweight Verifier

Recent events (1)

5arXiv · cs.CL·Jun 9, 2026·source ↗

Causal evaluation framework for learnability of formal language tasks in LMs

A new arXiv preprint proposes a causal framework for evaluating how much task-specific data language models need to learn a given task. The authors use formal languages generated by probabilistic finite automata as a controlled testbed, introducing the 'binning semiring' algebraic object to control property frequency in training corpora. Experiments show that standard correlational evaluation practices produce incorrect learnability conclusions due to confounders, with implications for how natural-language task learning is studied.

Evaluation and Benchmarking Kullback-Leibler divergence Causally Evaluating the Learnability of Formal Language Tasks binning semiring