paper

Causally Evaluating the Learnability of Formal Language Tasks

paperactiveprovisionalcausally-evaluating-the-learnability-of-formal-language-tasks-79df3a3b·1 events·first seen 8d ago

Aliases: Causally Evaluating the Learnability of Formal Language Tasks

Co-occurring entities

Kullback-Leibler divergence binning semiring

More like this (12)

Reasoning Language Models Scaling LLM Reasoning from Minimal Labels: A Semi-Supervised Framework with a Lightweight Verifier Meta-Learning for Compositionality Expert-Aware Causal Tracing of Factual Recall in Sparse MoE Language Models Operator Learning Reinforcement Learning Elicits Contextual Learning of Unseen Language Translation Operadic consistency: a label-free signal for compositional reasoning failures in LLMs From Correctness to Utility: Gain-Based Prefix Evaluation for LLM Reasoning Reinforcement Learning for Language Models Revisiting the Systematicity in Negation in the Era of In-Context Learning Recursive Language Models (RLMs)Arithmetic Pedagogy for Language Models

Recent events (1)

5arXiv · cs.CL·8d ago·source ↗

Causal evaluation framework for learnability of formal language tasks in LMs

A new arXiv preprint proposes a causal framework for evaluating how much task-specific data language models need to learn a given task. The authors use formal languages generated by probabilistic finite automata as a controlled testbed, introducing the 'binning semiring' algebraic object to control property frequency in training corpora. Experiments show that standard correlational evaluation practices produce incorrect learnability conclusions due to confounders, with implications for how natural-language task learning is studied.

Evaluation and Benchmarking Kullback-Leibler divergence Causally Evaluating the Learnability of Formal Language Tasks binning semiring