Entity · paper

Invariant Learning Dynamics of Transformers in Inductive Reasoning Tasks

paperactiveprovisionalinvariant-learning-dynamics-of-transformers-in-inductive-reasoning-tasks-db58505b·1 events·first seen 37h ago

Aliases: Invariant Learning Dynamics of Transformers in Inductive Reasoning Tasks

Co-occurring entities

More like this (12)

Invariant Learning Dynamics of Transformers in Inductive Reasoning Tasks Bridging the Gap Between Latent and Explicit Reasoning with Looped Transformers Fixed-Point Reasoners: Stable and Adaptive Deep Looped Transformers Associative Recurrent Memory Transformer Dynamic Short Convolutions Improve Transformers permutation-invariant transformers TRL (Transformer Reinforcement Learning)Operator Learning Learning to Reason by Analogy via Retrieval-Augmented Reinforcement Fine-Tuning Training-Free Looped Transformers The Key to Going Linear: Analysis-Driven Transformer Linearization Transformer Language Models

Recent events (1)

6arXiv · cs.AI·37h ago·source ↗

Theoretical framework reveals low-dimensional invariant manifolds governing Transformer inductive reasoning dynamics

A new arXiv preprint presents a theoretical framework explaining how Transformers develop inductive reasoning abilities by proving that training dynamics of attention models are confined to a low-dimensional invariant manifold. The framework unifies several synthetic inductive tasks (in-context n-grams, multi-hop reasoning) and characterizes how data statistics govern competition between in-context and in-weights learning. The authors show that random initializations determine which circuit 'wins' when multiple solutions exist, and that the manifold's coordinate frame can automatically detect learned circuits in trained models. The work advances mechanistic interpretability by casting circuit formation as a tractable low-dimensional dynamical phenomenon.

Evaluation and Benchmarking AI Safety Research Invariant Learning Dynamics of Transformers in Inductive Reasoning Tasks Invariant Learning Dynamics of Transformers in Inductive Reasoning Tasks