4arXiv cs.CL (Computation and Language)·
Sample complexity bounds for learning C-RASP constructions with Transformers
A new arXiv preprint bridges the gap between Transformer expressivity theory and learnability by proposing preliminary sample complexity bounds for learning C-RASP constructions. Prior theoretical work has largely characterized which tasks fall within the Transformer hypothesis class via handcrafted weights or complexity arguments, but has not addressed how efficiently these solutions can be learned. The authors draw on recent loss landscape analysis to derive their bounds, advancing the theoretical understanding of LLM training dynamics.
Related guides (1)
Related events (8)
Theoretical framework reveals low-dimensional invariant manifolds governing Transformer inductive reasoning dynamics
A new arXiv preprint presents a theoretical framework explaining how Transformers develop inductive reasoning abilities by proving that training dynamics of attention models are confined to a low-dimensional invariant manifold. The framework unifies several synthetic inductive tasks (in-context n-grams, multi-hop reasoning) and characterizes how data statistics govern competition between in-context and in-weights learning. The authors show that random initializations determine which circuit 'wins' when multiple solutions exist, and that the manifold's coordinate frame can automatically detect learned circuits in trained models. The work advances mechanistic interpretability by casting circuit formation as a tractable low-dimensional dynamical phenomenon.
Conditional Scale Entropy: A Wavelet-Derived Tool for Mechanistic Interpretability of Metaphor Processing in Transformers
This paper introduces Conditional Scale Entropy (CSE), a wavelet-derived measure of how transformer computation engages across frequency scales at each layer, and applies it to study metaphor processing in decoder-only language models. The authors prove CSE is invariant to update magnitude, isolating structural computation patterns from intensity. Across architectures ranging from GPT-2 (124M) to LLaMA-2 7B and GPT-oss 20B, metaphorical tokens consistently produce higher spectral breadth than literal tokens in early-to-mid layers, with the effect surviving permutation correction and specificity controls. The work establishes multi-scale coordination as a consistent mechanistic signature of metaphorical language processing and positions CSE as a general interpretability tool for cross-depth structure in transformers.
Survey chapter on LLM mechanisms, emergent capabilities, and cognition debates
A new arXiv preprint surveys current understanding of large language models, covering the Transformer architecture, emergent capabilities resembling human cognition (symbolic reasoning, theory of mind, deception), and explainability approaches from neuron activation analysis to circuit tracing. The chapter also engages the debate over whether LLMs genuinely understand or merely pattern-match, arguing against reductive anti-anthropomorphism while acknowledging human-LLM differences. It is framed as a book chapter synthesizing recent empirical findings and theoretical positions.
Dynamic short convolutions yield 1.33–1.60× compute advantage over standard Transformers
A new arXiv preprint introduces dynamic short convolutions as an architectural primitive for Transformers, using input-dependent filters to combine locality bias with increased expressivity. Experiments across 150M–2B parameter language models show consistent perplexity improvements over standard Transformers and static convolution variants, with scaling-law fits indicating a 1.33× compute advantage when applied to key/query/value vectors and 1.60× when added after every linear layer. The technique also improves linear RNNs (Mamba-2, Gated DeltaNet) and mixture-of-experts architectures, with custom Triton kernels making training practical.
Multi-stage explainability framework translates transformer speech models into clinical cognitive impairment narratives
A new arXiv preprint proposes a framework for making transformer-based speech cognitive impairment detection clinically interpretable by combining SHAP token attribution, linguistic feature analysis, and a four-stage LLM reasoning pipeline using LLaMA-3.1-70B-Instruct. The system is built on the SpeechCARE-Adaptive Gating Network multimodal model (F1=72.11% on NIA PREPARE) and maps outputs to four cognitive-linguistic dimensions. Physician evaluation on 70 samples showed strong alignment with clinical profiles and a System Usability Scale score of 82/100, suggesting practical clinical workflow integration potential.
Theoretical analysis reframes Transformer architecture design as rank-preservation tradeoffs across depth
A new arXiv preprint investigates how components of the Transformer feedforward block — skip connections, normalization placement, and two-matrix expansion/contraction structure — determine how much rank survives across depth at initialization. The authors reinterpret skip connections and normalization not merely as magnitude controllers but as mechanisms for preserving gradient rank, unifying the Pre-Norm vs. Post-Norm literature under a single rank-collapse framework. They derive that width expansion between the two MLP matrices follows a Marchenko-Pastur law to maintain full branch Jacobian rank, and show that the initialization rank of the input-output Jacobian predicts trainability on CIFAR-10. The work recasts deep network architecture design as navigating a three-way tradeoff among rank collapse, ensemble-like behavior, and parameter count.
LEAF-X: Entropy-guided explainability framework for transformer-based ASR models
Researchers introduce LEAF-X (Listening with Entropy-guided Attention for Faithful explainability), a model-intrinsic XAI framework for transformer-based automatic speech recognition systems like Whisper. The method combines entropy-guided attention weighting, multi-layer attention rollout, and optional causal ablations to produce sparse token-to-frame attributions. Evaluations show 32% improved faithfulness and 35-39% stronger locality/sparsity compared to perturbation-based explainers and raw attention maps, enabling more auditable ASR.
Associative Recurrent Memory Transformer enables long-context LLMs with constant memory scaling
Researchers propose using the Associative Recurrent Memory Transformer (ARMT) as a practical method for extending LLM context length beyond original limits while achieving constant memory scaling instead of linear. The training recipe combines continued pre-training, synthetic long-context data, curriculum learning, and selective layer integration of associative memory. Experiments show ARMT-augmented models generalize to out-of-distribution context lengths and require 30% fewer FLOPs while preserving in-window baseline performance.