6arXiv cs.CL (Computation and Language)·
Formal theory of when tool use increases expressive power of finite-precision recurrent models
A new arXiv paper provides an exact, architecture-level characterization of when external tool access increases the computational expressivity of finite-precision recurrent sequence models, including SSMs. The key dichotomy: finite-state tools add essentially nothing (absorbable at logarithmic bit cost), while a single infinite-state tool (a read/write tape) makes the system Turing complete with only O(log|Q| + log|Γ|) bits of controller state. The paper further shows this construction is realized by a natural one-layer finite-precision selective affine SSM, with selectivity identified as essential.
Related guides (2)
Related events (8)
Exact instrument for state-mode usage in Mamba SSMs reveals input-driven state migration
A new arXiv paper introduces an exact analytical instrument for measuring how selective state-space models (Mamba-1, Mamba-2, Falcon-Mamba) allocate their state modes across inputs, using a per-(layer, channel, window) Gram tensor that predicts pruning error to near-machine-precision. The key empirical finding is that trained models dynamically reallocate which modes carry signal depending on the input context, and that this migration is driven primarily by the input-dependent write map B_t rather than the timestep parameter typically associated with selectivity. Input-scheduled mode pruning based on this instrument outperforms static and Hankel-based rankings at all scales tested (130M–7B), matching unpruned model quality at half the state budget in offline evaluation.
Paper diagnoses RL collapse in multi-step tool-use training and proposes supervisory signal fixes
A new arXiv preprint identifies a failure mode in reinforcement learning for LLM tool use: catastrophic collapse caused by probability spikes in control tokens that disrupt structured execution while leaving underlying tool-use capability intact. The authors systematically evaluate supervisory signals—including off-policy supervision, hint-based guidance, and erroneous example supervision—under synchronous and interleaved training schemes. Interleaving SFT with RL improves stability but degrades performance under out-of-distribution format and content evaluation. Code is released as Tool-RL-Box.
Phantom specialization in circuit discovery: structural differences don't imply distinct mechanisms
A new arXiv preprint challenges a core assumption in mechanistic interpretability: that structurally different circuits discovered for the same task imply distinct computational mechanisms. Using Literal Sequence Copying across token-frequency bands in five Pythia models (70M–1.4B), the authors extract 75 circuits and show that structurally distinct circuits implement the same computation, with band-specific edges transferring broadly and a shared core recovering ≥99% of circuit performance. The paper introduces the term 'phantom specialization' for this pattern and argues that standard source-level evaluation inflates apparent faithfulness, while edge-level evaluation and cross-condition transfer tests are needed to detect the many-to-one mapping from structure to function.
Formal theory shows infinite trivial output is provably necessary for AI systems generating valuable mathematics
A new arXiv paper models AI-assisted formal mathematics generation as a nested language-generation-in-the-limit problem, using a proof checker as a membership oracle and an adversarial enumeration of the mathematical literature as the signal for 'valuable' content. The authors prove a sharp dichotomy: generators emitting only finitely many trivial (correct but worthless) statements achieve at most α/2 coverage of unseen valuable mathematics, while allowing an infinite (but asymptotically vanishing) stream of trivia raises the optimum to 1−α/2. The central result is that a perfect verifier cannot substitute for mathematical taste, and the flood of certified-but-trivial output from AI proof systems is a provable mathematical necessity, not an engineering failure. The work formalizes the gap between formal verifiability and mathematical value, which is increasingly the binding constraint as AI-proof-assistant systems scale.
Hyperfitting Explained: Terminal Geometric Expansion in Final Transformer Layers Drives Diversity Gains
This paper investigates the 'hyperfitting' phenomenon—where fine-tuning LLMs to near-zero loss on small datasets improves open-ended generation and reduces repetition—and demonstrates it is mechanistically distinct from temperature scaling. Entropy-matched control experiments falsify both the temperature-equivalence and static vocabulary reweighting hypotheses, instead localizing the effect to a 'Terminal Expansion' in the final transformer block where feature-space dimensionality expands by ~80.8 dimensions, enabling promotion of deep-tail tokens via context-dependent rank reordering. The authors introduce Late-Stage LoRA, a targeted fine-tuning strategy updating only the final 5 layers, achieving robust generation with minimal parameter updates.
Scaling limit theory of the Random Language Model reveals condensation transition and language statistics
A new arXiv preprint develops a quantitative theory of the Random Language Model (RLM), an ensemble of stochastic context-free grammars, in a scaling limit where grammar size and temperature are jointly tuned. The authors identify a condensation phase transition at a critical parameter value and derive explicit scaling laws for entropy, rule diversity, and related observables across distinct regimes. The work claims to resolve prior ambiguities about thermodynamic transitions in language models and offers a unified framework connecting generative grammar statistics to universal properties of natural language and LLM behavior.
HyperTool: Unified executable MCP-style interface reduces step-wise tool call overhead for LLM agents
HyperTool introduces a unified executable interface that allows LLM agents to invoke multiple tool calls within a single code block, hiding intermediate dataflow from the main reasoning trace. This addresses an 'execution-granularity mismatch' where step-wise atomic tool calls waste context and force models to manage low-level operations. On the MCP-Universe benchmark, HyperTool more than doubles accuracy for Qwen3-32B (15.69% → 35.29%) and Qwen3-8B (9.93% → 33.33%), outperforming GPT-OSS and Kimi-k2.5.
Express: Efficient causal attention approximation with formal guarantees and FlashAttention 2 speedups
A new tool called Express converts non-causal attention approximations into causal ones with matching theoretical guarantees, achieving log^(3/2)(n)/s approximation error with O(s) memory. Combined with the Thinformer approximation and an I/O-aware Triton implementation, it demonstrates substantial speedups over FlashAttention 2. The work targets four practical bottlenecks: long-context prefill, KV cache compression, and both memory- and compute-constrained long-form decoding.