6arXiv cs.CL (Computation and Language)·
T²MLR: Temporal Middle-Layer Recurrence enables persistent latent reasoning in Transformers
Researchers introduce T²MLR, a latent reasoning architecture that injects a cached middle-layer representation from the previous token into an earlier layer of the current token, allowing intermediate reasoning states to persist across autoregressive decoding steps. The approach consistently outperforms parameter-matched Transformer baselines on natural-language pretraining and multi-hop reasoning, with recurrence applied to only ~20% of the network often beating full-layer recurrence. Notably, T²MLR can be retrofitted into an existing pretrained 1.7B Transformer with brief finetuning, substantially improving math reasoning without pretraining from scratch. The work challenges prior assumptions that effective latent recurrence requires looping over all layers.
Related guides (2)
Related events (8)
Reasoning in Memory (RiM): Latent Reasoning via Working Memory Blocks in LLMs
RiM introduces a latent reasoning method that replaces autoregressive chain-of-thought token generation with fixed sequences of special 'memory block' tokens, allowing LLMs to perform internal computation without externalizing intermediate steps. These memory blocks are processed in a single forward pass rather than generated autoregressively, improving compute efficiency at test time. Training uses a two-stage curriculum: first grounding memory blocks by predicting explicit reasoning steps, then discarding step-level supervision and refining answers iteratively. Experiments across multiple model families and sizes show RiM matches or exceeds existing latent reasoning methods.
Training-Free Looped Transformers: Inference-Time Recurrence via ODE-Motivated Layer Reapplication
The paper introduces a method to retrofit recurrence onto frozen pretrained transformer checkpoints at inference time by looping a contiguous mid-stack block of layers without any fine-tuning or architectural changes. Naive block reapplication degrades performance, so the authors motivate their approach by treating pre-norm transformer blocks as forward Euler ODE steps and replacing one large update with smaller damped sub-steps. Evaluated across seven model families including dense, sparse MoE, and MLA+MoE architectures, the method yields consistent benchmark improvements (e.g., +2.64 pp on MMLU-Pro for Qwen3-4B-Instruct) at no training cost.
LOTUS: Looped Transformers bridge latent and explicit chain-of-thought reasoning at 3B scale
Researchers introduce LOTUS (Looped Transformers with parallel supervision on latents), a latent chain-of-thought method that processes reasoning steps in hidden states rather than decoded tokens. LOTUS is claimed to be the first latent-CoT approach to match explicit CoT performance at the 3B parameter scale, while reducing thought-phase latency by 2.5x–6.9x. The method uses a looped (recurrent-depth) Transformer backbone with parallel cross-entropy supervision on gold CoT-step tokens at each latent position, and the latent space is shown to be interpretable by projecting through the base LM head to recover reasoning steps.
NF-CoT: Latent reasoning with normalizing flows preserves autoregressive LLM advantages
Researchers propose NF-CoT, a latent reasoning framework that replaces discrete chain-of-thought token streams with continuous intermediate states modeled by normalizing flows embedded inside an LLM backbone. The approach uses a TARFlow-style normalizing flow head alongside the standard language model head, enabling exact likelihoods, KV-cache-compatible left-to-right decoding, and policy-gradient optimization in latent space. On code-generation benchmarks, NF-CoT improves pass rates over both explicit CoT and prior latent-reasoning baselines while reducing intermediate reasoning cost. The work addresses a key limitation of existing latent reasoning methods, which typically sacrifice probabilistic tractability or autoregressive compatibility.
Associative Recurrent Memory Transformer enables long-context LLMs with constant memory scaling
Researchers propose using the Associative Recurrent Memory Transformer (ARMT) as a practical method for extending LLM context length beyond original limits while achieving constant memory scaling instead of linear. The training recipe combines continued pre-training, synthetic long-context data, curriculum learning, and selective layer integration of associative memory. Experiments show ARMT-augmented models generalize to out-of-distribution context lengths and require 30% fewer FLOPs while preserving in-window baseline performance.
Theoretical framework reveals low-dimensional invariant manifolds governing Transformer inductive reasoning dynamics
A new arXiv preprint presents a theoretical framework explaining how Transformers develop inductive reasoning abilities by proving that training dynamics of attention models are confined to a low-dimensional invariant manifold. The framework unifies several synthetic inductive tasks (in-context n-grams, multi-hop reasoning) and characterizes how data statistics govern competition between in-context and in-weights learning. The authors show that random initializations determine which circuit 'wins' when multiple solutions exist, and that the manifold's coordinate frame can automatically detect learned circuits in trained models. The work advances mechanistic interpretability by casting circuit formation as a tractable low-dimensional dynamical phenomenon.
Latent Memory Palace: Variational inference framework enables adaptive reasoning for continuous control policies
Researchers introduce Latent Memory Palace (LMP), a method that formulates reasoning for continuous control policies as variational inference over an autoregressive latent distribution, analogous to a memory palace. The approach derives a latent-space reinforcement learning technique to optimize the variational lower bound, yielding a policy (LMP-π) with adaptive test-time compute allocation and a variable-length action tokenizer (LMP-tok) that improves downstream autoregressive policies. The work addresses the gap between language model chain-of-thought reasoning and continuous control, where language-space reasoning lacks spatial granularity. Results are demonstrated in both simulation and real-world domains.
ReContext: Training-free recursive evidence replay improves LLM long-context reasoning
Researchers introduce RECONTEXT, a training-free inference-time method for improving long-context reasoning in LLMs. The approach uses model-internal relevance signals to build a query-conditioned evidence pool that is replayed before final generation, without modifying the original context, external memory, or context pruning. Experiments across eight long-context datasets at 128K context length show consistent improvements on Qwen3-4B, Qwen3-8B, and Llama3-8B. The authors provide a theoretical grounding via associative memory theory, framing attention as cue-trace association and replay as trace reactivation.