Entity · technique

Associative Recurrent Memory Transformer

techniqueactiveprovisionalassociative-recurrent-memory-transformer-a8f37232·1 events·first seen 24h ago

Aliases: Associative Recurrent Memory Transformer

Co-occurring entities

Extending LLM Context via Associative Recurrent Memory

More like this (12)

autoregressive transformer Extending LLM Context via Associative Recurrent Memory Graph Transformer TRL (Transformer Reinforcement Learning)Bridging the Gap Between Latent and Explicit Reasoning with Looped Transformers Sparse Transformer Invariant Learning Dynamics of Transformers in Inductive Reasoning Tasks Invariant Learning Dynamics of Transformers in Inductive Reasoning Tasks Recurrent Neural Network Action Chunking Transformer Supervised Memory Training Dynamic Short Convolutions Improve Transformers

Recent events (1)

6arXiv · cs.CL·24h ago·source ↗

Associative Recurrent Memory Transformer enables long-context LLMs with constant memory scaling

Researchers propose using the Associative Recurrent Memory Transformer (ARMT) as a practical method for extending LLM context length beyond original limits while achieving constant memory scaling instead of linear. The training recipe combines continued pre-training, synthetic long-context data, curriculum learning, and selective layer integration of associative memory. Experiments show ARMT-augmented models generalize to out-of-distribution context lengths and require 30% fewer FLOPs while preserving in-window baseline performance.

Long Context Evolution Inference Economics Extending LLM Context via Associative Recurrent Memory Associative Recurrent Memory Transformer