5arXiv cs.CL (Computation and Language)·
Möbius RoPE: Anti-periodic positional encoding improves in-context retrieval reliability
A new arXiv preprint introduces Möbius RoPE, a rotary positional encoding variant using an anti-periodic frequency ladder (θ_i = π(2i+1)/N) that couples sequence endpoints via a closed-form Dirichlet 'dipole' — claimed as the first anti-periodic boundary condition in positional encoding. The authors pretrain 48 models (160M and 410M parameter classes) on 2B FineWeb-Edu tokens and find that a hybrid Möbius arm raises needle-in-a-haystack retrieval from 63.3% to 90.3% at context length 512 with no perplexity cost (29.66 vs. 29.72). The effect is isolated to the anti-periodic structure specifically, as aperiodic and periodic ladder controls do not reproduce it, and swapping the frequency table back to standard RoPE collapses retrieval performance.
Related guides (2)
Related events (8)
2D-RoPE positional encoding enables frontier-length exact copying in Transformers
A new arXiv preprint demonstrates that frontier LLMs systematically fail at exact string copying within their context windows, attributing the failure to 1D positional encodings that encourage shortcut matching of local contexts rather than precise position retrieval. The authors introduce 2D-RoPE, which assigns each token a row and column ID in a 2D grid, making copying equivalent to a fixed column-offset lookup. Shallow Transformers with 2D-RoPE achieve perfect copying at lengths hundreds of times beyond training distribution, and the advantage persists in large-scale pretraining on DCLM up to 1.4B parameters. The result challenges assumptions about what frontier models can reliably do and proposes a concrete architectural modification to positional encoding.
Data-driven theory explains RoPE frequency usage and long-context generalization in transformers
A new arXiv preprint proposes a data-centered explanation for why trained transformers use RoPE positional frequencies non-uniformly: frequencies are selected to match the relative-distance dependency structure of training data, with optimal frequency scaling as 1/W for dependency width W. The paper formalizes a field-resolution tradeoff and connects this frequency-matching principle to position-interpolation-based length generalization, showing that test-time frequency scaling succeeds when longer-context dependencies are approximate dilations of training-time dependencies. Empirical results demonstrate that natural language exhibits approximate self-similarity across positional scales, providing a mechanistic account of why context-length extrapolation methods work when they do.
You Could Have Designed State of the Art Positional Encoding
A Hugging Face blog post walks through the design space of positional encoding for transformer models, building intuition for why modern schemes like RoPE emerged. The post takes a pedagogical approach, showing how one could derive state-of-the-art positional encoding from first principles. It covers the evolution from absolute to relative positional encodings and the properties that make certain schemes preferable for long-context generalization.
ROPD: Routing-based On-Policy Distillation for template-robust LLM safety realignment
A new arXiv preprint introduces Routing-based On-Policy Distillation (ROPD), a safety realignment framework designed to counter malicious fine-tuning attacks that embed harmful behaviors into downstream model corpora. ROPD models the divergence between aligned and compromised output distributions rather than fitting specific prompt templates, addressing three failure modes of existing defenses: catastrophic forgetting, template-mismatch collapse, and re-jailbreaking via system prompt switches. Experiments across four baselines, three datasets, and three base models show ROPD substantially reduces template-mismatch degradation while preserving downstream task performance.
Randomized YaRN improves LLM length generalization for long-context reasoning
Researchers propose Randomized YaRN, a training method that combines YaRN-based positional extrapolation with randomized positional encodings and a length curriculum to improve LLM generalization to long contexts. Models trained on sequences under 8K tokens show consistent reasoning improvements on context lengths from 16K to 128K on BABILong and MRCR benchmarks. The key insight is that exposing models to out-of-distribution positional representations during short-context training enables better generalization at far longer inference-time lengths.
MoE²-LoRA: MoE-style low-rank adaptation for Mixture-of-Experts language models
Researchers introduce MoE²-LoRA, a parameter-efficient fine-tuning method specifically designed for Mixture-of-Experts (MoE) language models. The approach uses a dual-channel Routing-Conditioned Projection module that reuses base router activations to guide LoRA adapter routing, and introduces a single global LoRA expert pool shared across all layers. Evaluated across multiple MoE backbones at varying scales, MoE²-LoRA claims state-of-the-art downstream accuracy while better preserving general capabilities compared to existing PEFT methods.
Pair-In, Pair-Out (PIPO): Unified Latent Compression and Multi-Token Prediction for Efficient LLM Inference
PIPO is a new inference efficiency framework that unifies input-side latent compression with output-side multi-token prediction (MTP) by treating them as mirror operations: a compressor folds two input tokens into one latent, while an MTP head unfolds one hidden state into an additional output token. To avoid the expensive verifier pass typically required by speculative decoding, PIPO trains a lightweight confidence head using On-Policy Distillation (OPD), which naturally aligns with rejection-sampling criteria. Experiments on Qwen3.5-4B and 9B backbones across AIME 2025, GPQA-Diamond, LiveCodeBench v6, and LongBench v2 show up to 2.64× first-token-latency speedup and +7.15 pass@4 improvement over regular decoding.
Anthropic introduces Contextual Retrieval to reduce RAG retrieval failures by up to 67%
Anthropic published a technical method called Contextual Retrieval that combines Contextual Embeddings and Contextual BM25 to address the context-loss problem in traditional RAG pipelines. The approach prepends chunk-level context before encoding, reducing failed retrievals by 49% standalone and 67% when combined with reranking. The post also highlights prompt caching as a simpler alternative for knowledge bases under 200K tokens, and provides a cookbook for deployment with Claude.