2D-RoPE

2D-RoPE positional encoding enables frontier-length exact copying in Transformers

A new arXiv preprint demonstrates that frontier LLMs systematically fail at exact string copying within their context windows, attributing the failure to 1D positional encodings that encourage shortcut matching of local contexts rather than precise position retrieval. The authors introduce 2D-RoPE, which assigns each token a row and column ID in a 2D grid, making copying equivalent to a fixed column-offset lookup. Shallow Transformers with 2D-RoPE achieve perfect copying at lengths hundreds of times beyond training distribution, and the advantage persists in large-scale pretraining on DCLM up to 1.4B parameters. The result challenges assumptions about what frontier models can reliably do and proposes a concrete architectural modification to positional encoding.