How Data Shapes RoPE Frequency Usage: From Positional Scale Matching to Length Generalization

Data-driven theory explains RoPE frequency usage and long-context generalization in transformers

A new arXiv preprint proposes a data-centered explanation for why trained transformers use RoPE positional frequencies non-uniformly: frequencies are selected to match the relative-distance dependency structure of training data, with optimal frequency scaling as 1/W for dependency width W. The paper formalizes a field-resolution tradeoff and connects this frequency-matching principle to position-interpolation-based length generalization, showing that test-time frequency scaling succeeds when longer-context dependencies are approximate dilations of training-time dependencies. Empirical results demonstrate that natural language exhibits approximate self-similarity across positional scales, providing a mechanistic account of why context-length extrapolation methods work when they do.