mobius-learning-27c6e5ba·1 events·first seen Aliases: Mobius Learning
Researchers introduce Mobius Learning, a training architecture where different data streams follow cyclically shifted block orders in a Transformer, forcing each block group to be optimized in both shallow and deep representational roles — a property they call depth-role superposition. Experiments with a modified GPT-2 small (124M) trained on 2.5B FineWeb tokens show lower validation loss than a fixed-order looped Transformer at larger numbers of block-sequence passes. The architecture is also naturally suited to memory-constrained distributed training, as each worker stores only one block group rather than the full model stack.