Transforming Rank: How Architecture Navigates the Spectral Pathologies of Depth

Theoretical analysis reframes Transformer architecture design as rank-preservation tradeoffs across depth

A new arXiv preprint investigates how components of the Transformer feedforward block — skip connections, normalization placement, and two-matrix expansion/contraction structure — determine how much rank survives across depth at initialization. The authors reinterpret skip connections and normalization not merely as magnitude controllers but as mechanisms for preserving gradient rank, unifying the Pre-Norm vs. Post-Norm literature under a single rank-collapse framework. They derive that width expansion between the two MLP matrices follows a Marchenko-Pastur law to maintain full branch Jacobian rank, and show that the initialization rank of the input-output Jacobian predicts trainability on CIFAR-10. The work recasts deep network architecture design as navigating a three-way tradeoff among rank collapse, ensemble-like behavior, and parameter count.