vit-b-16-5e33f7de·2 events·first seen Aliases: ViT-B/16, ViT-L/16
Researchers introduce SLORR, a framework for in-training low-rank regularization that avoids SVD computation, architectural modifications, and stateful caching. Two variants based on Hoyer sparsity and nuclear norm are evaluated on ImageNet-1K (ResNet-50, ViT-B/16, ViT-L/16) and LLM pretraining at 135M and 560M parameter scales. SLORR-trained models compress more effectively than unregularized baselines while adding under 8% training overhead for vision models and under 1% for LLM pretraining, making post-training compression more viable.
A new arXiv preprint applies a causal intervention inspired by Oppenheim and Lim (1981) to probe whether trained image classifiers encode identity in Fourier phase rather than magnitude within their hidden layers. By transplanting phase or sign components between images at chosen layers in PRISM2D, GFNet, ViT-B/16, and ResNet-50, the authors find that predictions follow the phase/sign donor across all tested architectures, with image-specific magnitude largely dispensable. ResNet-50 requires a pre-ReLU intervention to reveal a latent sign code, exposing how rectification and readout geometry shape the basis in which the code is expressed. The findings offer a mechanistic account of the texture–shape gap between CNNs and attention-based models.