nanogpt-7becc343·2 events·first seen Aliases: nanoGPT
A new arXiv preprint presents a geometric analysis of the memorization-to-generalization delay (grokking) in neural networks, attributing it to radial inflation of hidden representations under cross-entropy optimization. The authors formalize a radial-angular decomposition of activation-space dynamics and derive three testable propositions about the effects of penalizing radial growth. Empirically, a single-hyperparameter norm penalty that constrains activations to a sqrt(d)-radius hypersphere accelerates grokking up to 6x on modular arithmetic tasks across MLPs and Transformers, and halves training steps for a 10M-parameter nanoGPT on 3-digit addition.
Hugging Face published nanoVLM, a minimal open-source repository designed to make training vision-language models (VLMs) as simple as possible using pure PyTorch. The project aims to lower the barrier to entry for VLM research and experimentation by providing a clean, readable codebase without heavy abstractions. It follows in the tradition of educational ML repositories like nanoGPT, targeting researchers and practitioners who want to understand or customize VLM training from scratch.