4arXiv cs.LG (Machine Learning)·
NeuronSoup: Asynchronous shared-neuron temporal graph architecture evolved without backpropagation
NeuronSoup is a neural computation architecture that replaces synchronous layer-by-layer processing with asynchronous, delay-mediated signal propagation through a pool of physically shared neurons, optimized entirely by a genetic algorithm rather than gradient descent. On MNIST digit classification using frozen ResNet18 features, the evolved network achieves 85.9% test accuracy with 204 active paths through 266 hidden neurons, fitting in 115 KB. The architecture requires no differentiable computation graph, adapts computation depth per sample, and discovers lateral pathway interactions without explicit engineering. The authors argue genetic algorithms are the appropriate optimizer for this problem class and discuss why CMA-ES fails at this scale.
Related guides (1)
Related events (8)
Online Neural Space-Time Memory enables real-time dynamic novel view synthesis from streaming video
A new arXiv preprint proposes a neural memory architecture for real-time novel view synthesis from multi-view streaming video of dynamic scenes. The method decouples memory update frequency from memory application frequency, using periodic gradient-based updates with per-frame cross-view attention to handle deformations. Two mechanisms — an auxiliary Memory Loss and a Memory Caching strategy — prevent catastrophic forgetting over long contexts. The approach achieves state-of-the-art performance on dynamic human motion scenes with minute-scale online memorization at real-time speeds.
Rosetta Neurons follow sublinear power-law scaling with model size, becoming more monosemantic at scale
A new arXiv paper investigates how neuron populations evolve with scale in both language models (up to 30B parameters) and vision models (up to 5B parameters), focusing on 'Rosetta Neurons' — neurons with similar activation patterns across independently trained models. The authors find Rosetta Neurons grow in absolute count but shrink as a fraction of total neurons, and exhibit a 'Neuron Polarization Effect' where they become increasingly monosemantic while non-Rosetta neurons remain less selective. An analytical model explains the sublinear power-law scaling, and the paper demonstrates practical utility via a targeted data-filtering case study for continued pretraining. The results extend scaling laws to neuron-level interpretability structure, linking model size to systematic changes in universality and specialization.
Neuronal Stochastic Attention Circuit (NSAC) for Probabilistic Representation Learning
Researchers introduce NSAC, a biologically-inspired continuous-time attention architecture that models attention logits as solutions to an Ornstein-Uhlenbeck stochastic differential equation, drawing on C. elegans Neuronal Circuit Policy wiring to induce Gaussian distributions over attention weights. The architecture enables joint quantification of aleatoric and epistemic uncertainty via a two-term objective combining Gaussian negative log-likelihood with an epistemic-separation regularizer. Empirical evaluation spans irregular time-series function approximation, multivariate regression, long-range forecasting, Industry 4.0 tasks, and autonomous vehicle lane-keeping, showing competitive accuracy with well-calibrated uncertainty estimates.
Neuron-OPSD: Annotation-free LLM self-distillation guided by internal neuron activations
Researchers introduce Neuron On-Policy Self-Distillation (Neuron-OPSD), a data-centric framework for post-training LLMs without human annotations or real-world interaction feedback. The method uses internal neuron activations to guide training-data selection and teacher context construction, then trains via on-policy distillation from the teacher distribution. Evaluated on specialized-domain benchmarks, Neuron-OPSD improves in-domain performance while preserving cross-domain generalization and avoiding the calibration collapse seen in prior SFT-, GRPO-, and reward-RL-based annotation-free approaches.
Transformer-guided swarm intelligence framework for low-resource neural architecture search
A new arXiv preprint proposes a hybrid NAS framework combining an autoregressive Transformer controller trained via reinforcement learning with an Artificial Bee Colony metaheuristic for local search. The system targets consumer-grade hardware (NVIDIA RTX 3060), completing architecture search in ~3 hours and finding compact models: 84.85% on CIFAR-10 with ~174K parameters and F1=0.71 on fraud detection with ~4,600 parameters. A dynamic entropy mechanism is introduced to prevent premature convergence during RL training. The work addresses the democratization angle of NAS, positioning results as suitable for edge deployment.
Supervised Memory Training enables parallel RNN pretraining without backpropagation through time
A new arXiv preprint proposes Supervised Memory Training (SMT), a method that trains recurrent neural networks by reducing the problem to supervised learning on one-step memory transitions, bypassing backpropagation through time entirely. A Transformer-based encoder generates memory labels via a predictive state objective, enabling time-parallel training with O(1) gradient path length between any two tokens. SMT outperforms BPTT on language modeling and pixel sequence modeling tasks across multiple RNN architectures. The approach could enable RNNs to scale more effectively by decoupling memory content from update mechanics.
Dynamic short convolutions yield 1.33–1.60× compute advantage over standard Transformers
A new arXiv preprint introduces dynamic short convolutions as an architectural primitive for Transformers, using input-dependent filters to combine locality bias with increased expressivity. Experiments across 150M–2B parameter language models show consistent perplexity improvements over standard Transformers and static convolution variants, with scaling-law fits indicating a 1.33× compute advantage when applied to key/query/value vectors and 1.60× when added after every linear layer. The technique also improves linear RNNs (Mamba-2, Gated DeltaNet) and mixture-of-experts architectures, with custom Triton kernels making training practical.
Training-Free Looped Transformers: Inference-Time Recurrence via ODE-Motivated Layer Reapplication
The paper introduces a method to retrofit recurrence onto frozen pretrained transformer checkpoints at inference time by looping a contiguous mid-stack block of layers without any fine-tuning or architectural changes. Naive block reapplication degrades performance, so the authors motivate their approach by treating pre-norm transformer blocks as forward Euler ODE steps and replacing one large update with smaller damped sub-steps. Evaluated across seven model families including dense, sparse MoE, and MLA+MoE architectures, the method yields consistent benchmark improvements (e.g., +2.64 pp on MMLU-Pro for Qwen3-4B-Instruct) at no training cost.