superposition

Paper argues Compressed Computation toy model is not computation in superposition

A new arXiv preprint challenges the Compressed Computation (CC) toy model introduced by Braun et al. (2025), which appeared to compute 100 ReLU functions using only 50 neurons. The authors show that apparent performance gains arise from unintended input mixing via a noisy residual stream rather than genuine superposition, with learned neuron directions concentrating in the subspace of the top 50 eigenvalues of the mixing matrix. A semi-non-negative matrix factorization baseline derived purely from the mixing matrix reproduces the qualitative loss profile, supporting the conclusion that CC is not a valid toy model of computation in superposition.