C²R: Cross-sample Consistency Regularization Mitigates Feature Splitting and Absorption in Sparse Autoencoders

C²R regularization method addresses feature splitting and absorption in Sparse Autoencoders

A new arXiv preprint introduces C²R (Cross-sample Consistency Regularization), a training technique for Sparse Autoencoders (SAEs) that mitigates two known failure modes: feature splitting, where coherent concepts fragment across multiple latents, and feature absorption, where general features develop arbitrary exceptions. C²R penalizes co-activation of directionally similar latents across a batch, encouraging each semantic concept to map consistently to a single latent. The authors report that C²R reduces both pathologies while preserving reconstruction fidelity, with source code released publicly.