sparse-autoencoders-c951245e·3 events·first seen Aliases: Sparse Autoencoders
A new arXiv preprint introduces C²R (Cross-sample Consistency Regularization), a training technique for Sparse Autoencoders (SAEs) that mitigates two known failure modes: feature splitting, where coherent concepts fragment across multiple latents, and feature absorption, where general features develop arbitrary exceptions. C²R penalizes co-activation of directionally similar latents across a batch, encouraging each semantic concept to map consistently to a single latent. The authors report that C²R reduces both pathologies while preserving reconstruction fidelity, with source code released publicly.
Researchers apply sparse autoencoders to inspect LLM internal states during forecasting tasks, identifying features associated with time-aware versus look-ahead-biased reasoning. Amplifying time-awareness features causally reduces look-ahead bias while preserving general reasoning performance, whereas directly steering look-ahead-bias features has no effect. The work demonstrates that interpretable temporal features can shift LLMs toward more historically grounded forecasting. This is a mechanistic interpretability result with practical implications for LLM-based prediction systems.
A new arXiv paper investigates feature stability in sparse autoencoders (SAEs), measuring the probability that individual learned features reappear across independent training runs. The authors find a functional asymmetry: stable features carry most reconstruction-relevant signal, while unstable features are individually non-reproducible but concentrate in reproducible lower-rank subspaces, suggesting seed dependence reflects basis ambiguity rather than noise. A synthetic model confirms that low-rank ground-truth features can be recovered at the subspace level even when individual SAE latents are non-identifiable across seeds. The work has direct implications for interpretability research that relies on SAE features as meaningful, stable units of analysis.