SST-2

Procrustes-conditioned Joint SAE extracts cross-seed universal features from BERT models

Researchers introduce a Procrustes-conditioned Joint End-to-end Top-K Sparse Autoencoder (SAE) to address cross-seed feature universality in mechanistic interpretability of BERT models. By applying an orthogonal Procrustes rotation between independently trained models' activation spaces before joint SAE training, the method produces more consistent features (Pearson r ≥ 0.70) than post-hoc alignment baselines across three NLP benchmarks. The work targets a fundamental challenge in dictionary learning: non-convex optimization causes independently trained networks to learn misaligned feature spaces, making it difficult to identify truly universal features. High-universality features are shown to encode interpretable sociolinguistic patterns.