Entity · paper

Train the Model, Not the Reader: Decodability Supervision for Verifiable Activation Explanations

paperactiveprovisionaltrain-the-model-not-the-reader-decodability-supervision-for-verifiable-activation-explanations-6b8c0e3e·1 events·first seen 2d ago

Aliases: Train the Model, Not the Reader: Decodability Supervision for Verifiable Activation Explanations

Co-occurring entities

RECAP Pythia-410M Qwen 2.5-7B

More like this (12)

Listening with Attention: Entropy-Guided Explainability for Transformer-Based Audio Models Towards Explainable Adjudicative Variance: Quantifying Judicial Discretion via Gated Multi-Task Learning A Practical Investigation of Training-free Relaxed Speculative Decoding encoder-only language models Evaluating RE Practices for Explainability: Synthesizing Insights from Daimler Truck into an Explainable RE Framework Proposal Introspective Coupling: Self-Explanation Training Tracks Behavioral Change Despite Fixed Supervision Verifier-in-the-Loop Training (ViL)Decomposing Factual Sycophancy in Language Models: How Size and Instruction Tuning Shape Robustness Reasoning Language Models Exposure is Optional: Learning Unlike Coordination in Language Models Expert-Aware Causal Tracing of Factual Recall in Sparse MoE Language Models Knowledgeless Language Models: Suppressing Parametric Recall for Evidence-Grounded Language Modeling

Recent events (1)

7arXiv · cs.CL·2d ago·source ↗

RECAP: Decodability supervision for verifiable neural activation explanations

A new arXiv paper identifies a fundamental flaw in natural-language autoencoder-based explanation methods for neural activations: high reconstruction scores do not certify individual factual claims, and the standard training recipe produces co-adapted private codes (false wordings that reconstruct well) in all tested runs. The authors introduce two audit protocols and RECAP (Readable Encodings via Co-trained Auxiliary Predictors), which trains linear probe heads alongside the target model to keep designated content independently decodable. On RECAP-trained models, fresh verbalizers state designated content truthfully and private codes vanish, with probes achieving AUC 0.96 for distinguishing true from false claims—and remaining robust (AUC 0.95) against adversarial explanation edits that collapse control probes to chance. The work has direct implications for mechanistic interpretability and AI safety, as it challenges a widely-used faithfulness test and proposes a more verifiable alternative.

Evaluation and Benchmarking AI Safety Research RECAP Train the Model, Not the Reader: Decodability Supervision for Verifiable Activation Explanations Pythia-410M +1 more