paper

The Neutral Mask: How RLHF Provides Shallow Alignment while Leaving Partisan Structure Intact in a Large Language Model

paperactiveprovisional

the-neutral-mask-how-rlhf-provides-shallow-alignment-while-leaving-partisan-structure-intact-in-a-large-language-model-382fa6b6

·1 events·first seen 8d ago

Aliases: The Neutral Mask: How RLHF Provides Shallow Alignment while Leaving Partisan Structure Intact in a Large Language Model

Co-occurring entities

Reinforcement Learning from Human Feedback Sparse Autoencoder Meta Llama-3.1-8B

More like this (12)

Contrastive-Difference CKA Reveals Concept-Specific Structural Alignment Across Language Model Architectures Recursive Language Models (RLMs)Exploring Adversarial Robustness and Safety Alignment in Multilingual Multi-Modal Large Language Models The Shibboleth Effect: Auditing the Cross-Lingual Distributional Skew of Large Language Models Does Reasoning Preserve Alignment? On the Trustworthiness of Large Reasoning Models Unintended Effects of Geographic Conditioning in Large Language Models Beyond Fully Random Masking: Attention-Guided Denoising and Optimization for Diffusion Language Models Large Language Models (frontier)Language Modeling Loss Scaling LLM Reasoning from Minimal Labels: A Semi-Supervised Framework with a Lightweight Verifier Transformer Language Models Multi-Faceted Interactivity Alignment in Full-Duplex Speech Models

Recent events (1)

7arXiv · cs.CL·8d ago·source ↗

RLHF produces shallow political neutrality by severing causal pathways, not erasing partisan structure

Researchers compare internal representations of Llama 3.1 8B before and after RLHF, finding that alignment training does not remove partisan political geometry from the model but instead compresses output variance to produce balanced responses. Sparse autoencoder decomposition shows that policy-encoding features active in the base model become completely inactive in the instruction-tuned version, while feature-level steering experiments confirm the causal disconnect is real. The underlying partisan structure remains intact and can be reactivated by inferring and amplifying a user's partisan identity, suggesting RLHF alignment is functionally fragile. The authors argue this 'disconnection rather than removal' pattern may generalize to other value domains beyond political orientation.

AI Safety Research Alignment and RLHF Reinforcement Learning from Human Feedback The Neutral Mask: How RLHF Provides Shallow Alignment while Leaving Partisan Structure Intact in a Large Language Model Sparse Autoencoder +2 more