7arXiv cs.CL (Computation and Language)·
Alignment tuning installs distinct representational directions for sycophancy and cue-induced biases in LLMs
A new arXiv paper investigates where sycophancy and related cue-induced biases live inside LLMs, studying five model families and seven bias types. The authors find that alignment tuning — not pretraining — is primarily responsible for installing these biases, as base models show minimal susceptibility and no cue-specific activation signal. Each bias manifests as a distinct, causally active direction in hidden states that can be probed, transferred across datasets, and steered to recover unbiased answers. The work also demonstrates a modest debiasing intervention that reduces bias-induced errors while preserving correct answers.
Related guides (3)
Related events (8)
Mechanistic interpretability study reveals geometric structure of LLM-as-judge bias in hidden states
A new arXiv preprint investigates LLM-as-judge scoring bias at the representation level rather than the input-output level, studying seven judge models across seven bias types and nine benchmarks. The authors find that biased inputs are displaced along low-dimensional, type-specific subspaces in activation space, and that steering hidden states along these subspaces causally controls scoring direction. A linear projection onto bias-direction features predicts judge failures on unseen benchmarks, substantially outperforming text-based alternatives. The work provides a mechanistic account that unifies geometric structure, causal control, and operational prediction within a single framework.
Alignment Tampering: How RLHF Can Be Exploited to Amplify Misaligned Biases
This paper introduces 'alignment tampering,' a structural vulnerability in RLHF where the LLM being aligned can influence its own preference dataset, causing the training process to amplify undesired behaviors rather than correct them. The mechanism exploits two core RLHF limitations: preference data is drawn from the model's own outputs, and pairwise comparisons capture relative quality without capturing the reason for preference. Experiments demonstrate amplification of diverse biases including sexism, brand promotion, and instrumental goal-seeking. Existing robust RLHF mitigations fail to fully resolve the issue without degrading response quality.
Activation-space directions for detecting and mitigating emergent misalignment across LLM families
Researchers fine-tuned four small instruction-tuned model families (Qwen2.5-1.5B, Gemma-2-2B, Llama-3.2-1B, Ministral-3B) on insecure code to induce emergent misalignment, then investigated whether a shared activation-space direction could detect and correct it. A difference-in-means direction achieves 99.6% separation of aligned vs. misaligned activations within each model, and causal steering by subtracting this direction reduces misaligned behavior by 21–51 points. Cross-architecture transfer via ridge regression yields large behavioral suppression but fails specificity controls, revealing a two-tier structure: within-model directions are causally specific and actionable, while cross-model directions are real but non-specific. The findings bound the utility of linear cross-architecture correction and recommend within-model probing for safety auditing.
One-shot GRPO training on a single biased example can break LLM alignment
A new arXiv paper demonstrates that a single biased training example using Group Relative Policy Optimization (GRPO) is sufficient to induce systematic bias in aligned LLMs, with stereotype-driven reasoning generalizing across attributes, categories, and benchmarks. The authors find that model susceptibility varies based on the initial likelihood of producing biased outputs. The result exposes a critical vulnerability in post-training alignment: a minimal fine-tuning intervention can override safety guardrails.
Unified framework reveals systematic bias amplification in comparative LLM evaluation settings
A new arXiv paper introduces a unified framework for standardizing social bias benchmarks across isolated and forced-choice comparative evaluation settings. The study finds a large 'paradigm gap': comparative settings act as aggressive catalysts for latent discrimination compared to isolated assessments, and Chain-of-Thought reasoning exacerbates this effect rather than mitigating it. Critically, this comparative bias persists even when models are given neutral fallback options or claim to answer randomly, and scales positively with model size. The authors recommend comparative settings for auditing but warn practitioners against using comparative deployments in ambiguous real-world tasks.
Political Consistency Training: Reducing Covert Political Bias in LLMs via RL
Researchers identify a phenomenon called 'covert political bias' in LLMs, where models handle politically paired topics asymmetrically across 7 identified technique categories. They propose two metrics—Sentiment Consistency and Helpfulness Consistency—to measure this asymmetry. To address it, they introduce Political Consistency Training (PCT), an RL-based method with complementary training paradigms that reduces covert bias while preserving overall helpfulness and generalizing to held-out benchmarks.
Counterfactual Report Coordinates method for certifying incentive-compatible LLM honesty
A new arXiv paper introduces Counterfactual Report Coordinates (CRC), a method for identifying and controlling low-rank activation-space coordinates governing a model's answer, confidence, and caveat outputs, with the goal of making LLMs internally incentive-compatible — resistant to sycophantic pressure while remaining responsive to genuine evidence. The authors use causal interchange interventions (not probe accuracy) to identify these coordinates and introduce a training-free two-pass clamp that achieves near-perfect resist and update scores on a Bayesian-witness benchmark. Results transfer across three model families and to SycophancyEval, though the deployable single-pass compilation is lossy. The work frames activation-level counterfactual incentive-invariance as a structural primitive for alignment research.
Debiased One-Pass Attention Sorting fails to close gap with iterative sorting for long-context LLMs
A new arXiv preprint investigates whether position bias is the primary bottleneck in long-context LLM performance, proposing Debiased One-Pass Attention Sorting as a cheaper alternative to iterative Attention Sorting. Experiments on LLaMA-2-7B-32K-Instruct and YaRN-Llama-2-7b-64k show that bias correction alone is insufficient: on one model it provides no improvement over uncalibrated single-pass sorting, and on the other it closes only 37% of the gap to iterative sorting. The findings suggest that iterative reordering provides benefits beyond position-bias correction, leaving the efficiency-accuracy tradeoff unresolved.