Entity · paper

Inside the Unfair Judge: A Mechanistic Interpretability Account of LLM-as-Judge Bias

paperactiveprovisionalinside-the-unfair-judge-a-mechanistic-interpretability-account-of-llm-as-judge-bias-3f060780·1 events·first seen 30h ago

Aliases: Inside the Unfair Judge: A Mechanistic Interpretability Account of LLM-as-Judge Bias

More like this (12)

Can LLMs Judge Better Than They Generate? Evaluating Task Asymmetry, Mechanistic Interpretability and Transferability for In-Context QA When the Judge Changes, So Does the Measurement: Auditing LLM-as-Judge Reliability LLM-as-a-Judge LLM-judge scoring LLM-judged explanation score Towards Explainable Adjudicative Variance: Quantifying Judicial Discretion via Gated Multi-Task Learning LLM Judges Can Be Too Generous When There Is No Reference Answer Flaws in the LLM Automation Narrative Surrogate Fidelity: When Can Open LLMs Explain Closed Ones?Beyond Third-Person Audits: Situated Interaction Auditing for User-Centered LLM Bias Research Challenges and Recommendations for LLMs-as-a-Judge in Multilingual Settings and Low-Resource Languages A Tree-of-Thoughts Inspired Hybrid Approach for Legal Case Judgement Summarization using LLMs

Recent events (1)

6arXiv · cs.CL·30h ago·source ↗

Mechanistic interpretability study reveals geometric structure of LLM-as-judge bias in hidden states

A new arXiv preprint investigates LLM-as-judge scoring bias at the representation level rather than the input-output level, studying seven judge models across seven bias types and nine benchmarks. The authors find that biased inputs are displaced along low-dimensional, type-specific subspaces in activation space, and that steering hidden states along these subspaces causally controls scoring direction. A linear projection onto bias-direction features predicts judge failures on unseen benchmarks, substantially outperforming text-based alternatives. The work provides a mechanistic account that unifies geometric structure, causal control, and operational prediction within a single framework.

Evaluation and Benchmarking AI Safety Research Inside the Unfair Judge: A Mechanistic Interpretability Account of LLM-as-Judge Bias