5arXiv cs.CL (Computation and Language)·
S2D framework evaluates regional bias in LLMs across China's 34 provincial regions
Researchers introduce Stereotypes-to-Decisions (S2D), a framework for evaluating regional bias in LLMs spanning abstract stereotype ratings (Warmth and Competence) through concrete social decisions (Education, Occupation, Social Interaction) across all 34 provincial-level regions of China. Six LLMs are evaluated, revealing substantial and consistent regional differences that correlate with economic and digital development indicators and persist across Chinese and English prompts. The findings suggest regional bias in LLMs is systematic and consequential, with cross-model agreement especially strong for Competence and Occupation decisions.
Related guides (2)
Related events (8)
C4STYLI Benchmark: Probing Cultural Aesthetic Stylistics Awareness in LLMs
Researchers introduce C4STYLI, a benchmark of stylized translated movie titles and advertising slogans from Hong Kong and mainland China, designed to evaluate LLMs on cross-cultural aesthetic stylistics. Evaluations reveal that LLMs diverge from human stylistic recognition, with recognition ability varying by text domain and not consistently predicting generation performance. Structural ablation using logistic regression probes shows that LLMs in the Hong Kong setting rely on surface-level linguistic cues rather than deeper stylistic structure, indicating limited cultural sensitivity.
Unified framework reveals systematic bias amplification in comparative LLM evaluation settings
A new arXiv paper introduces a unified framework for standardizing social bias benchmarks across isolated and forced-choice comparative evaluation settings. The study finds a large 'paradigm gap': comparative settings act as aggressive catalysts for latent discrimination compared to isolated assessments, and Chain-of-Thought reasoning exacerbates this effect rather than mitigating it. Critically, this comparative bias persists even when models are given neutral fallback options or claim to answer randomly, and scales positively with model size. The authors recommend comparative settings for auditing but warn practitioners against using comparative deployments in ambiguous real-world tasks.
Situated Interaction Auditing: A user-centered framework for LLM bias research
Researchers propose Situated Interaction Auditing (SIA), a new framework for studying LLM bias from the perspective of the user rather than third-party demographic representation. The core insight is that bias can manifest in how a model treats its interlocutor — varying response quality, content, and tone based on implicit sociodemographic signals, writing style, or stated identity — rather than only in how it describes external groups. The paper demonstrates SIA through a case study intersecting gender and socioeconomic status signals across multiple task domains and outlines a research agenda for the approach.
Geopolitical Bias in LLMs Originates in Post-Training, Not Pre-Training Data
A study testing seven open-weight LLM pairs (base vs. chat models) across seven labs finds that geopolitical bias is introduced during post-training rather than inherited from pre-training data. Six of seven labs showed post-training shifts favoring the developer's home country or region, with Alibaba's Qwen 2.5 showing the most extreme shift (18x increase in China-favourability log-odds). The effect is also language-dependent: Mistral becomes pro-France only under French prompting. The authors argue this implicates alignment and RLHF processes as active shapers of geopolitical perspective, calling for greater transparency and auditing of post-training pipelines.
Cross-lingual evaluation framework reveals LLMs redistribute cultural narrative structure while preserving semantic meaning
A new arXiv preprint introduces a multilingual evaluation framework using 414 proverbs across 15 languages to assess whether LLMs preserve culturally grounded meaning when generating narratives. Using four LLMs to produce 13k narratives, the study finds that cross-lingual prompting preserves proverb-level semantic meaning but systematically redistributes agency, social positioning, and narrative structure. Strong inter-model convergence across architectures suggests multilingual LLMs rely on shared semantic abstractions. The authors argue that semantic similarity metrics alone overestimate cultural preservation in multilingual evaluations.
LLMs fail to consistently simulate demographic perspective-taking in hate speech annotation
A new arXiv paper evaluates whether persona-conditioned LLMs can replicate how different demographic groups perceive hate speech, testing three dimensions: inter-group disagreement, in-group sensitivity, and vicarious prediction. No model consistently captures all three dimensions, and performance is highly model-dependent rather than emerging reliably from identity prompts alone. Vicarious prompting with Llama 3.1 provides the closest approximation to human disagreement patterns across demographic axes. The findings have implications for using LLMs as proxies for diverse human annotators in content moderation tasks.
Mechanistic interpretability study reveals geometric structure of LLM-as-judge bias in hidden states
A new arXiv preprint investigates LLM-as-judge scoring bias at the representation level rather than the input-output level, studying seven judge models across seven bias types and nine benchmarks. The authors find that biased inputs are displaced along low-dimensional, type-specific subspaces in activation space, and that steering hidden states along these subspaces causally controls scoring direction. A linear projection onto bias-direction features predicts judge failures on unseen benchmarks, substantially outperforming text-based alternatives. The work provides a mechanistic account that unifies geometric structure, causal control, and operational prediction within a single framework.
DiaLLM: Controlled study reveals dissociation between dialectal robustness and generation in LLMs
Researchers introduce DiaLLM, a framework that continually pretrains three open-weight LLM families on the International Corpus of English to study dialect adaptation across Australian, Indian, and Northern British English. The study finds that dialectal robustness (understanding) and generation are dissociated: benchmarks are shaped by pretraining and SFT, while alignment reshapes generation in ways benchmarks fail to capture. A key finding is that the alignment method most aggressively optimizing dialectal reward is not preferred by human evaluators, revealing a reward-quality gap. Code, checkpoints, and preference datasets are released.