6arXiv cs.CL (Computation and Language)·
Audit finds Grokipedia less politically neutral than Wikipedia, with distinct ideological biases
A large-scale arXiv study audits political neutrality in Grokipedia—an encyclopedia generated by xAI's Grok LLM—versus Wikipedia, analyzing 1,394 article pairs about government members across nine ideology dimensions using four LLM judges (Grok, Claude, Mistral, DeepSeek). All four judges, including Grok itself, rate Grokipedia as less neutral than Wikipedia. The study finds Grokipedia favors economically right-wing politicians and penalizes socially liberal ones, while Wikipedia shows the opposite bias pattern, raising questions about whether LLM-generated content can deliver ideological neutrality.
Related guides (3)
Related events (8)
Finetuning on narrow datasets causes broad ideological shifts in LLMs, including extremist outputs
A new arXiv paper demonstrates that finetuning LLMs on small, moderation-passing datasets with ideological slant causes broad ideological shifts across unrelated domains — a phenomenon the authors call 'ideological generalisation.' Training GPT-4.1 on economics Q&A with a political lean shifts model outputs on criminal justice, environment, and cultural topics, and can produce out-of-distribution extremist endorsements (e.g., race-IQ connections, political violence) not present in training data. The effect replicates on Gemma-3, survives mixing with generic data, and preserves general capabilities (GSM8K within ±1pp). The finding has significant implications for supply-chain safety of finetuned models deployed via third parties.
Defining and Evaluating Political Bias in LLMs
OpenAI has published a post describing their methodology for evaluating political bias in ChatGPT, introducing new real-world testing approaches aimed at improving objectivity and reducing bias. The piece outlines how OpenAI defines political bias in the context of large language models and the evaluation frameworks they are developing to measure it. This represents OpenAI's public commitment to systematic bias measurement as a component of responsible deployment.
Anthropic Publishes Political Even-Handedness Evaluation for Claude, Open-Sources Methodology
Anthropic has released a detailed account of how it trains and evaluates Claude for political even-handedness, including character traits instilled via reinforcement learning since early 2024 and a new automated evaluation methodology. The evaluation tests thousands of prompts across hundreds of political stances and benchmarks Claude Sonnet 4.5 against GPT-5, Llama 4, Grok 4, and Gemini 2.5 Pro, finding Claude comparable to Grok 4 and Gemini 2.5 Pro and more even-handed than GPT-5 and Llama 4. Anthropic is open-sourcing the evaluation framework to encourage shared industry standards for measuring political bias. The post also discloses the specific system prompt language used on Claude.ai to enforce even-handed behavior.
AI-Mediated Communication Can Steer Collective Opinion via LLM Editing Biases
This paper demonstrates empirically that LLMs from multiple model families introduce directional biases when editing human-written texts on contested topics (e.g., nudging toward gun control, against atheism). The authors develop a mathematical opinion-dynamics model showing these biases are amplified through social networks, shifting collective opinion at scale. An audit of X's 'Explain this post' feature finds evidence of pro-life bias in Grok's outputs on abortion content, traced to specific design choices. The paper concludes with implications for EU legislative efforts on AI-mediated communication.
Study finds state media in training data causes LLMs to reflect government propaganda in native languages
Researchers from University of Oregon, Purdue, UCSD, NYU, and Princeton found that state-controlled media is heavily overrepresented in web-scraped training datasets, causing Claude 3 Sonnet and GPT-4o to express significantly more favorable attitudes toward authoritarian governments when prompted in those governments' native languages. Chinese state media accounts for over 40x more documents in CulturaX than Chinese Wikipedia, and both models reproduced state-media strings at 3-5% rates. When prompted in Chinese, both models favored China's government roughly 68-75% of the time versus English prompts on the same topics, with the effect scaling with a country's World Press Freedom Index ranking.
Study finds RAG transfers ideological bias into LLM outputs, with temperature modulating strength of transfer
A new arXiv paper investigates how ideological content in retrieved documents propagates into LLM-generated answers under a RAG framework. Using Lexical Multidimensional Analysis on 1,117 COVID-19 treatment articles as the retrieval corpus, the authors find that RAG reliably transmits ideological discourse into model outputs, with the effect peaking at moderate sampling temperatures and diminishing at low temperatures where deterministic sampling suppresses discourse transfer. The work highlights an underexplored bias vector in RAG pipelines beyond factual hallucination.
Empirical study finds web search in public AI services trades coverage for source trustworthiness
A pre-launch expert evaluation of Evrópuvefur, a government-funded LLM service answering EU questions for Icelandic citizens, compared curated RAG retrieval against open web search across 551 expert evaluations of 449 AI-generated answers. Web search answered more questions but flagged at least one untrustworthy or irrelevant source in 35% of answers; curated sources were rarely flagged but limited coverage caused the model to decline responses. A prompt ablation showed weak steering: adding a trusted-domain list to the system prompt raised citations to those domains only from 12% to 21%. The paper argues source trustworthiness is a measurable but largely invisible quality dimension in public AI deployments.
Benchmarking LLM judges for citation verification in deep-research systems
A new arXiv paper evaluates 8 LLM judges from 3 model families on citation quality assessment for deep-research systems, testing across 1,248 rubric decisions with human-reviewed gold labels. The study finds that cheaper models remain competitive with frontier models — GPT-5-mini achieves the strongest source-relevance F1 at 0.908 — but judges differ substantially in directional bias (pass-rate drift, false positive/negative rates) even when scalar F1 scores are similar. The key finding is that scalar F1 obscures biases that would be directly reinforced in an RL training loop, making judge calibration a prerequisite before using citation rubrics as reward signals.