6arXiv cs.CL (Computation and Language)·
Polistemics benchmark evaluates LLMs as political information mediators in elections
Researchers introduce Polistemics, a theory-grounded benchmark for evaluating how LLMs mediate political information during elections, grounded in a normative standard called Epistemic Modesty. Applied to three state-of-the-art LLMs on the 2025 German and Dutch elections, the benchmark reveals that high aggregate scores mask systematic failures: models perform well under clear evidence but break down under absent, vague, or contradictory information, and flatten political language intensity. The failures appear driven by party priors and are sensitive to party labels and output language, suggesting no current model delivers consistently responsible political mediation.
Related guides (2)
Related events (8)
Political Consistency Training: Reducing Covert Political Bias in LLMs via RL
Researchers identify a phenomenon called 'covert political bias' in LLMs, where models handle politically paired topics asymmetrically across 7 identified technique categories. They propose two metrics—Sentiment Consistency and Helpfulness Consistency—to measure this asymmetry. To address it, they introduce Political Consistency Training (PCT), an RL-based method with complementary training paradigms that reduces covert bias while preserving overall helpfulness and generalizing to held-out benchmarks.
Digital Pantheon: Multi-agent LLM framework simulates and audits political coalition formation
Researchers present a multi-agent framework using SFT, DPO, and RAG to simulate partisan political negotiation with LLM agents, applied to the 2019 Flemish regional election. DPO is used to overcome RLHF neutrality biases and instill stable party-specific personas, while per-party RAG pipelines ground each agent in official party manifestos. The framework introduces MILT (Multi-Layered Information Lineage Topology) and a Coalition Influence Score to trace negotiation outcomes back to source documents and benchmark them against the historically adopted coalition agreement. Across three simulations, the framework produces stable rankings and finds that manifesto-anchored content reliably predicts real-world policy materialization while hallucinated content does not.
Benchmark and study of LLM agents in deliberative multi-agent collaboration under partial observability
Researchers introduce a scalable benchmark for evaluating LLM agents on cooperative joint decision-making tasks where agents must exchange information under partial and asymmetric observations to reach a shared decision. A systematic evaluation of representative LLMs finds that state-of-the-art models still struggle with complex deliberative collaboration, failing in either information alignment or downstream reasoning even with external mathematical tools. Diagnostic analysis also reveals that deliberation can enable reflection and error correction, sometimes outperforming centralized baselines, offering a nuanced picture of multi-agent LLM capabilities.
Paper challenges LLM expert-level claims by measuring variance and error magnitude in code-based data analysis tasks
A new arXiv paper argues that standard LLM benchmarks overstate model capabilities by focusing on average performance on training-data-adjacent tasks while ignoring response variance and error magnitude. The authors introduce a novel benchmark requiring frontier LLMs to write code for data analysis tasks, comparing results against human expert submissions. Human experts outperformed the frontier LLM on average across multiple metrics and showed lower performance variability. The findings challenge the prevailing narrative that LLMs perform at human-expert level on knowledge economy tasks.
Defining and Evaluating Political Bias in LLMs
OpenAI has published a post describing their methodology for evaluating political bias in ChatGPT, introducing new real-world testing approaches aimed at improving objectivity and reducing bias. The piece outlines how OpenAI defines political bias in the context of large language models and the evaluation frameworks they are developing to measure it. This represents OpenAI's public commitment to systematic bias measurement as a component of responsible deployment.
Cross-lingual evaluation framework reveals LLMs redistribute cultural narrative structure while preserving semantic meaning
A new arXiv preprint introduces a multilingual evaluation framework using 414 proverbs across 15 languages to assess whether LLMs preserve culturally grounded meaning when generating narratives. Using four LLMs to produce 13k narratives, the study finds that cross-lingual prompting preserves proverb-level semantic meaning but systematically redistributes agency, social positioning, and narrative structure. Strong inter-model convergence across architectures suggests multilingual LLMs rely on shared semantic abstractions. The authors argue that semantic similarity metrics alone overestimate cultural preservation in multilingual evaluations.
LLM psychological profiles are largely measurement artifacts, not model properties
A new arXiv preprint administers a battery of personality and risk-preference instruments to 56 instruction-tuned LLMs alongside large human reference samples, finding that 81-90% of between-model variation is explained by directional response bias rather than the traits the instruments target. The authors introduce the concept of 'response orthogonality' to explain why some instruments appear more reliable than others, and show that apparent psychological profiles can be manufactured through item selection. The findings challenge the validity of using human-designed psychometric tools to characterize LLMs, with direct implications for safety assessment and the use of LLMs as proxies for human participants in research.
ESFP benchmark measures LLM epistemic stance flexibility across attribution conditions
Researchers introduce ESFP (Epistemic Stance Flexibility Probing), a 104-item behavioral benchmark evaluating whether LLMs appropriately shift between externally attributed and self-attributed epistemic registers when prompted differently. The benchmark spans six epistemic categories and five phrasing templates, assessing models on lexical self-attribution, representation-level responsiveness, stance content density, and cross-condition consistency. Evaluating eight frontier models from five vendors, the authors find epistemic flexibility is largely orthogonal to general capability, with a 27B open-weight model matching top proprietary systems and reasoning-optimized models showing no consistent advantage. Notably, surface-level lexical markers like 'I think' can shift without corresponding changes in actual expressed stance.