4arXiv cs.CL (Computation and Language)·
Pancasila-Dilemmas: Benchmark for LLM Value Alignment on Indonesian Cultural Values
Researchers introduce Pancasila-Dilemmas, a 1,834-question evaluation dataset derived from Indonesian news and grounded in the five values of Pancasila (Religion, Humanity, Unity, Democracy, Social Justice). The benchmark is designed to measure LLM value alignment for Indonesian deployment contexts, with human annotations from five diverse Indonesian citizens. Evaluation of 50 open and closed-source LLMs reveals all models score below 0.5 Probability Match Score, with particular struggles in Religion and Unity dilemma cases, highlighting a significant gap in non-Western value alignment.
Related guides (2)
Related events (8)
LKValues: First benchmark and instruction corpus for Sri Lankan societal value alignment in LLMs
Researchers introduce LKValues, a resource suite for aligning LLMs with Sri Lankan cultural values, derived from a trilingual survey of 205 respondents. The suite includes LKvaluesIT, a 150k-instance Sinhala-English instruction corpus, and LKvaluesBench, a 1,000-instance evaluation benchmark. Fine-tuning experiments on Qwen and Aya-Expanse models show that current LLMs exhibit cultural and low-resource alignment gaps, and that LKValues fine-tuning reduces invalid outputs and cross-lingual disparities. The work offers a replicable pipeline for country-specific pluralist value alignment in underrepresented languages.
Schwartz-based study reveals systematic value-recognition confusions in LLMs
A new arXiv paper evaluates 21 instruction-tuned LLMs on their ability to identify which of Schwartz's ten basic human values is expressed in 1,000 Russian situational texts. Models achieve pooled Acc@1 of 0.683 and Acc@3 of 0.892, but exhibit systematic directional confusions — e.g., Universalism→Benevolence and Tradition→Conformity — that are consistent across checkpoints. The findings challenge the validity of value-alignment evaluations that assume models can reliably recognize values in context, and propose a richer evaluation protocol combining exact accuracy, ranked recovery, and directed error analysis.
Auditing Value Pluralism in Clinical Ethics of Large Language Models
Researchers present a framework for auditing ethical value pluralism in medical AI, comprising a benchmark of clinician-verified dilemmas and an attribution method that recovers value priorities from model decisions. While frontier LLMs span physician-level value heterogeneity in aggregate and discuss competing values in reasoning, individual model decisions are near-deterministic and fail to reproduce the distributional pluralism of physician panels. Some models systematically underweight patient autonomy. The authors warn that deploying a single LLM at scale risks replacing clinical pluralism with a 'deployment monoculture.'
Cross-lingual evaluation framework reveals LLMs redistribute cultural narrative structure while preserving semantic meaning
A new arXiv preprint introduces a multilingual evaluation framework using 414 proverbs across 15 languages to assess whether LLMs preserve culturally grounded meaning when generating narratives. Using four LLMs to produce 13k narratives, the study finds that cross-lingual prompting preserves proverb-level semantic meaning but systematically redistributes agency, social positioning, and narrative structure. Strong inter-model convergence across architectures suggests multilingual LLMs rely on shared semantic abstractions. The authors argue that semantic similarity metrics alone overestimate cultural preservation in multilingual evaluations.
Survey and recommendations for LLM-as-a-Judge evaluation in multilingual and low-resource language settings
A new arXiv paper surveys 650 ACL Anthology papers that use LLM-as-a-Judge evaluation, finding only 33 address multilingual or low-resource language settings. Analysis of those 33 papers reveals inconsistent outcomes, overtrust in LLM judgments, and over-reliance on single judge models. The authors provide recommendations for improving evaluation practice in these underserved settings.
FilBench: Benchmarking LLM Capabilities in Filipino Language
FilBench is a new benchmark introduced to evaluate large language models on their ability to understand and generate Filipino. The benchmark targets a historically underrepresented language in NLP evaluation suites, assessing both comprehension and generation tasks. This work addresses gaps in multilingual LLM evaluation coverage, particularly for Southeast Asian languages.
Moral Semantics Survive Machine Translation: Cross-Lingual Evidence from Moral Foundations Corpora
This paper investigates whether LLM-based machine translation can preserve moral semantic content well enough to enable cross-lingual moral values classification, using Polish as a test case with ~50k annotated social media posts. A four-method validation pipeline (LaBSE embedding similarity, CKA, LLM-as-judge, and classifier parity) shows mean cosine similarity of 0.86 and AUC gaps of only 0.01–0.02 across Moral Foundations categories. The results suggest machine translation is a practical path to extending moral values NLP research to under-resourced languages, with expected generalization to related Slavic languages.
Paper challenges LLM expert-level claims by measuring variance and error magnitude in code-based data analysis tasks
A new arXiv paper argues that standard LLM benchmarks overstate model capabilities by focusing on average performance on training-data-adjacent tasks while ignoring response variance and error magnitude. The authors introduce a novel benchmark requiring frontier LLMs to write code for data analysis tasks, comparing results against human expert submissions. Human experts outperformed the frontier LLM on average across multiple metrics and showed lower performance variability. The findings challenge the prevailing narrative that LLMs perform at human-expert level on knowledge economy tasks.