4arXiv cs.CL (Computation and Language)·
LKValues: First benchmark and instruction corpus for Sri Lankan societal value alignment in LLMs
Researchers introduce LKValues, a resource suite for aligning LLMs with Sri Lankan cultural values, derived from a trilingual survey of 205 respondents. The suite includes LKvaluesIT, a 150k-instance Sinhala-English instruction corpus, and LKvaluesBench, a 1,000-instance evaluation benchmark. Fine-tuning experiments on Qwen and Aya-Expanse models show that current LLMs exhibit cultural and low-resource alignment gaps, and that LKValues fine-tuning reduces invalid outputs and cross-lingual disparities. The work offers a replicable pipeline for country-specific pluralist value alignment in underrepresented languages.
Related guides (2)
Related events (8)
Pancasila-Dilemmas: Benchmark for LLM Value Alignment on Indonesian Cultural Values
Researchers introduce Pancasila-Dilemmas, a 1,834-question evaluation dataset derived from Indonesian news and grounded in the five values of Pancasila (Religion, Humanity, Unity, Democracy, Social Justice). The benchmark is designed to measure LLM value alignment for Indonesian deployment contexts, with human annotations from five diverse Indonesian citizens. Evaluation of 50 open and closed-source LLMs reveals all models score below 0.5 Probability Match Score, with particular struggles in Religion and Unity dilemma cases, highlighting a significant gap in non-Western value alignment.
Schwartz-based study reveals systematic value-recognition confusions in LLMs
A new arXiv paper evaluates 21 instruction-tuned LLMs on their ability to identify which of Schwartz's ten basic human values is expressed in 1,000 Russian situational texts. Models achieve pooled Acc@1 of 0.683 and Acc@3 of 0.892, but exhibit systematic directional confusions — e.g., Universalism→Benevolence and Tradition→Conformity — that are consistent across checkpoints. The findings challenge the validity of value-alignment evaluations that assume models can reliably recognize values in context, and propose a richer evaluation protocol combining exact accuracy, ranked recovery, and directed error analysis.
Calibrated LLM annotation and encoder transfer for measuring human values in social media text
A new arXiv preprint investigates how different LLMs, prompts, and instruction languages operationalize Schwartz's theory of basic human values when annotating non-English social media posts. The authors evaluate annotation quality beyond standard F1 metrics, examining structural alignment, error structure, and confidence-ambiguity relations, finding that iterative prompt calibration reduces misattributions. They also demonstrate that LLM annotations can be transferred to a smaller encoder model via soft-label training, preserving theory-grounded value interpretations and uncertainty information.
Cross-lingual evaluation framework reveals LLMs redistribute cultural narrative structure while preserving semantic meaning
A new arXiv preprint introduces a multilingual evaluation framework using 414 proverbs across 15 languages to assess whether LLMs preserve culturally grounded meaning when generating narratives. Using four LLMs to produce 13k narratives, the study finds that cross-lingual prompting preserves proverb-level semantic meaning but systematically redistributes agency, social positioning, and narrative structure. Strong inter-model convergence across architectures suggests multilingual LLMs rely on shared semantic abstractions. The authors argue that semantic similarity metrics alone overestimate cultural preservation in multilingual evaluations.
Introducing the Open Ko-LLM Leaderboard: Leading the Korean LLM Evaluation Ecosystem
Upstage and Hugging Face have launched the Open Ko-LLM Leaderboard, a public benchmark platform for evaluating large language models specifically on Korean language tasks. The leaderboard aims to standardize Korean LLM evaluation and foster competition among models targeting the Korean-language market. This initiative extends the Open LLM Leaderboard framework to a non-English language context, reflecting growing interest in multilingual and language-specific model evaluation.
C4STYLI Benchmark: Probing Cultural Aesthetic Stylistics Awareness in LLMs
Researchers introduce C4STYLI, a benchmark of stylized translated movie titles and advertising slogans from Hong Kong and mainland China, designed to evaluate LLMs on cross-cultural aesthetic stylistics. Evaluations reveal that LLMs diverge from human stylistic recognition, with recognition ability varying by text domain and not consistently predicting generation performance. Structural ablation using logistic regression probes shows that LLMs in the Hong Kong setting rely on surface-level linguistic cues rather than deeper stylistic structure, indicating limited cultural sensitivity.
Survey and recommendations for LLM-as-a-Judge evaluation in multilingual and low-resource language settings
A new arXiv paper surveys 650 ACL Anthology papers that use LLM-as-a-Judge evaluation, finding only 33 address multilingual or low-resource language settings. Analysis of those 33 papers reveals inconsistent outcomes, overtrust in LLM judgments, and over-reliance on single judge models. The authors provide recommendations for improving evaluation practice in these underserved settings.
FilBench: Benchmarking LLM Capabilities in Filipino Language
FilBench is a new benchmark introduced to evaluate large language models on their ability to understand and generate Filipino. The benchmark targets a historically underrepresented language in NLP evaluation suites, assessing both comprehension and generation tasks. This work addresses gaps in multilingual LLM evaluation coverage, particularly for Southeast Asian languages.