4arXiv cs.CL (Computation and Language)·
Systematic study of LLM machine translation failures on culturally loaded Chinese-Japanese text using Dream of the Red Chamber
Researchers construct a 500-segment Chinese-Japanese bilingual dataset from Dream of the Red Chamber to probe LLM-based machine translation on culturally loaded content. The study identifies three failure modes: frontier LLMs show notable performance gaps on cultural expressions, human evaluators disagree substantially based on background, and standard automatic metrics fail to reliably assess quality in this domain. The work highlights a systematic blind spot in current MT evaluation and model capability.
Related guides (1)
Related events (8)
Towards Reliable Multilingual LLMs-as-a-Judge: An Empirical Study
This paper systematically investigates strategies for extending LLM-based automatic evaluation (LLMs-as-a-Judge) to multilingual settings, covering high-, mid-, and low-resource languages (English, Spanish, Basque). The authors compare instruction translation, monolingual vs. multilingual supervision, and model size, finding that fine-tuned smaller models can match proprietary models when in-domain data is available, while zero-shot larger models are preferable out-of-domain. Two meta-evaluation datasets are extended to Spanish and Basque, and all data and code are publicly released.
Benchmarking Local LLMs for Confidential Translation Workflows
This paper evaluates locally runnable LLMs (via Ollama) for offline, privacy-constrained translation workflows targeting freelance translators and smaller language service providers. The authors expand their Reeve Foundation corpus to include German and Simplified Chinese, then benchmark local models across four language directions against commercial NMTs (DeepL, Baidu), a frontier LLM (GPT-5.2), and professional local NMT systems. Results show substantial performance variation by language direction and model size, with the best local LLMs matching or exceeding local NMT systems and the frontier LLM, though falling short of top commercial NMTs. The study supports the viability of local LLMs for confidentiality-sensitive translation use cases.
First Komi-Yazva–Russian parallel corpus and LLM translation evaluation protocol for endangered low-resource language
Researchers introduce the first Komi-Yazva–Russian parallel corpus of 457 aligned sentence pairs from 74 narrative texts, paired with a rigorous evaluation protocol for studying LLM translation under extreme data scarcity. The protocol includes story-level cross-validation, deterministic retrieval-based few-shot prompting, and both reference-based and judge-based metrics to ensure leakage-aware, reproducible evaluation. Results show LLMs produce non-trivial translations but performance varies strongly by model family; retrieval-based few-shot prompting consistently outperforms zero-shot, though gains plateau quickly. The work frames the corpus as both a dataset contribution and a reproducible testbed for endangered-language machine translation research.
Study finds readers prefer human literary translations over LLM-based MT, but cannot reliably distinguish them
A new arXiv paper presents a reader-centered evaluation of AI vs. human literary translation across 15 novels in French, Polish, and Japanese translated into English. Fifteen avid readers compared human translations (HT) to machine translations (MT) from an agentic LLM pipeline, finding MT 'fine' but preferring HT for ease, clarity, and immersiveness—especially at the chunk level (522/772 preferences). Critically, readers could not reliably identify which version was human-produced (17/30 correct), and automatic metrics including LLM-as-a-judge consistently favored MT over HT, diverging from human preference. The authors release LAIT, a dataset with 1K reader comments, 2K judgments, and 7.2K span-level annotations.
Systematic evaluation of LLMs and NMT systems for English-Tamil translation across multiple datasets
A preprint from arXiv evaluates multiple multilingual translation models—NLLB, mBART, and TamilLLaMA—on English-Tamil and Tamil-English translation across four datasets (NTREX, EnTamV2, WikiMatrix, PMIndia) using BLEU and chrF metrics. The study includes attention-based interpretability analysis and few-shot prompting experiments with TamilLLaMA. Key findings indicate dataset quality and domain alignment strongly affect performance, and that few-shot LLM approaches can produce structurally coherent Tamil translations despite limited supervised data.
Cross-lingual evaluation framework reveals LLMs redistribute cultural narrative structure while preserving semantic meaning
A new arXiv preprint introduces a multilingual evaluation framework using 414 proverbs across 15 languages to assess whether LLMs preserve culturally grounded meaning when generating narratives. Using four LLMs to produce 13k narratives, the study finds that cross-lingual prompting preserves proverb-level semantic meaning but systematically redistributes agency, social positioning, and narrative structure. Strong inter-model convergence across architectures suggests multilingual LLMs rely on shared semantic abstractions. The authors argue that semantic similarity metrics alone overestimate cultural preservation in multilingual evaluations.
C4STYLI Benchmark: Probing Cultural Aesthetic Stylistics Awareness in LLMs
Researchers introduce C4STYLI, a benchmark of stylized translated movie titles and advertising slogans from Hong Kong and mainland China, designed to evaluate LLMs on cross-cultural aesthetic stylistics. Evaluations reveal that LLMs diverge from human stylistic recognition, with recognition ability varying by text domain and not consistently predicting generation performance. Structural ablation using logistic regression probes shows that LLMs in the Hong Kong setting rely on surface-level linguistic cues rather than deeper stylistic structure, indicating limited cultural sensitivity.
Survey and recommendations for LLM-as-a-Judge evaluation in multilingual and low-resource language settings
A new arXiv paper surveys 650 ACL Anthology papers that use LLM-as-a-Judge evaluation, finding only 33 address multilingual or low-resource language settings. Analysis of those 33 papers reveals inconsistent outcomes, overtrust in LLM judgments, and over-reliance on single judge models. The authors provide recommendations for improving evaluation practice in these underserved settings.