3arXiv cs.CL (Computation and Language)·
Systematic evaluation of LLMs and NMT systems for English-Tamil translation across multiple datasets
A preprint from arXiv evaluates multiple multilingual translation models—NLLB, mBART, and TamilLLaMA—on English-Tamil and Tamil-English translation across four datasets (NTREX, EnTamV2, WikiMatrix, PMIndia) using BLEU and chrF metrics. The study includes attention-based interpretability analysis and few-shot prompting experiments with TamilLLaMA. Key findings indicate dataset quality and domain alignment strongly affect performance, and that few-shot LLM approaches can produce structurally coherent Tamil translations despite limited supervised data.
Related guides (1)
Related events (8)
Towards Reliable Multilingual LLMs-as-a-Judge: An Empirical Study
This paper systematically investigates strategies for extending LLM-based automatic evaluation (LLMs-as-a-Judge) to multilingual settings, covering high-, mid-, and low-resource languages (English, Spanish, Basque). The authors compare instruction translation, monolingual vs. multilingual supervision, and model size, finding that fine-tuned smaller models can match proprietary models when in-domain data is available, while zero-shot larger models are preferable out-of-domain. Two meta-evaluation datasets are extended to Spanish and Basque, and all data and code are publicly released.
Benchmarking Local LLMs for Confidential Translation Workflows
This paper evaluates locally runnable LLMs (via Ollama) for offline, privacy-constrained translation workflows targeting freelance translators and smaller language service providers. The authors expand their Reeve Foundation corpus to include German and Simplified Chinese, then benchmark local models across four language directions against commercial NMTs (DeepL, Baidu), a frontier LLM (GPT-5.2), and professional local NMT systems. Results show substantial performance variation by language direction and model size, with the best local LLMs matching or exceeding local NMT systems and the frontier LLM, though falling short of top commercial NMTs. The study supports the viability of local LLMs for confidentiality-sensitive translation use cases.
Systematic study of LLM machine translation failures on culturally loaded Chinese-Japanese text using Dream of the Red Chamber
Researchers construct a 500-segment Chinese-Japanese bilingual dataset from Dream of the Red Chamber to probe LLM-based machine translation on culturally loaded content. The study identifies three failure modes: frontier LLMs show notable performance gaps on cultural expressions, human evaluators disagree substantially based on background, and standard automatic metrics fail to reliably assess quality in this domain. The work highlights a systematic blind spot in current MT evaluation and model capability.
LLM pipeline extracts grammar books to generate synthetic MT data for endangered languages
Researchers introduce a pipeline that uses LLMs to extract grammatical rules, example sentences, and lexicons from descriptive grammar books, then generates synthetic parallel corpora for fine-tuning machine translation models. Validated on three typologically diverse low-resource languages (Kalamang, Tuatschin, Mandan), fine-tuning on synthetic data outperforms seed-data baselines in 59–75% of configurations, with best-case ChrF++ gains up to +8.8. A systematic factorial study across 96 configurations identifies which combinations of target part-of-speech, retrieval granularity, and sample volume drive improvements.
First Komi-Yazva–Russian parallel corpus and LLM translation evaluation protocol for endangered low-resource language
Researchers introduce the first Komi-Yazva–Russian parallel corpus of 457 aligned sentence pairs from 74 narrative texts, paired with a rigorous evaluation protocol for studying LLM translation under extreme data scarcity. The protocol includes story-level cross-validation, deterministic retrieval-based few-shot prompting, and both reference-based and judge-based metrics to ensure leakage-aware, reproducible evaluation. Results show LLMs produce non-trivial translations but performance varies strongly by model family; retrieval-based few-shot prompting consistently outperforms zero-shot, though gains plateau quickly. The work frames the corpus as both a dataset contribution and a reproducible testbed for endangered-language machine translation research.
Translation-based fine-tuning of English BERT compared to native-language models across six NLP tasks and five languages
A new arXiv preprint systematically compares translation-based fine-tuning of English BERT against native-language BERT models across six NLP tasks using datasets from Bulgarian, Chinese, Dutch, Italian, and Russian. The translation approach was comparable or superior in 53.3% of cases, with strongest gains in Question Answering, POS Tagging, and NLI, but weaker performance on Named Entity Recognition and Hate Speech Detection. Results suggest translation-based fine-tuning is most effective for syntactic tasks and typologically English-close languages like Dutch, offering a resource-efficient path for low-resource NLP.
Cross-lingual evaluation framework reveals LLMs redistribute cultural narrative structure while preserving semantic meaning
A new arXiv preprint introduces a multilingual evaluation framework using 414 proverbs across 15 languages to assess whether LLMs preserve culturally grounded meaning when generating narratives. Using four LLMs to produce 13k narratives, the study finds that cross-lingual prompting preserves proverb-level semantic meaning but systematically redistributes agency, social positioning, and narrative structure. Strong inter-model convergence across architectures suggests multilingual LLMs rely on shared semantic abstractions. The authors argue that semantic similarity metrics alone overestimate cultural preservation in multilingual evaluations.
Text Analytics Evaluation Framework: Benchmarking LLMs on Social Media NLP Tasks
Researchers introduce a 470-question evaluation framework to assess LLM performance on aggregated social media text, applied to Twitter datasets across sentiment analysis, hate speech detection, and emotion recognition. Results show performance degrades substantially as input scale exceeds 500 instances, particularly for open-weights models on numerical tasks. Multi-label and target-dependent scenarios also show notable performance drops, and task complexity progressively erodes accuracy from basic semantic identification to comparison and counting operations. The findings point to architectural bottlenecks in current LLMs for rigorous quantitative analysis over large text collections.