4arXiv cs.CL (Computation and Language)·
CKTN: First multilingual corpus and benchmark for Vietnamese ethnic minority languages Cham, Khmer, and Tay-Nung
Researchers introduce CKTN, the first NLP corpus and benchmark for three underrepresented Vietnamese ethnic minority languages (Cham, Khmer, Tay-Nung), comprising 44,367 documents and 24M subword tokens across pretraining, classification, and retrieval tasks. The paper demonstrates that standard multilingual encoders severely fragment these languages and that common adaptation metrics (language-modeling loss, lexical-overlap retrieval) can be misleading proxies for semantic generalization. The authors propose a script-aware adaptation recipe combining vocabulary augmentation with calibrated replaced-token pretraining, achieving substantially reduced fragmentation and best-in-class classification performance. The work highlights systematic gaps in multilingual NLP coverage and evaluation methodology for low-resource, script-diverse languages.
Related guides (1)
Related events (8)
CN-NewsTTS Bench: automatic benchmark for Chinese news TTS pronunciation of complex written forms
Researchers introduce CN-NewsTTS Bench v0.1, an open benchmark for evaluating Chinese news text-to-speech systems on challenging written forms such as scores, abbreviations, unit symbols, and mixed-script names — all from raw text without preprocessing aids. The benchmark includes a 200-record dev set, 800-record public test set, an automatic scorer, and baseline results for seven commercial TTS systems. Best-in-class accuracy reaches 0.879 strict accuracy while several systems fall below 0.60, revealing meaningful performance gaps on a practically important but underexplored evaluation dimension.
BamiBERT: New BERT-based language model sets state of the art for Vietnamese NLP
Qualcomm AI Research introduces BamiBERT, a BERT-based encoder pre-trained from scratch on 129GB of Vietnamese text for 20 epochs, supporting up to 2048-token context without requiring external word segmentation. It outperforms PhoBERT, the previous de facto Vietnamese encoder, achieving best scores on 11 of 15 metrics across 8 Vietnamese benchmarks. The model is released publicly on Hugging Face.
SkMTEB: First comprehensive MTEB-style text embedding benchmark for Slovak with adapted E5 models
Researchers introduce SkMTEB, the first MTEB-style embedding benchmark for Slovak, covering 31 datasets across 7 task types — roughly 4× the existing multilingual benchmark coverage for the language. Evaluation of 31 embedding models shows large instruction-tuned multilingual models outperform Slovak-specific NLU models on embedding tasks. The authors also release e5-sk-small (45M) and e5-sk-large (365M), derived from Multilingual E5 via vocabulary trimming and fine-tuning, achieving competitive performance with proprietary APIs at up to 62% size reduction.
First Komi-Yazva–Russian parallel corpus and LLM translation evaluation protocol for endangered low-resource language
Researchers introduce the first Komi-Yazva–Russian parallel corpus of 457 aligned sentence pairs from 74 narrative texts, paired with a rigorous evaluation protocol for studying LLM translation under extreme data scarcity. The protocol includes story-level cross-validation, deterministic retrieval-based few-shot prompting, and both reference-based and judge-based metrics to ensure leakage-aware, reproducible evaluation. Results show LLMs produce non-trivial translations but performance varies strongly by model family; retrieval-based few-shot prompting consistently outperforms zero-shot, though gains plateau quickly. The work frames the corpus as both a dataset contribution and a reproducible testbed for endangered-language machine translation research.
CzechDocs: Multiway parallel dataset for format-preserving machine translation of minority languages
CzechDocs is a new multiway parallel dataset of formatted documents (HTML, DOCX, PDF) covering Czech, Ukrainian, English, Vietnamese, Russian, and other minority languages used in Czechia. The dataset is designed to evaluate machine translation systems that preserve document formatting during translation. A validation split and evaluation toolkit are publicly released; a held-out test split is reserved for a future shared task.
Benchmarking Local LLMs for Confidential Translation Workflows
This paper evaluates locally runnable LLMs (via Ollama) for offline, privacy-constrained translation workflows targeting freelance translators and smaller language service providers. The authors expand their Reeve Foundation corpus to include German and Simplified Chinese, then benchmark local models across four language directions against commercial NMTs (DeepL, Baidu), a frontier LLM (GPT-5.2), and professional local NMT systems. Results show substantial performance variation by language direction and model size, with the best local LLMs matching or exceeding local NMT systems and the frontier LLM, though falling short of top commercial NMTs. The study supports the viability of local LLMs for confidentiality-sensitive translation use cases.
Forgotten Words: Benchmarking NeoBERT for Dementia Detection in Low-Resource Conversational Filipino and English Speech
This paper presents the first NLP-based dementia detection study for Filipino speech, constructing a parallel bilingual dataset of 4,000 DementiaBank-derived transcripts with manual Filipino translations. Five model families are evaluated across monolingual, zero-shot cross-lingual, and bilingual fine-tuning settings. English-trained BERT degrades sharply on Filipino (Macro-F1 = 0.455), but bilingual fine-tuning recovers performance to Macro-F1 = 0.969–0.973 across all transformer models. The key finding is that multilingual clinical NLP performance is driven by linguistic coverage during training rather than model scale or architecture.
Pluralis v0.1: Multicultural, multimodal, multilingual benchmark for AI safety across Asia-Pacific
Researchers introduce Pluralis v0.1, a 6,448-prompt multimodal benchmark spanning six Asia-Pacific countries and eight languages, designed to evaluate Vision-Language Models on culturally localized safety hazards rather than Western-adapted defaults. The benchmark introduces a novel evaluation paradigm where text and image are individually innocuous but synergistically trigger cultural or legal violations. The authors also present Judge-Pluralis, an LLM-as-a-Judge ensemble for scoring, and document recurring locale-specific failure modes in current VLMs including image misidentification and missed item-context-locale interactions.