3arXiv cs.CL (Computation and Language)·
Poller: LLM-as-evaluator method for Chinese poetry understanding tasks
Researchers propose Poller (Poetry LLM Evaluator), a method that prompts LLMs to adopt a poem author's perspective to evaluate poetry understanding tasks, targeting the gap left by traditional automatic metrics on modern Chinese poetry. The approach is tested across eight evaluation dimensions and achieves large error reductions versus baseline LLM evaluation methods, including 94.55% and 89.53% error reduction for rhetorical techniques and defamiliarization respectively. The work contributes to the LLM-as-judge evaluation paradigm applied to a specialized literary domain.
Related guides (1)
Related events (8)
PoetryQwen: LoRA-fine-tuned Qwen2.5-14B for classical Chinese poetry understanding with new 49K dataset
Researchers introduce CCPoetry-49K, a 49,404-pair instruction dataset for classical Chinese poetry appreciation, decomposed into term interpretation, semantic interpretation, and emotional inference subtasks. They fine-tune Qwen2.5-14B using LoRA to produce PoetryQwen, achieving a 9.7% improvement over the baseline on the CCL25-Eval Task 5 benchmark (0.757 vs 0.690). The work addresses a gap in domain-specific LLM adaptation for classical Chinese literary tasks.
Cross-lingual evaluation framework reveals LLMs redistribute cultural narrative structure while preserving semantic meaning
A new arXiv preprint introduces a multilingual evaluation framework using 414 proverbs across 15 languages to assess whether LLMs preserve culturally grounded meaning when generating narratives. Using four LLMs to produce 13k narratives, the study finds that cross-lingual prompting preserves proverb-level semantic meaning but systematically redistributes agency, social positioning, and narrative structure. Strong inter-model convergence across architectures suggests multilingual LLMs rely on shared semantic abstractions. The authors argue that semantic similarity metrics alone overestimate cultural preservation in multilingual evaluations.
Survey and recommendations for LLM-as-a-Judge evaluation in multilingual and low-resource language settings
A new arXiv paper surveys 650 ACL Anthology papers that use LLM-as-a-Judge evaluation, finding only 33 address multilingual or low-resource language settings. Analysis of those 33 papers reveals inconsistent outcomes, overtrust in LLM judgments, and over-reliance on single judge models. The authors provide recommendations for improving evaluation practice in these underserved settings.
Image-Semantic Guided Detection of AI-Generated Modern Chinese Poetry Using MLLMs
This paper proposes a multimodal detection method for identifying AI-generated modern Chinese poetry by incorporating images that reflect poetic content alongside text. The approach uses example-driven prompting to integrate meaning, imagery, and emotional cues from images as a complement to textual analysis. A Gemini-based detector using this method achieves 85.65% Macro-F1, outperforming both plain-text LLM baselines and the traditional RoBERTa detector. The work extends AI-generated content detection research into a domain—modern Chinese poetry—previously unaddressed by prior studies.
Phun-Bench: A Chinese benchmark for evaluating LLM phonological understanding
Researchers introduce Phun-Bench, a purpose-built benchmark for evaluating LLMs on phonological understanding in Chinese across three dimensions: Homophony, Rhyme, and Phonetic Similarity. The benchmark is designed to avoid rote-memorization shortcuts that plague existing phonological evals. Results show LLMs can recall correct pronunciations but fail to apply phonological knowledge flexibly as human speakers do, and the authors propose a hypothesis about the underlying mechanism of LLM phonological 'perception'.
LLMs judge worse than they generate: empirical challenge to self-evaluation pipeline assumptions
A new arXiv preprint tests the implicit assumption that LLM evaluation is easier than generation, using a controlled in-context QA setup across four benchmarks (SQuAD 2.0, DROP, HotpotQA, MuSiQue) and two models. Results show generation accuracy exceeds self-evaluation accuracy on three of four benchmarks, with attention analysis revealing that evaluation attends to context 3–5x less than generation does. LoRA fine-tuning experiments confirm the asymmetry is not a training artifact, with cross-task interference observed in both directions. The findings directly challenge assumptions underlying LLM-as-a-Judge and self-evaluation pipelines widely used in RLHF and agentic systems.
Judge Arena: Benchmarking LLMs as Evaluators
Hugging Face and Atla have launched Judge Arena, a platform for benchmarking large language models in their role as automated evaluators. The initiative uses an Elo-based ranking system to compare how well different LLMs judge the quality of model outputs, addressing the growing reliance on LLM-as-judge paradigms in evaluation pipelines. This fills a meta-evaluation gap: as LLM judges become standard practice, understanding their relative reliability and biases becomes critical infrastructure for the field.
Systematic study of LLM machine translation failures on culturally loaded Chinese-Japanese text using Dream of the Red Chamber
Researchers construct a 500-segment Chinese-Japanese bilingual dataset from Dream of the Red Chamber to probe LLM-based machine translation on culturally loaded content. The study identifies three failure modes: frontier LLMs show notable performance gaps on cultural expressions, human evaluators disagree substantially based on background, and standard automatic metrics fail to reliably assess quality in this domain. The work highlights a systematic blind spot in current MT evaluation and model capability.