Entity · paper

Challenges and Recommendations for LLMs-as-a-Judge in Multilingual Settings and Low-Resource Languages

paperactiveprovisionalchallenges-and-recommendations-for-llms-as-a-judge-in-multilingual-settings-and-low-resource-languages-ee5b8d5d·1 events·first seen 3d ago

Aliases: Challenges and Recommendations for LLMs-as-a-Judge in Multilingual Settings and Low-Resource Languages

Co-occurring entities

ACL Anthology

More like this (12)

LLM-as-a-Judge LLM-judge scoring Can LLMs Judge Better Than They Generate? Evaluating Task Asymmetry, Mechanistic Interpretability and Transferability for In-Context QA The Masked Advantage: Uncovering Local-Language Access to Cultural Knowledge in LLMs Audio-LLM Arabic LLMs SpeechLLM Trade-offs in Medical LLM Adaptation: An Empirical Study in French QA frontier LLMs Reassessing High-Performing LLMs on Polish Medical Exams: True Competence or Bias-Driven Performance?long-context LLMs Open Leaderboard for Japanese LLMs

Recent events (1)

5arXiv · cs.AI·3d ago·source ↗

Survey and recommendations for LLM-as-a-Judge evaluation in multilingual and low-resource language settings

A new arXiv paper surveys 650 ACL Anthology papers that use LLM-as-a-Judge evaluation, finding only 33 address multilingual or low-resource language settings. Analysis of those 33 papers reveals inconsistent outcomes, overtrust in LLM judgments, and over-reliance on single judge models. The authors provide recommendations for improving evaluation practice in these underserved settings.

Evaluation and Benchmarking Challenges and Recommendations for LLMs-as-a-Judge in Multilingual Settings and Low-Resource Languages ACL Anthology