5arXiv cs.CL (Computation and Language)·
TriviaRoomQA: Multilingual benchmark reveals LLM gaps in everyday popular-culture knowledge
Researchers introduce TriviaRoomQA, a multilingual benchmark with 3,300 parallel multiple-choice questions across six European languages and 5,340 French-only questions, covering 288 topics drawn from quiz and trivia contexts. Evaluating 30 open-weight LLMs (7B–70B parameters) from European, Asian, and North American providers, the study finds models perform well on history, geography, and math but substantially worse on popular-culture topics like celebrities, music, and movies. Performance also varies across languages for identical underlying questions, indicating factual knowledge access is language-dependent. The work highlights a knowledge gap not captured by existing saturated academic benchmarks.
Related guides (1)
Related events (8)
Study finds local languages provide better cultural knowledge access in LLMs once proficiency is controlled
A new arXiv paper introduces a controlled evaluation framework to disentangle language proficiency from culture-specific knowledge access in LLMs. Using real-world cultural questions across 13 locales and ~80 models, the authors apply item response theory to show that while English dominates on culture-agnostic questions, local languages yield a consistent knowledge-access advantage on culture-specific questions once proficiency differences are factored out. The finding challenges the common interpretation that weaker local-language accuracy implies weaker cultural knowledge, and has implications for how multilingual and regionally-aligned models are evaluated.
LexNeo-Bench: Probing LLM Knowledge of Lexical Borrowing in Luxembourgish via Knowledge-Graph Prompting
Researchers introduce LexNeo-Bench, a 3,050-instance benchmark for evaluating LLM performance on lexical borrowing classification and neology detection in Luxembourgish, a low-resource contact language. Three multilingual LLMs are tested across 34 prompt configurations; without external context, models perform near chance on borrowing classification (25–35%). Injecting instance-specific subgraphs from a linguistic knowledge graph raises accuracy to 71–81% and largely closes the gap between small and large models, though neology detection remains difficult. The study highlights the value of lexicon-aware, structured prompting for low-resource multilingual evaluation.
New Polish medical exam benchmark reveals MCQA overestimates LLM clinical competence
Researchers introduce an expanded Polish medical exam benchmark with over 15,000 new questions, two new domains, and four structural modifications designed to reduce multiple-choice artifacts and better test reasoning. Evaluating 21 LLMs under the harder setup, the best-performing model (Qwen3.5-122B) drops 28-31 percentage points compared to standard MCQA scores. The findings suggest standard MCQA benchmarks do not reliably reflect true medical competence, even when data contamination is low. The benchmark is publicly released to support further research.
MedQADE benchmark reveals LLM evaluators match physician agreement scores but lack clinical caution and show lineage bias
Researchers introduce MedQADE, a standardized open-response clinical benchmark for German comprising 3,800 items annotated by ten physicians and nine LLM evaluators. The top LLM evaluator (Gemini 3 Flash) reached statistical alignment near the physician inter-rater ceiling (κ=0.694 vs. κ=0.709), but automated evaluators showed near-zero clinical metacognition: unlike physicians, they never abstained regardless of item difficulty. The study also documents systematic lineage-dependent scoring bias, where models preferentially rate architectural siblings more favorably, independent of language.
NuclearQAv2: A benchmark for evaluating LLM competence in nuclear engineering
Researchers introduce NuclearQAv2, a ~1,240 question benchmark for assessing LLM performance on nuclear engineering knowledge across boolean, numeric, and verbal question types. The benchmark is constructed via a hybrid pipeline combining expert-authored questions, existing datasets, and LLM-assisted generation from domain-specific corpora. Evaluation of multiple LLMs reveals strong performance on factual recall but significant gaps in quantitative reasoning and conceptual understanding, highlighting the need for multi-faceted domain-specific evaluation.
Empirical study of LLM medical domain adaptation trade-offs in French QA
Researchers present a systematic comparison of continual pretraining (CPT), supervised fine-tuning (SFT), and their combination for adapting LLMs to French medical question answering. The study spans three model families, multiple sizes, and three initialization types, evaluating both multiple-choice and open-ended QA formats. Key findings: CPT+SFT yields the best MCQA scores but gains over SFT alone are often not statistically significant, making SFT a cost-effective default; for open-ended QA, CPT improves overlap metrics while SFT degrades generation quality. Cross-lingual transfer from French adaptation to English benchmarks is also demonstrated.
Cross-lingual prompting strategies unlock hidden parametric knowledge in LLMs
A new arXiv preprint investigates how cross-lingual prompting can surface factual knowledge that standard inference techniques fail to retrieve in multilingual LLMs. The authors identify four dimensions of cross-lingual exploration governing parametric knowledge retrieval and evaluate them on multilingual factual benchmarks across 17 typologically diverse languages. Results show cross-lingual exploration improves both factual recall and cross-lingual consistency, and is claimed to be a more compute-efficient approach than scaling native-language inference.
Co-LMLM: Continuous-query limited memory language models outperform vanilla LLMs on factual tasks at small scale
Researchers introduce CO-LMLM, a limited memory language model that externalizes factual knowledge to a knowledge base during pretraining and retrieves it at inference via continuous vector queries paired with human-readable text values. The approach removes prior restrictions to relational knowledge bases and Wikipedia-only data by introducing an annotation pipeline for arbitrary text. At 360M parameters, CO-LMLM achieves lower perplexity than models trained on 40x more data and SimpleQA factual performance comparable to GPT-4o mini and above Claude Sonnet 4.5, suggesting significant efficiency gains for factual grounding.