6arXiv cs.CL (Computation and Language)·
GAMUT benchmark introduces two-level meta-rubrics for evaluating factual completeness in long-form generation
Researchers introduce GAMUT (Grounded Assessment of Multimodal Factuality), a benchmark of 1,813 questions targeting factual completeness—the recall side of factuality—in long-form LLM outputs. The framework uses a two-level meta-rubric that captures content organization and importance, then compiles it into flat binary checklists for reliable LLM-judge scoring. Evaluating 14 frontier and open-weight models, the best score is 58.7% from Gemini 3.1 Pro, indicating the benchmark is genuinely challenging and discriminative. The work addresses a gap in existing factuality evaluation pipelines, which focus on precision but not recall of required information.
Related guides (2)
Related events (8)
FACTS Benchmark Suite: Systematically evaluating the factuality of large language models
DeepMind has released the FACTS Benchmark Suite, a systematic evaluation framework for measuring the factuality of large language models. The benchmark is designed to assess how accurately LLMs produce factually grounded outputs. This represents a structured contribution to the growing field of LLM evaluation, specifically targeting hallucination and factual reliability. The announcement comes from a Tier 1 lab, lending it credibility as a reference benchmark in the field.
Introducing SimpleQA: OpenAI's Factuality Benchmark for Language Models
OpenAI has released SimpleQA, a benchmark designed to measure language model factuality on short, fact-seeking questions. The benchmark targets a specific and well-defined capability: answering direct factual queries accurately. It is intended to provide a clean signal on model truthfulness and calibration for this class of questions.
Disagreement among frontier LLMs on real-world fact-checks
A study examines how frontier large language models diverge in their responses to real-world fact-checking queries, surfacing systematic disagreements across models on factual claims. The work appears to benchmark multiple leading models against a set of verifiable facts, revealing inconsistencies that have implications for reliability and deployment. With 475 HN points and 333 comments, the piece has generated substantial community discussion. The findings are relevant to evaluation methodology, model calibration, and trust in AI-generated factual content.
FACTOR: Risk-aware adaptive verification for factual long-form LLM generation
Researchers propose FACTOR (FACTuality-Oriented Risk-aware Verification), an inference-time framework that adapts verification effort based on claim-level hallucination risk rather than applying uniform verification to all claims. The system combines uncertainty estimation, adaptive language inference verification, and candidate re-ranking to focus resources on high-risk claims. Evaluated on the FactScore benchmark, FACTOR improves factuality while simultaneously reducing verification cost, with model-agnostic performance reported across ablation studies.
CommunityFact: A Dynamic, Multilingual, Multi-domain Benchmark for Misinformation Detection in the Wild
CommunityFact is a refreshable benchmark for misinformation detection containing 15,992 standalone claims across five languages and two domains, designed to address limitations of static benchmarks. The authors evaluate ten LLMs under varying inference-time conditions including chain-of-thought reasoning and web-search augmentation, finding that web access yields the largest performance gains. A key finding is that web-enabled LLMs' source-selection policies are systematically misaligned with sources that human Community Notes raters converge on, a gap addressable through retrieval expansion or pruning. The benchmark also proposes using Community Notes as a training signal for claim-conditioned source suggesters.
Meta-evaluation of LLM-generated rubrics for paper reproduction benchmarks
A new arXiv preprint presents the first systematic meta-evaluation of LLM-generated rubrics for assessing paper reproduction tasks, addressing scalability limitations of expert-constructed rubrics in benchmarks like PaperBench. The authors test four generation settings across two backbone models, evaluating rubric quality both intrinsically (semantic similarity) and extrinsically (score alignment with ground-truth). Results show augmented generation settings can approach human-baseline alignment, but LLM-generated rubrics tend to be overly fine-grained, score-biased, and domain-insensitive.
Empirical taxonomy of factual errors in human-written text reveals LLM detection gaps
A new arXiv paper introduces a taxonomy of factual error types in human-written text, derived from analysis of newspaper article corrections, identifying categories like kanji misconversions and numeral classifier errors absent from existing hallucination benchmarks. The authors evaluate several LLMs on Factual Error Detection (FED) tasks using both synthetic and real correction data. Even high-performance models like GPT-5.4 achieve only ~52% word-level F1 on synthetic data, underscoring the difficulty of detecting human-induced factual errors versus LLM hallucinations. The work highlights a neglected subproblem in factual accuracy research as the field has shifted focus toward LLM-generated hallucinations.
Benchmarking LLM judges for citation verification in deep-research systems
A new arXiv paper evaluates 8 LLM judges from 3 model families on citation quality assessment for deep-research systems, testing across 1,248 rubric decisions with human-reviewed gold labels. The study finds that cheaper models remain competitive with frontier models — GPT-5-mini achieves the strongest source-relevance F1 at 0.908 — but judges differ substantially in directional bias (pass-rate drift, false positive/negative rates) even when scalar F1 scores are similar. The key finding is that scalar F1 obscures biases that would be directly reinforced in an RL training loop, making judge calibration a prerequisite before using citation rubrics as reward signals.