Entity · paper

Do You Need a Frontier Model as a Citation Verifier? Benchmarking Rubric LLMs for Deep-Research Source Attribution

paperactiveprovisional

do-you-need-a-frontier-model-as-a-citation-verifier-benchmarking-rubric-llms-for-deep-research-source-attribution-de19d4b1

·1 events·first seen 5h ago

Aliases: Do You Need a Frontier Model as a Citation Verifier? Benchmarking Rubric LLMs for Deep-Research Source Attribution

Co-occurring entities

OpenAI GPT-5.4 mini

More like this (12)

Can LLMs Judge Better Than They Generate? Evaluating Task Asymmetry, Mechanistic Interpretability and Transferability for In-Context QA Which Models Are Our Models Built On? Auditing Invisible Dependencies in Modern LLMs When the Judge Changes, So Does the Measurement: Auditing LLM-as-Judge Reliability frontier LLMs frontier model evaluation Can LLMs Reliably Self-Report Adversarial Prefills, and How?Clinician-Level Agreement Without Clinical Caution: LLM Evaluator Limits in Medical AI Benchmarking ProvenanceGuard: Source-Aware Factuality Verification for MCP-Based LLM Agents When LLMs Read Tables Carelessly: Measuring and Reducing Data Referencing Errors Validity of LLMs as data annotators: AMALIA on authority How Does Research Evolve? Tracing Cross-Domain Trajectories in NLP, ML, and CV with Claim-Grounded Typed Citations LLM-as-a-Verifier

Recent events (1)

5arXiv · cs.CL·5h ago·source ↗

Benchmarking LLM judges for citation verification in deep-research systems

A new arXiv paper evaluates 8 LLM judges from 3 model families on citation quality assessment for deep-research systems, testing across 1,248 rubric decisions with human-reviewed gold labels. The study finds that cheaper models remain competitive with frontier models — GPT-5-mini achieves the strongest source-relevance F1 at 0.908 — but judges differ substantially in directional bias (pass-rate drift, false positive/negative rates) even when scalar F1 scores are similar. The key finding is that scalar F1 obscures biases that would be directly reinforced in an RL training loop, making judge calibration a prerequisite before using citation rubrics as reward signals.

Evaluation and Benchmarking Alignment and RLHF OpenAI GPT-5.4 mini Do You Need a Frontier Model as a Citation Verifier? Benchmarking Rubric LLMs for Deep-Research Source Attribution