DROP
drop-79495e58·2 events·first seen 28d agoAliases: DROP
Co-occurring entities
More like this (12)
Recent events (2)
Open LLM Leaderboard: DROP Deep Dive
Hugging Face published a detailed analysis of the DROP benchmark as used in the Open LLM Leaderboard, examining how models are evaluated on this reading comprehension and numerical reasoning task. The post investigates scoring methodology, potential issues with evaluation consistency, and what DROP results actually reveal about model capabilities. This is part of ongoing efforts to improve transparency and reliability of the Open LLM Leaderboard.
Operadic consistency: a label-free signal for detecting compositional reasoning failures in LLMs
Researchers introduce operadic consistency (OC), a label-free inference-time signal that checks whether an LLM's direct answer to a compositional query agrees with the answer produced by composing its own stated decomposition of that query. Evaluated across 12 instruction-tuned LLMs (4B–671B parameters) on four multi-hop QA datasets, OC achieves Pearson r ∈ [0.86, 0.94] with accuracy uniformly across all datasets, outperforming self-consistency, semantic entropy, and P(True) in cross-dataset robustness. At the per-question level, OC provides information beyond existing baselines and yields selective-prediction improvements (AUARC lifts +0.086–0.096, AUROC lifts +0.092–0.164) at equal sampling cost, with results extending to frontier thinking models using chain-of-thought decompositions.