Hindcast

Hindcast: A benchmark framework that closes data-leakage loopholes in LLM forecasting evaluation

A new arXiv preprint introduces Hindcast, an evaluation framework for LLM forecasters that addresses two systematic data-leakage channels in standard backtesting: post-event retrieval and training-data contamination from newer models. The system replays resolved Polymarket prediction markets against a frozen Reddit snapshot, grading models only on information available before a chosen cutoff date and comparing against contemporaneous market prices as a human-forecast baseline. Key finding: retrieval helps forecasting only when pre-event Reddit discussion existed; where only speculation was available, retrieval hurts performance. The framework is designed to remain valid as new models and markets emerge without going stale.