EpiCurveSimilarity

EpiCurveBench: A Benchmark for Evaluating VLMs on Epidemic Curve Digitization

EpiCurveBench introduces a benchmark of 1,000 real-world epidemic curve images and a new evaluation metric (EpiCurveSimilarity, ECS) designed to assess vision-language models on time-series chart extraction, addressing limitations of existing metrics that ignore temporal structure. Evaluating six methods including three frontier closed VLMs, one open VLM, and two specialized chart-extraction systems, the best model achieves only 52.3% ECS, revealing substantial headroom compared to saturating scores on ChartQA. ECS is validated against downstream epidemiological statistics and shown to correlate 1.5–3.6× more strongly than Dynamic Time Warping across four summary metrics. The benchmark targets the public-health use case of digitizing historical outbreak data trapped in published figures, but generalizes to any structured time-series chart-extraction task.