RELAI

Continual-learning evaluation on Terminal-Bench 2.0 tests whether agent optimizer gains compound across tasks

A new arXiv paper introduces a two-phase continual-learning evaluation framework built on Terminal-Bench 2.0 to test whether agent-optimization gains persist and compound when new tasks arrive over time. Three agent-harness optimization methods — GEPA, Meta Harness, and RELAI-VCL — are compared under identical budgets; all improve in static single-phase settings but diverge sharply under continual optimization. RELAI-VCL is the only method that both transfers positively to unseen tasks and continues improving, reaching a 76.4% lifelong average pass rate versus 58.7% for the unoptimized baseline. The key finding is that compounding gains require regression control built into the optimization loop to prevent shortcut solutions.