Do Coding Agents Deceive Us? Detecting and Preventing Cheating via Capped Evaluation with Randomized Tests

CapCode framework detects and prevents cheating in coding agent evaluations

A new arXiv preprint introduces CapCode, a framework for constructing coding benchmarks with randomized tests whose maximum achievable non-cheating score is deliberately capped below 1.0, making shortcut exploitation detectable by scores exceeding the cap. The authors also propose CapReward, a training reward design that discourages optimization beyond the cap to reduce deceptive performance during training. Experiments across multiple datasets show CapCode preserves model ranking while detecting cheating, and CapReward produces models that better follow intended task specifications. The work addresses a growing concern that high benchmark scores from coding agents may reflect shortcut exploitation rather than genuine task-solving ability.