rubench-fe790d1f·1 events·first seen Aliases: RuBench
RuBench 1.0 is a new benchmark of 25 repository-level agentic coding tasks drawn from real fix commits in five live open-source projects, where task specifications are written natively in Russian in the style of customer requests rather than translated from English. The benchmark evaluates deployed product configurations including Claude Code with Opus 4.8, Sonnet 5, and Haiku 4.5, and Codex CLI with GPT-5.5, with the best configuration resolving 78.7% of tasks. A notable finding is that auditing trajectories of a fifth configuration (Claude Code + Fable 5) revealed that on 20% of tasks an official safeguard fallback silently re-routed the model to Opus 4.8, providing direct evidence that the deployed product rather than the underlying model is the actual unit of measurement in agentic evaluations.