The Blind Curator: How a Biased Judge Silently Disables Skill Retirement in Self-Evolving Agents

Biased LLM judges silently disable skill retirement in self-evolving agents

A new arXiv paper analyzes how biased LLM judges used as reward signals in self-evolving agent systems can silently disable the 'skill retirement' mechanism that prevents skill libraries from degrading below a no-skill baseline. The authors show that false-pass bias—where failures are incorrectly scored as passes—disables contribution-based retirement past a sharp threshold that cannot be overcome with more data, while leaving aggregate performance metrics unchanged, making the failure invisible. The paper proposes a defect-injection audit to determine pre-deployment whether a judge falls above or below the critical threshold.