Entity · paper

Who Grades the Grader? Co-Evolving Evaluation Metrics and Skills for Self-Improving LLM Agents

paperactiveprovisionalwho-grades-the-grader-co-evolving-evaluation-metrics-and-skills-for-self-improving-llm-agents-d0a4b954·1 events·first seen 4d ago

Aliases: Who Grades the Grader? Co-Evolving Evaluation Metrics and Skills for Self-Improving LLM Agents

Co-occurring entities

Double Ratchet MBPP Spider 2.0-Snow

More like this (12)

Skill-RM: Unifying Heterogeneous Evaluation Criteria via Agent Skill Grading the Grader: Lessons from Evaluating an Agentic Data Analysis System multi-level agent evaluation Generative Skill Composition for LLM Agents Can LLMs Judge Better Than They Generate? Evaluating Task Asymmetry, Mechanistic Interpretability and Transferability for In-Context QA Multi-Turn Evaluation of Deep Research Agents Under Process-Level Feedback Evaluation Cards: An Interpretive Layer for AI Evaluation Reporting Do Agent Optimizers Compound? A Continual-Learning Evaluation on Terminal-Bench 2.0 Personalized Evaluation as Learning Model-Generated Agent Skills (paper)GEIS: A Generation-Evaluation-Improvement Loop of Agent Skills for Long-Form Article Generation Role-Agent: Bootstrapping LLM Agents via Dual-Role Evolution

Recent events (1)

6arXiv · cs.CL·4d ago·source ↗

Double Ratchet: Co-evolving evaluation metrics and skills for self-improving LLM agents

A new arXiv preprint introduces Double Ratchet, a system that co-evolves both evaluation metrics and agent skills in settings where no reliable automatic verifier exists. The metric loop uses evolutionary search over small drawback detectors anchored to a small reference set, while the skill loop uses a lifecycle-managed approach; together they retain 88–110% of the performance lift achievable with ground-truth metrics across code generation (MBPP+), text-to-SQL (Spider 2.0-Snow), and report generation tasks. The paper also addresses safety, showing that anchor discipline and outer audits can catch and repair cases where evolved skills game the rubric. This work directly addresses a core bottleneck in self-improving agent systems: the chicken-and-egg problem of needing a reliable evaluator to improve.

Evaluation and Benchmarking AI Safety Research Double Ratchet MBPP Who Grades the Grader? Co-Evolving Evaluation Metrics and Skills for Self-Improving LLM Agents +2 more