5arXiv cs.CL (Computation and Language)·
GEIS: Generation-Evaluation-Improvement loop for long-form article generation with modular agent skills
Researchers introduce GEIS, a framework that decomposes long-form Wikipedia-style article generation into named, declarative agent skills organized in a generation-evaluation-improvement loop. Implemented in the Tasi Harness, GEIS outperforms STORM on structural and content quality dimensions and improves over the default writer by 8.0 points on a 100-point rubric across 20 Wikipedia Featured Article topics. A key contribution is the improvement skill, which maps recurrent evaluation findings into permanent patches to the writing skill, enabling iterative self-refinement without rewriting fixed pipelines.
Related guides (2)
Related events (8)
Systematic Study of Model-Generated Agent Skills Across the Full Skill Lifecycle
This paper presents a utility-grounded evaluation framework for model-generated agent skills, covering the full lifecycle of experience generation, skill extraction, and skill consumption across five agentic task domains. The authors find that while such skills are beneficial on average, they exhibit non-trivial negative transfer, and that skill utility is independent of model scale or baseline task strength. A key finding is that strong extractors are not necessarily strong consumers and vice versa. The work culminates in a 'meta-skill' that guides extraction toward utility-correlated features, consistently improving skill quality and reducing negative transfer.
SkillGenBench: Benchmarking Skill Generation Pipelines for LLM Agents
SkillGenBench is a new benchmark designed to evaluate the ability of LLM agents to generate correct, reusable, and executable skills from raw repositories and documents, rather than merely using pre-provided skills. It covers two generation regimes (task-conditioned and task-agnostic) and two procedural sources (repository-grounded and document-grounded), with standardized execution-based evaluation protocols. Experiments across multiple skill-generation methods reveal substantial performance variation and distinct failure modes depending on source type. The benchmark aims to establish skill generation as an independent research problem within agent systems.
SkillOpt: Systematic Text-Space Optimizer for Self-Evolving Agent Skills
SkillOpt introduces a principled optimization framework for agent skills, treating the skill document as an external trainable state analogous to model weights. A separate optimizer model converts scored rollouts into bounded edits (add/delete/replace) on a skill document, accepting only edits that improve held-out validation scores. Evaluated across six benchmarks, seven target models, and three execution harnesses (direct chat, Codex, Claude Code), SkillOpt achieves best or tied performance on all 52 evaluated cells, lifting GPT-5.5 no-skill accuracy by up to +24.8 points inside the Codex agentic loop. Optimized skill artifacts also transfer across model scales and execution environments without further optimization.
Double Ratchet: Co-evolving evaluation metrics and skills for self-improving LLM agents
A new arXiv preprint introduces Double Ratchet, a system that co-evolves both evaluation metrics and agent skills in settings where no reliable automatic verifier exists. The metric loop uses evolutionary search over small drawback detectors anchored to a small reference set, while the skill loop uses a lifecycle-managed approach; together they retain 88–110% of the performance lift achievable with ground-truth metrics across code generation (MBPP+), text-to-SQL (Spider 2.0-Snow), and report generation tasks. The paper also addresses safety, showing that anchor discipline and outer audits can catch and repair cases where evolved skills game the rubric. This work directly addresses a core bottleneck in self-improving agent systems: the chicken-and-egg problem of needing a reliable evaluator to improve.
AI-Assisted Systematization for Evaluating GenAI Systems
This paper addresses a foundational gap in GenAI evaluation: the underspecification of broad, contested concepts like 'reasoning,' 'fairness,' or 'creativity.' The authors introduce a structured artifact called a 'concept spec' and a validation worksheet, then build two AI-assisted systematizers—a zero-shot approach and a multi-agent approach—to convert vague evaluation targets into measurable, structured accounts. They apply these tools to hate-based rhetoric and digital empathy, assessing the resulting specs on content validity and information recoverability. The work positions AI assistance as a scalable aid for the cognitively demanding process of evaluation design.
Latent Space: Skill engineering and the case against one-shot AI design
Paul Bakaus discusses 'skill engineering' as a design philosophy for AI-assisted workflows, arguing against fully automated one-shot AI pipelines in favor of keeping humans in the loop. The conversation centers on Impeccable, a tool or approach Bakaus is developing, and the concept of 'loopmaxxing' — iterative human-agent collaboration cycles. The piece addresses why current agents still require human steering to produce high-quality outputs.
Lilian Weng summarizes 35 papers on Harness Engineering for RSI
Latent Space's AINews digest covers a summary by Lilian Weng of 35 papers on Harness Engineering for Recursive Self-Improvement (RSI), a topic at the intersection of agent scaffolding, self-improvement loops, and AI safety. Weng, a prominent researcher at OpenAI, synthesizing this volume of work signals growing institutional attention to RSI as a research area. The digest frames this as a quiet-day read, suggesting it is a curated secondary synthesis rather than a primary research release.
Self-evolving agent harness framework uses quality-diversity archive to generalize harness improvements across model families
A new arXiv preprint introduces a framework for automatically improving LLM agent harnesses (prompts, injected knowledge, runtime control, configuration) without modifying model weights. The system separates change proposal (done by an LLM diagnosing failures) from credit assignment (done by deterministic statistical testing), populating a gated quality-diversity archive keyed on failure pathology rather than task identity to prevent overfitting. Evaluated across seven domains with a frozen open-weight model, the framework achieves +9 to +15.5 percentage point gains on sealed test sets, retaining 86–147% of training gains, with the key finding that pathology-to-patch mappings transfer across model families.