5arXiv cs.CL (Computation and Language)·
EvoPolicyGym benchmark evaluates autonomous policy evolution by LLM agents in RL environments
A new arXiv preprint introduces EvoPolicyGym, a benchmark for evaluating how LLM-based agents iteratively improve executable policies in compact interactive RL environments under a fixed interaction budget. The benchmark provides trajectory-level diagnostics beyond aggregate scores, distinguishing how agents allocate budget and convert feedback into parametric tuning. GPT-5.5 achieves the strongest aggregate rank score and top-two performance across all 16 environments. The work targets a gap in agent evaluation where iterative policy refinement is conflated with open-ended software engineering progress.
Related guides (4)
Related events (8)
RevengeBench: Benchmark for Reconstructing Agent Decision Programs from Behavioral Observations
RevengeBench is a new benchmark of 75 LLM-generated, Elo-calibrated policies across five game environments that tests whether a learner can reconstruct a hidden agent's decision program as executable code from behavioral traces alone. The benchmark draws from CodeClash tournament trajectories and allows the learner to design controlled behavioral probes (custom opponent policies) to elicit informative behavior before submitting an executable hypothesis. Evaluated across twelve frontier LLMs, recovery quality ranges from 34 to 72% of initial action-distance closed, with reconstructed policies providing measurable competitive advantage especially for weaker models. The work frames policy reconstruction as a tractable inverse problem in code-space, with implications for opponent modeling and policy interpretability.
EvoArena benchmark and EvoMem memory paradigm for LLM agents in dynamic environments
Researchers introduce EvoArena, a benchmark suite that evaluates LLM agents in dynamic environments by modeling changes as progressive update sequences across terminal, software, and social domains. Alongside it, they propose EvoMem, a patch-based memory paradigm that records memory evolution as structured update histories to help agents reason about environmental change. Current agents score only 39.6% average accuracy on EvoArena, while EvoMem yields consistent gains on EvoArena and also improves performance on GAIA and LoCoMo benchmarks. The work highlights a significant gap between static-benchmark performance and real-world dynamic deployment requirements.
Evolved Policy Gradients: OpenAI Meta-Learning via Loss Function Evolution
OpenAI released Evolved Policy Gradients (EPG), a meta-learning method that evolves the loss function used to train reinforcement learning agents rather than hand-designing it. The approach enables faster adaptation to novel tasks, with agents demonstrating generalization to test-time scenarios outside their training distribution, such as navigating to objects placed in new locations. EPG represents an experimental direction in automated algorithm discovery for RL.
GraphPO: Graph-based Policy Optimization reduces redundancy in LLM reasoning RL
GraphPO is a new reinforcement learning framework that represents reasoning rollouts as directed acyclic graphs rather than independent chains or trees, merging semantically equivalent reasoning paths into equivalence classes to share suffixes and reduce redundant exploration. The approach assigns efficiency advantages to incoming edges and correctness advantages to outgoing edges, deriving process supervision from outcome rewards. Experiments on three LLMs across reasoning and agentic search benchmarks show consistent improvements over chain- and tree-based baselines under equal token or response budgets. The method also provides theoretical guarantees on reduced advantage-estimation variance.
PlanBench-XL: Benchmark for LLM Agent Planning in Large-Scale Tool Ecosystems
Researchers introduce PlanBench-XL, an interactive benchmark of 327 retail tasks spanning 1,665 tools designed to evaluate LLM agents on long-horizon planning under retrieval-limited tool visibility. The benchmark includes a blocking mechanism simulating real-world disruptions such as missing or failing tools, forcing agents to detect and recover from broken execution paths. Experiments on ten leading LLMs reveal severe performance degradation: GPT-5.4 drops from 51.90% accuracy in unblocked settings to 11.36% under the most severe blocking condition, highlighting fragility in adaptive planning for large, imperfect tool environments.
AgenticRL: Self-refining LLM-guided reward design and policy refinement for UAV navigation
AgenticRL is a framework that uses a multimodal GPT agent to automate reward function generation, policy training via PPO, and closed-loop self-refinement for UAV navigation tasks. The agent evaluates trained policies through diagnostic feedback, identifies failure modes, and iteratively refines rewards without human intervention. Evaluated across five navigation tasks, the closed-loop refinement improves policy behavior by 71% over initial rewards, with sim-to-real transfer achieving 91% real-world success rate and 94% sim-to-real accuracy.
LamPO: Lambda-Style Policy Optimization with Pairwise Decomposed Advantage for Reasoning LMs
LamPO proposes a new RLVR training objective that replaces GRPO's scalar group-relative advantages with a Pairwise Decomposed Advantage, aggregating pairwise reward gaps within response groups and weighting comparisons by confidence-aware log-probability differences. The method retains a critic-free, clipped-update PPO-style structure and optionally adds a ROUGE-L-based dense auxiliary reward to reduce sparsity. Experiments on AIME24, AIME25, MATH-500, and GPQA-Diamond using Qwen3-1.7B, Qwen3-4B, and Phi-4-mini show consistent improvements over GRPO and other RLVR variants with more stable training dynamics.
APPO: Fine-grained branching and credit assignment for agentic RL in LLMs
Researchers introduce Agentic Procedural Policy Optimization (APPO), a reinforcement learning method that shifts branching and credit assignment from coarse tool-call boundaries to fine-grained decision points within generated sequences. APPO uses a Branching Score combining token uncertainty with policy-induced likelihood gains to select exploration points, plus procedure-level advantage scaling for credit distribution. Evaluated on 13 benchmarks, APPO improves strong agentic RL baselines by nearly 4 points while maintaining efficient tool use and interpretability. The work addresses a known weakness in multi-turn agentic RL: that influential decisions are distributed throughout sequences, not concentrated at tool-call boundaries.