Evolved Policy Gradients: OpenAI Meta-Learning via Loss Function Evolution
OpenAI released Evolved Policy Gradients (EPG), a meta-learning method that evolves the loss function used to train reinforcement learning agents rather than hand-designing it. The approach enables faster adaptation to novel tasks, with agents demonstrating generalization to test-time scenarios outside their training distribution, such as navigating to objects placed in new locations. EPG represents an experimental direction in automated algorithm discovery for RL.
Related guides (3)
Related events (8)
Evolution Strategies as a Scalable Alternative to Reinforcement Learning
OpenAI published research showing that evolution strategies (ES), a decades-old optimization technique, can match standard reinforcement learning performance on benchmarks like Atari and MuJoCo. The approach offers practical advantages over RL including easier parallelization and fewer hyperparameter sensitivities. This positions ES as a viable alternative training paradigm for policy optimization tasks.
Equivalence between Policy Gradients and Soft Q-Learning
OpenAI published a research result establishing a formal equivalence between policy gradient methods and soft Q-learning, two major families of reinforcement learning algorithms. The work shows that under entropy regularization, these approaches are mathematically equivalent, unifying previously separate lines of RL research. This has implications for algorithm design, theoretical understanding, and the development of hybrid RL methods.
OpenAI Releases Proximal Policy Optimization (PPO)
OpenAI introduced Proximal Policy Optimization (PPO), a new class of reinforcement learning algorithms that match or exceed state-of-the-art performance while being simpler to implement and tune. PPO was adopted as OpenAI's default RL algorithm due to its balance of ease of use and strong performance. The release marked a significant methodological contribution to the RL field that would go on to underpin many subsequent AI training pipelines.
Reptile: A Scalable Meta-Learning Algorithm from OpenAI
OpenAI introduced Reptile, a meta-learning algorithm that works by repeatedly sampling tasks, running stochastic gradient descent, and updating initial parameters toward the task-specific learned parameters. It is mathematically related to first-order MAML but requires only black-box access to standard optimizers like SGD or Adam. The algorithm is positioned as computationally efficient and comparably performant to MAML-based approaches.
Some considerations on learning to explore via meta-reinforcement learning
OpenAI published a research post examining exploration strategies learned through meta-reinforcement learning. The work investigates how agents can acquire exploration behaviors through meta-learning rather than having them hand-designed. This is an early OpenAI contribution to the intersection of meta-learning and RL, predating the current frontier model era.
Variance Reduction for Policy Gradient with Action-Dependent Factorized Baselines
OpenAI published a research paper on variance reduction techniques for policy gradient methods in reinforcement learning. The work introduces action-dependent factorized baselines as a way to reduce variance in policy gradient estimates without introducing bias. This is a foundational RL training methodology contribution relevant to improving sample efficiency in reinforcement learning.
AgenticRL: Self-refining LLM-guided reward design and policy refinement for UAV navigation
AgenticRL is a framework that uses a multimodal GPT agent to automate reward function generation, policy training via PPO, and closed-loop self-refinement for UAV navigation tasks. The agent evaluates trained policies through diagnostic feedback, identifies failure modes, and iteratively refines rewards without human intervention. Evaluated across five navigation tasks, the closed-loop refinement improves policy behavior by 71% over initial rewards, with sim-to-real transfer achieving 91% real-world success rate and 94% sim-to-real accuracy.
Vector Policy Optimization: Training for Diversity Improves Test-Time Search
Vector Policy Optimization (VPO) is a new RL post-training algorithm for LLMs that replaces the scalar reward paradigm with vector-valued rewards, explicitly training models to produce diverse solution sets that specialize across different reward trade-offs. VPO is designed as a near-drop-in replacement for the GRPO advantage estimator and targets inference-scaling search procedures like AlphaEvolve. Across four tasks, VPO matches or outperforms scalar RL baselines on pass@k and best@k metrics, with advantages growing as search budget increases, and unlocks evolutionary search problems that GRPO-trained models cannot solve. The paper argues that diversity-optimized post-training may need to become the default as inference-time search becomes standard.