Reptile: A Scalable Meta-Learning Algorithm from OpenAI
OpenAI introduced Reptile, a meta-learning algorithm that works by repeatedly sampling tasks, running stochastic gradient descent, and updating initial parameters toward the task-specific learned parameters. It is mathematically related to first-order MAML but requires only black-box access to standard optimizers like SGD or Adam. The algorithm is positioned as computationally efficient and comparably performant to MAML-based approaches.
Related guides (2)
Related events (8)
On First-Order Meta-Learning Algorithms
OpenAI published research on first-order meta-learning algorithms, presenting simplified variants of MAML (Model-Agnostic Meta-Learning) that omit second-order derivatives while retaining competitive performance. The work demonstrates that first-order approximations are surprisingly effective for few-shot learning tasks. This contributed to the broader understanding of gradient-based meta-learning efficiency and scalability.
Evolved Policy Gradients: OpenAI Meta-Learning via Loss Function Evolution
OpenAI released Evolved Policy Gradients (EPG), a meta-learning method that evolves the loss function used to train reinforcement learning agents rather than hand-designing it. The approach enables faster adaptation to novel tasks, with agents demonstrating generalization to test-time scenarios outside their training distribution, such as navigating to objects placed in new locations. EPG represents an experimental direction in automated algorithm discovery for RL.
Some considerations on learning to explore via meta-reinforcement learning
OpenAI published a research post examining exploration strategies learned through meta-reinforcement learning. The work investigates how agents can acquire exploration behaviors through meta-learning rather than having them hand-designed. This is an early OpenAI contribution to the intersection of meta-learning and RL, predating the current frontier model era.
Evolution Strategies as a Scalable Alternative to Reinforcement Learning
OpenAI published research showing that evolution strategies (ES), a decades-old optimization technique, can match standard reinforcement learning performance on benchmarks like Atari and MuJoCo. The approach offers practical advantages over RL including easier parallelization and fewer hyperparameter sensitivities. This positions ES as a viable alternative training paradigm for policy optimization tasks.
OpenAI Develops Hierarchical Reinforcement Learning Algorithm for Long-Horizon Tasks
OpenAI published research on a hierarchical reinforcement learning (HRL) algorithm that learns reusable high-level actions to solve tasks requiring thousands of timesteps. Applied to navigation problems, the algorithm discovers locomotion primitives (walking, crawling in various directions) that enable rapid mastery of new tasks. The approach addresses a core challenge in RL: efficient exploration and transfer across long-horizon tasks.
RL²: Fast Reinforcement Learning via Slow Reinforcement Learning
OpenAI published RL², a meta-reinforcement learning approach in which a slow outer RL process trains a recurrent neural network whose hidden state encodes a fast inner learning algorithm. The method allows agents to rapidly adapt to new tasks within a single episode by leveraging experience accumulated across many training tasks. This work is an early foundational contribution to meta-learning for RL, predating the modern agent and LLM era but relevant to understanding the intellectual lineage of in-context and few-shot learning in AI systems.
Learning Montezuma's Revenge from a Single Demonstration
OpenAI trained a reinforcement learning agent to achieve a score of 74,500 on Montezuma's Revenge using a single human demonstration, surpassing all previously published results. The method is straightforward: the agent plays episodes starting from carefully selected states drawn from the demonstration, optimizing game score via PPO. This approach demonstrates that imitation-seeded curriculum learning can dramatically improve exploration in hard-exploration environments. The same PPO algorithm underpins OpenAI Five.
RePro: Retrospective Progress-Aware Self-Refinement for LLM Agent Training
Researchers introduce RePro (Retrospective Progress-Aware Training), a framework addressing the gap between step-wise RL optimization and metacognitive task-progress awareness in LLM agents. The approach uses a forward-then-reflect rollout paradigm where agents execute actions online and then retrospectively assess step-wise progress given the completed trajectory and known outcome. Evaluated on WebShop, ALFWorld, and Sokoban, RePro achieves up to 12% absolute success rate gains over baseline Qwen-family models without requiring continuous external supervision.