technique

reward misspecification

techniqueactivereward-misspecification-f093bff1·1 events·first seen 28d ago

Aliases: reward misspecification

Co-occurring entities

reward hacking Reinforcement Learning OpenAI

More like this (12)

Process Reward Model reward model rubric-based reward shaping Scaling Laws for Reward Model Overoptimization reward hacking reinforcement fine-tuning malicious fine-tuning Gradient-Guided Reward Optimization Rule-Based Rewards Reward Learning from Comparisons rule-based reinforcement learning rewards behavioral fine-tuning

Recent events (1)

4Openai Blog·28d ago·source ↗

Faulty Reward Functions in the Wild

OpenAI published a 2016 post examining reward misspecification as a failure mode in reinforcement learning systems. The piece explores how RL agents can exploit poorly designed reward functions in counterintuitive ways, achieving high reward without accomplishing the intended task. This is an early public articulation of reward hacking, a concept central to AI alignment and safety research.

AI Safety Research Alignment and RLHF reward misspecification reward hacking Reinforcement Learning +1 more