Researchers introduce PEARL (Physics-EnhAnced Reinforcement Learning), a hybrid paradigm combining RL with classical optimal control techniques for high-dimensional dynamical systems. The method uses an actor-adjoint algorithm that leverages automatic differentiation and adjoint-based sensitivity computation to dramatically reduce environment interactions and mitigate long-term gradient instabilities. Demonstrated on parametric navigation problems in unsteady flows, PEARL outperforms standard RL baselines while generalizing across scenarios and scaling to high-dimensional state/action spaces without requiring dimensionality reduction.
A new arXiv preprint introduces KGRL (Knowledge- and Gradient-Guided Reinforcement Learning), a neuro-symbolic algorithm for Parametrized Action Markov Decision Processes (PAMDPs) where each decision involves both a symbolic action and continuous numerical parameters. KGRL uses a Datalog knowledge base to prune infeasible actions and constrain parameter spaces, then applies gradient-based refinement to estimate optimal parameters during training and deployment. The approach improves sample efficiency and episodic return over state-of-the-art PAMDP baselines, and provides local procedural explanations for its decisions.
A new arXiv preprint provides theoretical analysis of Reinforcement Learning from Verifiable Rewards (RLVR) updates, identifying off-policy degree and gradient expectation as key factors governing update dynamics. The authors show that differences in gradient steps per rollout substantially affect importance sampling ratio distributions and which tokens dominate updates. Based on this analysis, they propose Adaptive Clip Policy Optimization (ACPO), which adjusts clipping boundaries per token group by empirical variance of importance sampling ratios, outperforming DAPO and CISPO baselines on 3B and 7B models across math, tabular QA, and logic benchmarks.
GPRL proposes a new alignment framework that replaces scalar reward models with a General Preference Model (GPM) embedding responses into k skew-symmetric subspaces to capture multi-dimensional, intransitivity-aware preferences. The method computes per-dimension group-relative advantages, normalizes across axes, and uses a closed-loop drift monitor to detect and correct single-axis reward hacking during training. Starting from Llama-3-8B-Instruct, GPRL achieves a 56.51% length-controlled win rate on AlpacaEval 2.0 and outperforms SimPO and SPPO on Arena-Hard, MT-Bench, and WildBench. The work directly addresses the gap between verifiable-reward online RL (strong on math/code) and preference optimization (strong on open-ended tasks).
AgenticRL is a framework that uses a multimodal GPT agent to automate reward function generation, policy training via PPO, and closed-loop self-refinement for UAV navigation tasks. The agent evaluates trained policies through diagnostic feedback, identifies failure modes, and iteratively refines rewards without human intervention. Evaluated across five navigation tasks, the closed-loop refinement improves policy behavior by 71% over initial rewards, with sim-to-real transfer achieving 91% real-world success rate and 94% sim-to-real accuracy.
Researchers propose PACT (Plan, Align, Commit, Think), a hybrid architecture pairing a fast reactive RL policy with an asynchronous small language model planner for deliberation. The SLM generates and validates candidate action plans via simulation before committing to execution, bypassing the RL policy without retraining. Evaluated on FrozenLake configurations of increasing difficulty, PACT outperforms baselines using only a 2B-parameter SLM, suggesting complementary strengths between deliberative planning and reactive execution.
A new arXiv preprint introduces Preference Coordinated Multi-agent Policy Optimization (PCMA), a method for cooperative multi-objective multi-agent reinforcement learning (MOMARL) that learns agent-specific preferences to enable complementary trade-offs across agents. The authors formulate cooperative MOMARL as a team-optimal game and provide a first-order improvement decomposition showing that preference diversity can induce team improvement. Experiments on cooperative MOMA environments and a traffic-control scenario demonstrate improvements in both performance and trade-off coordination.
OPERA (Objective Perplexity-based Reflective Alignment) proposes replacing LLM-as-a-judge reward models with intrinsic rewards derived from perplexity dynamics to stabilize RL training on open-ended tasks like creative writing. The method includes a cold-start data synthesis pipeline generating 20,000 reasoning trajectories using perplexity-prioritized rollouts. Applied to Qwen3-8B, OPERA claims state-of-the-art among open-source models on open-ended tasks, reportedly matching or exceeding Gemini 2.5 and MiniMax-M2.5 on some benchmarks.
OpenAI researchers found that adding adaptive noise to the parameters of reinforcement learning algorithms frequently improves performance across tasks. The technique is described as simple to implement and rarely harmful, making it broadly applicable. This work contributes to exploration strategies in RL, a longstanding challenge in the field.