Entity · paper

Selective Timestep Weighting and Advantage-Based Replay for Sample-Efficient Diffusion RLHF

paperactiveprovisionalselective-timestep-weighting-and-advantage-based-replay-for-sample-efficient-diffusion-rlhf-af5a35a4·1 events·first seen 36h ago

Aliases: Selective Timestep Weighting and Advantage-Based Replay for Sample-Efficient Diffusion RLHF

Co-occurring entities

Proximal Policy Optimization

More like this (12)

Hierarchical Advantage Weighting for Online RL Fine-Tuning of VLAs from Sparse Episode Outcomes A Diffusion Approximation for Temporal-Difference Learning with Linear Features under Markovian Noise LESS: Mutual-Stability Sampling for Diffusion Language Models diffusion posterior sampling Score Accuracy Along the Forward Diffusion Does Not Certify Numerical Stability in Diffusion Sampling Gibbs-Accelerated Discrete Diffusion (GADD)FORCE: Efficient VLA Reinforcement Fine-Tuning via Value-Calibrated Warm-up and Self-Distillation Multi-Agent Reinforcement Learning from Delayed Marketplace Feedback for Objective-Weight Adaptation in Three-Sided Dispatch REDDIT: Correcting Model-Generated Timestamp Drift in ASR without Forgetting via Replay-Based Distribution Editing Reinforcement Learning from Rich Feedback with Distributional DAgger discrete diffusion models SDE (Stochastic Differential Equation LR scaling)

Recent events (1)

5arXiv · cs.LG·36h ago·source ↗

Selective timestep weighting and advantage-based replay improve diffusion RLHF sample efficiency by up to 6×

A new arXiv preprint proposes two complementary techniques to improve feedback efficiency in diffusion model RLHF: a per-timestep weighting scheme grounded in PPO convergence theory, and a replay mechanism that prioritizes informative trajectories to reduce redundant reward queries. Together, the methods achieve up to 6× improvement in sample efficiency over standard diffusion RLHF baselines under identical hyperparameter settings. The work addresses a practical bottleneck—feedback cost—that limits real-world deployment of RLHF-aligned diffusion models.

Alignment and RLHF Multimodal Progress Selective Timestep Weighting and Advantage-Based Replay for Sample-Efficient Diffusion RLHF Proximal Policy Optimization