5arXiv cs.LG (Machine Learning)·
MeanFlowNFT extends forward-process RL to few-step MeanFlow generators for image and video
MeanFlowNFT is a new RL fine-tuning framework that adapts the DiffusionNFT forward-process RL objective to MeanFlow generators, which use average-velocity predictions for fast few-step sampling. The key contribution is an induced instantaneous-velocity predictor derived from the MeanFlow identity, allowing reward optimization without disrupting MeanFlow's efficient sampling. The method inherits DiffusionNFT's policy-improvement guarantee and outperforms prior RL-tuned few-step generators on SD3.5-M and Wan 2.1, with 4-step MeanFlowNFT surpassing 50-step RL-tuned diffusion on VBench.
Related guides (2)
Related events (8)
Finetune Stable Diffusion Models with DDPO via TRL
Hugging Face's TRL library adds support for DDPO (Denoising Diffusion Policy Optimization), enabling reinforcement learning-based finetuning of Stable Diffusion models. This extends TRL's RLHF tooling beyond language models to image generation, allowing reward-driven optimization of diffusion models. The post demonstrates practical usage of the new DDPO trainer within the TRL ecosystem.
FMLM+ introduces Posterior Refinement for fast non-autoregressive language generation
Researchers introduce FMLM+, a framework combining Flow Map Language Models with masking-style noise schedules to enable joint sequence generation with per-token global consistency scoring. The key contribution is Posterior Refinement, an inference-time self-correction strategy that matches discrete baseline performance with 32x fewer neural function evaluations (NFEs). The approach improves the speed-quality tradeoff over both Masked Diffusion Models and standard FLMMs across multiple benchmarks, addressing longstanding factorization error problems in non-autoregressive generation.
IMPFM: Multi-particle flow-map framework for online feedback-driven generative search
A new arXiv preprint introduces Sequentially-Controlled Interactive Multi-Particle Flow-Maps (IMPFM), a framework for sample-efficient online search under sequential human feedback. The method transports an ensemble of particles toward a target distribution using flow maps and a posterior sample-sharing mechanism, aiming to balance global exploration with reward over-optimization avoidance. The authors prove convergence to a KL-tilted target via a Feynman-Kac corrector and demonstrate improvements over baselines on alignment and search tasks. The work addresses a gap in training-free reward alignment methods that struggle with unknown or sequentially revealed preferences.
Selective timestep weighting and advantage-based replay improve diffusion RLHF sample efficiency by up to 6×
A new arXiv preprint proposes two complementary techniques to improve feedback efficiency in diffusion model RLHF: a per-timestep weighting scheme grounded in PPO convergence theory, and a replay mechanism that prioritizes informative trajectories to reduce redundant reward queries. Together, the methods achieve up to 6× improvement in sample efficiency over standard diffusion RLHF baselines under identical hyperparameter settings. The work addresses a practical bottleneck—feedback cost—that limits real-world deployment of RLHF-aligned diffusion models.
FlowPipe: LLM-conditioned Generative Flow Networks for automated data preparation pipeline construction
FlowPipe is a new framework that frames ML data preparation pipeline synthesis as conditional probabilistic flow generation over a directed acyclic graph, using Conditional Generative Flow Networks (C-GFlowNets) with a Trajectory Balance objective. LLM-derived semantic priors are injected into the policy via Feature-wise Linear Modulation (FiLM), and a failure-aware flow objective steers search away from invalid states. Evaluated on 74 real-world datasets across two benchmark suites, FlowPipe improves accuracy by 11.96% on average over SOTA baselines and achieves 12.5x faster training convergence. The work addresses long-standing limitations in automated data pipeline construction including weak credit assignment and inefficient exploration.
DEFAR framework uses exposure bias signals to self-rectify Flow Matching during training
A new arXiv preprint introduces DEFAR (DirEctional-Frequency Adaptive Rectification), a training framework for Flow Matching generative models that addresses exposure bias — the train/inference discrepancy — by extracting dynamic correction signals from the bias itself. The method has two components: Anti-Drift Rectification (ADR), which steers deviated inference states back toward targets, and Frequency Compensation (FC), which reinforces missing low-frequency components using bias as a self-feedback weight. Experiments on CIFAR-10, CelebA-64, and ImageNet-256/512 show improvements over prior baselines with favorable scalability and inference robustness.
DiT-Reward converts text-to-image diffusion transformers into reward models, outperforming HPSv3
DiT-Reward is a new reward modeling approach that repurposes pretrained text-to-image Diffusion Transformers (DiTs) by processing near-clean image latents and aggregating text-conditioned representations across transformer layers. Under matched training data, it outperforms HPSv3 on all four evaluated preference benchmarks, reaching 85.6% on HPDv2 and 77.6% on HPDv3. When used to optimize Stable Diffusion 3.5 Large via Flow-GRPO, it shows clear gains in realism and achieves a 1.65x inference speedup over HPSv3. The work demonstrates that generative DiT representations transfer meaningfully to reward modeling and policy optimization.
FORCE: Efficient RL fine-tuning for Vision-Language-Action models via value-calibrated warm-up and self-distillation
Researchers introduce FORCE, a 3-stage reinforcement learning fine-tuning framework for Vision-Language-Action (VLA) models that addresses sample inefficiency caused by unstable Q-functions and low-quality exploration data. The framework uses a Value-Calibrated Warm-Up phase followed by Q-function-filtered policy updates, eliminating the need for costly human interventions during training. Evaluated on simulation and real-world robotic tasks, FORCE achieves a 79% absolute improvement in task success rates, outperforms prior RL methods by 10%, and accelerates training by 32.5%.