4arXiv cs.LG (Machine Learning)·
RAG-based policy learning formalized under causal inference framework with regret bounds
A new arXiv preprint proposes one-step and two-step methods for policy learning using retrieval-augmented generation, formulated under the potential outcome framework from causal inference. The two-step method connects action-specific vector search to nearest-neighbor matching, with regret decomposed into candidate-generation and within-candidate choice components. Theoretical bounds on regret are derived using prediction-error guarantees for nearest-neighbor estimators and transformers. The work bridges RAG methodology with formal causal inference theory, offering a principled treatment of action selection in RAG-based systems.
Related guides (2)
Related events (8)
DynaKRAG: Learnable state-conditioned control policy for multi-hop RAG evidence acquisition
DynaKRAG is a new framework that formulates multi-hop retrieval-augmented generation as a state-conditioned control problem over atomic evidence operations (iterative retrieval, query reformulation, sufficiency judging, etc.), using a learned controller to select among valid operations at each step. Evaluated with Qwen2.5-7B-Instruct, it achieves F1 scores of 0.5998 on HotpotQA, 0.5340 on 2WikiMultiHopQA, and 0.3061 on MuSiQue, outperforming the strongest baselines on all three benchmarks. Ablations show that replacing the learned controller with a uniform policy costs 3.96–5.78 F1 points, and that additional retrieval is not uniformly beneficial.
Repeated Policy Regret (RP-Regret): Regret minimization against adaptive opponents in repeated games
This arXiv paper introduces Repeated Policy Regret (RP-Regret), a new game-theoretic metric for regret minimization in repeated games where opponents can adapt based on play history — a setting where standard external regret fails. The authors prove necessary conditions for sublinear RP-Regret and propose three algorithms to minimize it, including oracle-based, linearized surrogate, and slow-opponent variants. When all players minimize RP-Regret, certain subgame perfect equilibria can be learned, and experiments show more cooperative outcomes in games like Stag-Hunt.
Agency-transferring technique improves RL policy training by bootstrapping from baseline policies
A new arXiv paper proposes a model-free reinforcement learning method that embeds an existing suboptimal baseline policy into training via an arbitration mechanism, progressively transferring control from the baseline to a trainable neural network. The approach yields high goal-reaching rates from the start of training and produces a standalone policy that outperforms the baseline without requiring it at inference time. Theoretical bounds on goal-reaching probability are derived, and empirical results on continuous-control benchmarks show competitive or superior returns compared to existing methods.
DARP: Semi-parametric retrieval-based imitation learning reduces compounding errors by 15-46%
Researchers introduce DARP (Difference-Aware Retrieval Policies), a semi-parametric imitation learning method that retrieves k-nearest neighbor demonstrations at inference time and predicts actions based on relative distance vectors between neighbor and query states. The approach reparameterizes behavior cloning around local neighborhood structure rather than global state-to-action mappings, requiring no additional data collection or online expert feedback. Across continuous control and robotic manipulation tasks, DARP shows 15-46% performance improvements over standard behavior cloning, including on high-dimensional visual inputs.
Latent Memory Palace: Variational inference framework enables adaptive reasoning for continuous control policies
Researchers introduce Latent Memory Palace (LMP), a method that formulates reasoning for continuous control policies as variational inference over an autoregressive latent distribution, analogous to a memory palace. The approach derives a latent-space reinforcement learning technique to optimize the variational lower bound, yielding a policy (LMP-π) with adaptive test-time compute allocation and a variable-length action tokenizer (LMP-tok) that improves downstream autoregressive policies. The work addresses the gap between language model chain-of-thought reasoning and continuous control, where language-space reasoning lacks spatial granularity. Results are demonstrated in both simulation and real-world domains.
RA-RFT: Retrieval-Augmented Reinforcement Fine-Tuning teaches LLMs to reason by analogy
Researchers propose Retrieval-Augmented Reinforcement Fine-Tuning (RA-RFT), a post-training framework that trains a retriever to rank contexts by expected reasoning benefit rather than semantic similarity, then fine-tunes a policy model via reinforcement learning using retrieved analogous demonstrations. The key insight is that reasoning-relevant retrieval surfaces complementary solution strategies rather than superficially similar problems. On mathematical reasoning benchmarks, RA-RFT improves AIME 2025 average@32 accuracy by 7.1 and 2.8 points over GRPO for Qwen3-1.7B and Qwen3-4B respectively, suggesting reasoning-aware retrieval is orthogonal to reward design and training curriculum improvements.
Active policy selection for offline-to-online RL fine-tuning under limited interaction budgets
A new arXiv preprint introduces an active policy selection framework for offline-to-online reinforcement learning (O2O-RL), addressing the problem of choosing which candidate policy to fine-tune when online interactions are scarce or risky. The method uses upper-confidence bounds derived from locally linear performance forecasts to balance exploration (policy evaluation) against exploitation (fine-tuning). Experiments show consistent outperformance over existing O2O-RL baselines. The authors claim this is the first work to formally address active policy selection in the O2O-RL setting.
Imitation learning technique infers red agent policy in partially observable cyber-defense environments
Researchers propose a Policy Learning Technique using imitation learning to infer attacker (red agent) policies from network observations and defender actions in partially observable autonomous cyber environments. The method integrates with neurosymbolic cyber-defense agents that use behavior trees with learning-enabled components. Evaluated across diverse simulated scenarios, the approach achieves high prediction accuracy for red agent actions, improving the defender's ability to anticipate intrusions.