4arXiv cs.LG (Machine Learning)·
Minimal Markov sufficient statistics for POMDPs via stable quotients in holonomy-cover decision processes
A new arXiv preprint characterizes the minimal memory required to restore the Markov property in a structured class of POMDPs called holonomy-cover decision processes, where hidden state evolves via permutations applied to a hidden mode. The authors construct a 'stable quotient' abstraction and prove that the current observation paired with the stable class forms an exact finite Markov state, with a minimality guarantee under reachability and decision-separation conditions. The framework enables 'Holonomy Memory Reinforcement Learning,' which uses the stable class as memory and applies standard finite-MDP RL after synchronization, achieving perfect paired-order accuracy with only three memory states in experiments.
Related guides (1)
Related events (8)
Kolmogorov Regression lifts diffusion policies to Cameron-Martin space for robust long-horizon control
Researchers introduce a backward Kolmogorov equation framework that reformulates diffusion policy training as a deterministic boundary-value PDE problem in Cameron-Martin space, replacing stochastic score matching. The approach uses a precision-weighted Cameron-Martin loss and a Kolmogorov residual as an inference-time failure detector, yielding convergence guarantees tied to kernel effective rank rather than action dimension. Validation on the PushT manipulation benchmark shows 17% improvement in episode reward and 67.6% reduction in inter-step drift; a 6-station manufacturing scheduling task shows 28.4% lower RMSE than LSTM baselines and 96% reduction in deadlock events via Hamilton-Jacobi reachability certification.
Latent Memory Palace: Variational inference framework enables adaptive reasoning for continuous control policies
Researchers introduce Latent Memory Palace (LMP), a method that formulates reasoning for continuous control policies as variational inference over an autoregressive latent distribution, analogous to a memory palace. The approach derives a latent-space reinforcement learning technique to optimize the variational lower bound, yielding a policy (LMP-π) with adaptive test-time compute allocation and a variable-length action tokenizer (LMP-tok) that improves downstream autoregressive policies. The work addresses the gap between language model chain-of-thought reasoning and continuous control, where language-space reasoning lacks spatial granularity. Results are demonstrated in both simulation and real-world domains.
GraphPO: Graph-based Policy Optimization reduces redundancy in LLM reasoning RL
GraphPO is a new reinforcement learning framework that represents reasoning rollouts as directed acyclic graphs rather than independent chains or trees, merging semantically equivalent reasoning paths into equivalence classes to share suffixes and reduce redundant exploration. The approach assigns efficiency advantages to incoming edges and correctness advantages to outgoing edges, deriving process supervision from outcome rewards. Experiments on three LLMs across reasoning and agentic search benchmarks show consistent improvements over chain- and tree-based baselines under equal token or response budgets. The method also provides theoretical guarantees on reduced advantage-estimation variance.
MemCon: Learned adaptive memory management for LLM agents via MDP-based policy
Researchers introduce Memory as a Controlled Process (MemCon), a framework that models LLM agent memory operations as a Markov Decision Process and learns an online policy for adaptive retrieval, plan injection, and memory consolidation. The system is backend-agnostic, wraps existing memory implementations, and uses a lightweight tabular contextual bandit with UCB exploration requiring no pretraining and no additional LLM calls. Evaluated across 6 benchmarks, 3 agent frameworks, and 3 LLM backbones, MemCon outperforms static memory baselines by up to 15.2 points in task success while reducing token consumption by 5–20%.
Solver-dependent Nash equilibrium selection on zero-sum polytopes: regularized methods select max-entropy members
A new arXiv preprint investigates whether different Nash equilibrium solvers systematically select different members of the Nash polytope in two-player zero-sum games. Using six analytically tractable games including Kuhn poker, the authors find that regularized last-iterate methods (R-NaD, magnetic mirror descent) converge to the maximum-entropy Nash equilibrium — interpretable as an information projection — while regret-averaging methods (CFR, CFR+, fictitious play) drift to lower-entropy boundary solutions. The distinction has downstream consequences for performance against sub-optimal opponents in games with sequential or hidden-information structure, with implications for multi-agent AI training and game-solving pipelines.
Physics-aware policy distillation makes deep RL agents interpretable via decision tree surrogates
A new arXiv preprint presents a policy distillation framework that transfers a high-performance TD3 reinforcement learning agent into a shallow decision tree surrogate for interpretable continuous control, demonstrated on the Inverted Pendulum benchmark. The method uses physics-aware features and 'Noisy Oracle Rollouts' to match teacher performance while providing global and local interpretability. A key finding is a fundamental trade-off: discretizing control induces Bang-Bang actuation and a bimodal limit cycle, though BIBO stability is preserved. The work targets safety-critical deployment contexts where regulatory compliance and human-agent trust require explainability.
Purified OPSD fixes on-policy self-distillation failures in long chain-of-thought reasoning models
A new arXiv preprint identifies why on-policy self-distillation (OPSD) consistently degrades long chain-of-thought reasoning models: the teacher's supervision signal is dominated by reference-induced shortcuts rather than question-conditioned, transferable corrections. The authors propose a two-step fix using a reference-only teacher to isolate the non-transferable component and pointwise mutual information (PMI) to construct a cleaner distillation target. Experiments across four long-CoT models on two datasets show consistent improvements over both the base model and standard OPSD while preserving reflective reasoning behavior.
FORE: Fitted occupancy-ratio evaluation for offline RL without Bellman completeness
A new arXiv preprint introduces Fitted Occupancy-Ratio Evaluation (FORE), a fixed-point method for estimating discounted occupancy ratios in offline reinforcement learning and off-policy evaluation. FORE uses an adjoint Bellman recursion and projects onto a log-ratio class in KL divergence, requiring only realizability of the occupancy ratio rather than Bellman completeness or projected-operator stability. The authors establish finite-sample regret bounds and show convergence in KL up to approximation and statistical error, supporting value estimation via reward reweighting, occupancy-weighted fitted Q-evaluation, and doubly robust estimation. The result identifies occupancy-ratio realizability as a sufficient condition for offline policy evaluation, relaxing a standard assumption in the field.