4arXiv cs.LG (Machine Learning)·
Online Neural Space-Time Memory enables real-time dynamic novel view synthesis from streaming video
A new arXiv preprint proposes a neural memory architecture for real-time novel view synthesis from multi-view streaming video of dynamic scenes. The method decouples memory update frequency from memory application frequency, using periodic gradient-based updates with per-frame cross-view attention to handle deformations. Two mechanisms — an auxiliary Memory Loss and a Memory Caching strategy — prevent catastrophic forgetting over long contexts. The approach achieves state-of-the-art performance on dynamic human motion scenes with minute-scale online memorization at real-time speeds.
Related guides (1)
Related events (8)
RayDer: Scalable Self-Supervised Novel View Synthesis via Unified Feed-Forward Transformer
RayDer is a unified feed-forward transformer that consolidates camera estimation, scene reconstruction, and rendering into a single backbone for self-supervised novel view synthesis (NVS). By treating dynamic content as a nuisance factor absorbed by a minimal dynamic state, it enables stable training on unconstrained real-world video without requiring dynamic-scene reconstruction. The model exhibits clean power-law scaling with both data and compute across multiple model sizes, and achieves zero-shot open-set performance competitive with supervised state-of-the-art methods on multiple benchmarks.
MemDreamer: Hierarchical graph memory and agentic retrieval for long video understanding
MemDreamer is a plug-and-play framework that decouples perception and reasoning for long-video understanding by incrementally building a three-tier Hierarchical Graph Memory capturing spatiotemporal and causal relations. During inference, a reasoning model uses an Observation-Reason-Action loop with agentic tool-augmented retrieval to navigate the memory graph, constraining the context window to 2% of full-context ingestion while achieving a 12.5-point absolute accuracy gain. The system reaches SOTA on four benchmarks, narrowing the gap with human experts to 3.7 points. The authors also report a strong linear correlation between logical reasoning performance and long-video understanding, proposing agentic capability scaling as a new paradigm for multimodal comprehension.
VideoMLA: Low-Rank Latent KV Cache for Minute-Scale Autoregressive Video Diffusion
VideoMLA applies Multi-Head Latent Attention (MLA) to causal video diffusion, replacing per-head keys and values with a shared low-rank content latent and decoupled 3D-RoPE positional key, achieving 92.7% reduction in per-token KV memory. The paper investigates why MLA works despite pretrained video attention not being low-rank (unlike the spectral assumption motivating MLA in LLMs), finding that the MLA bottleneck itself determines effective rank rather than the pretrained spectrum. On VBench, VideoMLA matches short-horizon baselines, achieves best overall score at long horizons, and delivers 1.23x throughput improvement on a single NVIDIA B200 GPU.
VisualMem: Personal Visual Memory Benchmark and Architecture for Personalized AI Agents
This paper introduces a benchmark and hybrid architecture (VisualMem) for personal visual memory in long-term AI agent memory systems. The work addresses a gap in existing text-centric memory systems by capturing both explicit evidence (recurring user-associated entities) and implicit evidence (latent user facts from visual/multimodal cues) from images. VisualMem augments a text-memory backend with a structured personal visual memory module that uses conversational context to resolve identity, ownership, and durable user facts. Experiments show VisualMem substantially outperforms prior memory systems on the new benchmark while remaining competitive on standard text-memory benchmarks.
ARDY: NVIDIA's streaming autoregressive diffusion framework for real-time controllable 3D human motion generation
NVIDIA researchers introduce ARDY, a streaming motion generation framework that combines autoregressive transformers with diffusion-based denoising to produce high-fidelity 3D human motions in real-time. The system uses a hybrid representation pairing explicit root features with latent body embeddings, enabling online text prompting and flexible long-horizon kinematic constraints simultaneously. ARDY is evaluated on HumanML3D and a large-scale proprietary dataset (Bones Rigplay), with an interactive demo showing dynamic text control, keyframe constraints, and locomotion. The work targets animation, simulation, and humanoid robotics applications where both controllability and inference speed are required.
Survey: Human-View Video Understanding with MLLMs — Watch, Remember, Reason Framework
A new arXiv survey paper proposes a unified 'human-view' framework for analyzing multimodal LLM-based video understanding, organized around three functional abilities: watching (perception), remembering (memory), and reasoning. The authors introduce a formulation characterizing video understanding systems by perceptual representations, memory states, reasoning traces, and predictions, then survey methods, datasets, and benchmarks across these dimensions. The work covers challenges including spatio-temporal perception, long-video processing, streaming understanding, and faithful reasoning, with application domains spanning egocentric, sports, medical, and narrative video.
ManimAgent: Self-evolving multimodal agent with cross-task episodic memory for code generation
ManimAgent is a multimodal agent system that accumulates reflection experience across tasks via a dual-channel Episodic Memory Bank, without weight updates or human-curated seeds. The agent generates Python/Manim animations from scientific paper sections, and a vision-language model scores rendered keyframes to populate positive (success rationales) and negative (failure patterns) memory channels. On a fixed-probe evaluation, Pass@1 improves and reflection rounds decrease as memory grows, outperforming no-memory, RAG, and shuffled-memory baselines. The work addresses a known limitation of single-episode reflection in LLM agents by enabling persistent, self-generated learning across task boundaries.
Supervised Memory Training enables parallel RNN pretraining without backpropagation through time
A new arXiv preprint proposes Supervised Memory Training (SMT), a method that trains recurrent neural networks by reducing the problem to supervised learning on one-step memory transitions, bypassing backpropagation through time entirely. A Transformer-based encoder generates memory labels via a predictive state objective, enabling time-parallel training with O(1) gradient path length between any two tokens. SMT outperforms BPTT on language modeling and pixel sequence modeling tasks across multiple RNN architectures. The approach could enable RNNs to scale more effectively by decoupling memory content from update mechanics.