4arXiv cs.AI (Artificial Intelligence)·
SM4RT: Structured Motion 4D Reconstruction Transformer for monocular RGB video
SM4RT is a new transformer architecture for end-to-end 4D reconstruction from monocular RGB video that models scene motion as a structured set of rigid-body transformations in SE(3) rather than independent point-wise displacements. The method decomposes scene dynamics into a compact set of motion bases represented as temporal sequences of 6D twists, with per-pixel assignment weights ensuring physically coherent rigid-body trajectories. A parallel motion geometry encoder-decoder jointly infers 3D geometry, world-coordinate motion, and kinematic structure in a single forward pass. The work targets a known gap in Geometry Foundation Models, which have advanced monocular 3D reconstruction but struggled with dynamic 4D understanding.
Related guides (1)
Related events (8)
D4RT: DeepMind's Unified 4D Reconstruction and Tracking System, Up to 300x Faster
DeepMind has announced D4RT, a system for unified four-dimensional (spatial + temporal) scene reconstruction and tracking. The method claims up to 300x speed improvements over prior approaches. The announcement positions D4RT as a significant efficiency advance in dynamic 3D scene understanding, with potential applications in robotics, video understanding, and embodied AI.
HAT-4D: Agentic framework for 4D multi-object interaction reconstruction from monocular video
HAT-4D is a new agentic framework that reconstructs 3D geometry, temporal dynamics, and physical interactions of multiple objects from single monocular videos, targeting scalable data collection for Embodied AI and Vision-Language-Action (VLA) model training. The system integrates VLMs with a multi-level human-in-the-loop feedback mechanism to resolve depth ambiguities and occlusions without expensive multi-camera rigs. The authors also introduce MVOIK-4D, an open-world benchmark for monocular 4D interaction reconstruction with a novel evaluation protocol focused on physical plausibility and temporal consistency. Experiments show state-of-the-art performance on most metrics, and HAT-4D-generated data improves downstream model fine-tuning.
World from Motion: generative dynamic 3D Gaussian reconstruction from monocular video
Researchers present World from Motion, a method that generates freely renderable dynamic 3D Gaussian Splatting (3DGS) representations from monocular video by conditioning a video model on pixel-aligned renderings encoding appearance, geometry, and 3D scene motion. The approach trains on aligned multiview video pairs with simulated monocular reconstruction artifacts, then distills model generations back into a consistent dynamic 3DGS at test time. The method claims state-of-the-art results in 4D reconstruction and generalizes to in-the-wild videos with large viewpoint changes and dynamic content.
Illumination-robust rPPG heart-rate estimation via spatial-temporal transformer for robot-mounted cameras
A new arXiv paper presents an end-to-end spatial-temporal transformer framework for remote photoplethysmography (rPPG) heart-rate estimation that is robust to illumination variation, targeting robot-mounted RGB cameras. The system integrates 3D face alignment, illumination augmentation, a Residual Temporal Standardization Module, and a hybrid waveform-plus-spectral loss. On a new dataset spanning three illumination levels, the method achieves 0.79 bpm MAE and 0.982 HR correlation, reducing error by 93.6% relative to the PhysFormer baseline. The work is relevant to physiological sensing in service and assistive robotics.
IVGT: Implicit Visual Geometry Transformer for Neural Scene Representation
IVGT is a new neural architecture that implicitly models continuous 3D geometry from unposed multi-view images without requiring explicit pointmap regression. It learns a continuous neural scene representation in a canonical coordinate system, supporting SDF-based surface queries and color prediction via lightweight decoders. The model is trained with multi-dataset joint optimization using 2D supervision and 3D geometric regularization, achieving strong generalization across mesh reconstruction, novel view synthesis, depth/normal estimation, and camera pose estimation tasks.
SemRF: Semantic Reference Frames for analyzing residual-stream dynamics in language models
A new arXiv preprint introduces Semantic Reference Frames (SemRF), a formal framework for analyzing how language model computation evolves across transformer layers via the residual stream. The method uses anchor-based coordinates with pseudo-inverse tying to separate semantic measurement from residual dynamics, enabling stable cross-layer comparison without measurement drift. The framework defines layerwise semantic trajectories, Voronoi-based coarse cell assignments, and a minimum-action canonical trace, with theoretical links to parameter efficiency and knowledge density. This is a mechanistic interpretability contribution aimed at understanding internal model computation.
DynaFLIP: Dynamics-Aware Multimodal Pre-Training for Robot Manipulation Perception
DynaFLIP is a pre-training framework that injects motion understanding into visual encoders for robot manipulation by constructing image-language-3D flow triplets from human and robot videos. The method encourages tri-modal alignment via simplex-volume minimization in a shared hyperspherical space, combined with cosine regularization and contrastive objectives. The resulting dynamics-aware visual backbone consistently outperforms baselines across diverse downstream policies including VLAs, with gains up to +22.5% in out-of-distribution scenarios. The work argues that robot generalization requires encoding how the world changes under action, not just static scene content.
AnyMo: Geometry-Aware Setup-Agnostic Framework for Wearable IMU Human Motion Understanding
AnyMo is a geometry-aware framework that addresses the setup-dependence problem in wearable IMU-based human motion modeling by using physics-grounded simulation over dense body-surface placements to generate synthetic training signals. It pre-trains a graph encoder from synthetic placement views and masked partial observations, then tokenizes multi-position IMU data into full-body motion tokens aligned with an LLM for motion-language understanding. Evaluated across zero-shot activity recognition (14 unseen datasets), cross-modal retrieval, and motion captioning, AnyMo improves average Accuracy/F1 by ~11.7%/11.6%, zero-shot retrieval MRR by 15.9–28.6%, and captioning BERT-F1 by 18.8%. The work positions itself as a generalist model for wearable motion understanding transferable across devices and sensing configurations.