Walrus: A 1.3B-Parameter General Transformer for Fluid Dynamics Simulation
Polymathic AI Collaboration released Walrus, a 1.3 billion-parameter transformer model that simulates fluids, gases, and plasmas across 19 physical domains, outperforming prior specialized physics models. The model addresses aliasing artifacts in transformers—errors that compound at specific spatial locations over time—by randomly jittering input data at each time step before encoding, distributing errors rather than allowing accumulation. Walrus achieved lowest VRMSE in 18 of 19 domains for one-step predictions, reducing error by 63.6% on average versus best competing models. The jittering technique may generalize to vision and video transformer architectures where similar pixelation artifacts occur.
Related guides (3)
Related events (8)
Looped World Models introduce iterative latent depth as a new scaling axis for world simulation
A new arXiv preprint introduces Looped World Models (LoopWM), a parameter-shared transformer architecture that iteratively refines latent environment states to achieve up to 100x parameter efficiency over conventional world models. The approach uses adaptive computation to scale depth dynamically per prediction step, addressing the tension between long-horizon simulation fidelity and deployment cost. The authors position iterative latent depth as a new scaling axis orthogonal to model size and training data.
Falcon Mamba: First Strong Attention-Free 7B Model
Technology Innovation Institute (TII) releases Falcon Mamba, a 7B parameter state space model (SSM) based on the Mamba architecture, announced as the first attention-free model at this scale to match or exceed transformer-based models on standard benchmarks. The model is hosted on Hugging Face and represents a significant milestone for SSM-based architectures competing with transformers. This release advances the case for pure SSM models as viable alternatives to attention-based LLMs at the 7B scale.
Nvidia Nemotron 3 Ultra: hybrid Mamba-transformer open-weights model targeting agentic workloads
Nvidia released Nemotron 3 Ultra, a 550B parameter (55B active) hybrid Mamba-transformer mixture-of-experts model with a 1M token context window, publishing weights, training data, and RL environments under an open license. The model ranks as the highest-scoring U.S. open-weights model on the Artificial Analysis Intelligence Index (47.7-48.2) and is approximately three times faster than comparable open-weights rivals, though it trails leading Chinese models like Kimi K2.6 and DeepSeek V4 Pro on intelligence benchmarks. Nvidia used a novel Multi-Teacher On-Policy Distillation approach with 10+ specialized teacher models and trained using NVFP4 quantization. The release is strategically motivated by Nvidia's interest in a healthy open-weights ecosystem that drives AI semiconductor adoption.
AHA-WAM: Asynchronous world-action modeling with temporal decoupling for robot manipulation
AHA-WAM introduces a dual Diffusion Transformer architecture that decouples world prediction (low-frequency) from action execution (high-frequency) in robot manipulation policies, addressing the inefficiency of existing world-action models that force both branches to operate at the same temporal resolution. The system uses a rolling key-value memory video DiT as a long-horizon scene planner and a fast action DiT that queries layerwise latent context via joint attention, with Observation-Guided Video-Context Routing enabling asynchronous execution. On RoboTwin benchmarks, AHA-WAM achieves 92.80% average success and 78.3% on real-world tasks at 24.17 Hz, a 4.59x speedup over Fast-WAM, without robot-data pretraining.
Variable-Width Transformers: X-shaped architecture outperforms uniform-width baselines with 22% fewer FLOPs
Researchers propose the ><former (X-shaped transformer), a decoder-only architecture that uses wider early and late layers with narrower middle layers, implemented via a parameter-free residual resizing mechanism. Evaluated on models from 200M to 2B dense parameters and 3B MoE, the architecture consistently outperforms parameter-matched uniform-width baselines on language modeling loss. The design yields a 22% reduction in FLOPs and 15% reduction in KV cache memory under fitted scaling curves, suggesting nonuniform width allocation is a viable path to more compute-efficient language models.
Memory-efficient Diffusion Transformers with Quanto and Diffusers
This Hugging Face blog post describes integrating the Quanto quantization library with the Diffusers framework to reduce memory requirements for diffusion transformer models. The approach enables running large image/video generation models on consumer-grade hardware by applying int8 and int4 quantization to model weights. The post covers practical implementation details and benchmarks showing memory savings for models like Flux and others in the diffusion transformer family.
Dynamic short convolutions yield 1.33–1.60× compute advantage over standard Transformers
A new arXiv preprint introduces dynamic short convolutions as an architectural primitive for Transformers, using input-dependent filters to combine locality bias with increased expressivity. Experiments across 150M–2B parameter language models show consistent perplexity improvements over standard Transformers and static convolution variants, with scaling-law fits indicating a 1.33× compute advantage when applied to key/query/value vectors and 1.60× when added after every linear layer. The technique also improves linear RNNs (Mamba-2, Gated DeltaNet) and mixture-of-experts architectures, with custom Triton kernels making training practical.
Mistral AI Highlights Physics AI Research Portfolio Following Emmi AI Acquisition
Mistral AI has published a research overview showcasing Physics AI work stemming from its acquisition of Emmi AI, targeting industrial engineering domains including aerospace, automotive, semiconductors, and energy. The portfolio spans neural surrogate models for CFD simulations, plasma turbulence modeling for nuclear fusion, and universal physics transformers capable of handling large-scale multi-physics processes. Key published works include AB-UPT (Anchored-Branched Universal Physics Transformer), NeuralDEM, UPT, and GyroSwin, several with open-source code. The announcement signals Mistral's strategic push into scientific and industrial AI beyond language modeling.