Falcon Mamba: First Strong Attention-Free 7B Model
Technology Innovation Institute (TII) releases Falcon Mamba, a 7B parameter state space model (SSM) based on the Mamba architecture, announced as the first attention-free model at this scale to match or exceed transformer-based models on standard benchmarks. The model is hosted on Hugging Face and represents a significant milestone for SSM-based architectures competing with transformers. This release advances the case for pure SSM models as viable alternatives to attention-based LLMs at the 7B scale.
Related guides (3)
Related events (8)
Bamba: Inference-Efficient Hybrid Mamba2 Model
Hugging Face published a blog post introducing Bamba, a hybrid architecture combining Mamba2 state-space layers with attention layers, designed for inference efficiency. The model targets reduced KV-cache memory and improved throughput compared to pure transformer architectures. The post covers architecture details, training approach, and benchmarking results positioning Bamba as a practical alternative for deployment-constrained settings.
Codestral Mamba: Mistral AI Releases Apache 2.0 Mamba-Architecture Code Model
Mistral AI has released Codestral Mamba, a 7.3B-parameter code-focused language model built on the Mamba state-space architecture rather than the Transformer architecture. The model offers linear-time inference and theoretically infinite sequence length, tested up to 256k tokens in-context retrieval. Developed with Mamba co-creators Albert Gu and Tri Dao, it is released under Apache 2.0 and available via HuggingFace, mistral-inference SDK, TensorRT-LLM, and Mistral's la Plateforme API. Mistral positions it as a local code assistant that performs on par with state-of-the-art transformer-based code models.
Falcon-H1: A Family of Hybrid-Head Language Models Redefining Efficiency and Performance
TII UAE has released Falcon-H1, a new family of hybrid-head language models combining attention and state-space mechanisms to improve efficiency and performance. The models are published on Hugging Face and represent TII's latest iteration in the Falcon series. The hybrid architecture targets better inference economics and competitive benchmark results relative to model size.
Nvidia Nemotron 3 Ultra: hybrid Mamba-transformer open-weights model targeting agentic workloads
Nvidia released Nemotron 3 Ultra, a 550B parameter (55B active) hybrid Mamba-transformer mixture-of-experts model with a 1M token context window, publishing weights, training data, and RL environments under an open license. The model ranks as the highest-scoring U.S. open-weights model on the Artificial Analysis Intelligence Index (47.7-48.2) and is approximately three times faster than comparable open-weights rivals, though it trails leading Chinese models like Kimi K2.6 and DeepSeek V4 Pro on intelligence benchmarks. Nvidia used a novel Multi-Teacher On-Policy Distillation approach with 10+ specialized teacher models and trained using NVFP4 quantization. The release is strategically motivated by Nvidia's interest in a healthy open-weights ecosystem that drives AI semiconductor adoption.
Falcon 180B Released: New Open-Weights Frontier Model
Technology Innovation Institute (TII) has released Falcon 180B, a 180-billion parameter open-weights language model announced via Hugging Face. At the time of release, it was positioned as the largest publicly available open-weights model, trained on 3.5 trillion tokens. The model is available on Hugging Face Hub for research and commercial use under a custom license.
Nvidia releases Nemotron 3 Super 120B-A12B open-weights model with hybrid Mamba-2/MoE architecture
Nvidia released Nemotron 3 Super 120B-A12B, an open-weights LLM with a hybrid Mamba-2/transformer/MoE architecture that activates only 12B parameters per token and supports up to 1 million token context. The model claims the fastest inference speed in its size class at 442 tokens/second and leads open-weights models on PinchBench agentic task evaluation, outperforming larger models including Kimi K2.5 (1T parameters). Nvidia is releasing weights, training data, and recipes under a permissive commercial license, and plans a $26B five-year investment in open-weights models — framed partly as a strategic response to Chinese labs building capable open-weights models on non-Nvidia hardware.
Falcon 2: 11B Parameter Pretrained LLM and VLM Trained on 5T+ Tokens Across 11 Languages
Technology Innovation Institute (TII) has released Falcon 2, an 11B parameter language model pretrained on over 5 trillion tokens spanning 11 languages. The release includes both a base language model and a vision-language model (VLM) variant. This represents a significant update to the Falcon model family, expanding multilingual and multimodal capabilities.
Qwen1.5-MoE: Matching 7B Model Performance with 1/3 Activated Parameters
Alibaba's Qwen team releases Qwen1.5-MoE-A2.7B, a mixture-of-experts model with only 2.7 billion activated parameters that claims performance parity with 7B dense models such as Mistral 7B and Qwen1.5-7B. The model activates roughly one-third of its total parameters during inference, offering significant compute efficiency gains. This release follows growing industry interest in MoE architectures sparked by Mixtral, and the model is available on GitHub, HuggingFace, and ModelScope.