falcon-mamba-222421a8·2 events·first seen Aliases: Falcon Mamba, Falcon-Mamba
A new arXiv paper introduces an exact analytical instrument for measuring how selective state-space models (Mamba-1, Mamba-2, Falcon-Mamba) allocate their state modes across inputs, using a per-(layer, channel, window) Gram tensor that predicts pruning error to near-machine-precision. The key empirical finding is that trained models dynamically reallocate which modes carry signal depending on the input context, and that this migration is driven primarily by the input-dependent write map B_t rather than the timestep parameter typically associated with selectivity. Input-scheduled mode pruning based on this instrument outperforms static and Hankel-based rankings at all scales tested (130M–7B), matching unpruned model quality at half the state budget in offline evaluation.
Technology Innovation Institute (TII) releases Falcon Mamba, a 7B parameter state space model (SSM) based on the Mamba architecture, announced as the first attention-free model at this scale to match or exceed transformer-based models on standard benchmarks. The model is hosted on Hugging Face and represents a significant milestone for SSM-based architectures competing with transformers. This release advances the case for pure SSM models as viable alternatives to attention-based LLMs at the 7B scale.