5arXiv cs.LG (Machine Learning)·
Exact instrument for state-mode usage in Mamba SSMs reveals input-driven state migration
A new arXiv paper introduces an exact analytical instrument for measuring how selective state-space models (Mamba-1, Mamba-2, Falcon-Mamba) allocate their state modes across inputs, using a per-(layer, channel, window) Gram tensor that predicts pruning error to near-machine-precision. The key empirical finding is that trained models dynamically reallocate which modes carry signal depending on the input context, and that this migration is driven primarily by the input-dependent write map B_t rather than the timestep parameter typically associated with selectivity. Input-scheduled mode pruning based on this instrument outperforms static and Hankel-based rankings at all scales tested (130M–7B), matching unpruned model quality at half the state budget in offline evaluation.
Related guides (3)
Related events (8)
Falcon Mamba: First Strong Attention-Free 7B Model
Technology Innovation Institute (TII) releases Falcon Mamba, a 7B parameter state space model (SSM) based on the Mamba architecture, announced as the first attention-free model at this scale to match or exceed transformer-based models on standard benchmarks. The model is hosted on Hugging Face and represents a significant milestone for SSM-based architectures competing with transformers. This release advances the case for pure SSM models as viable alternatives to attention-based LLMs at the 7B scale.
Formal theory of when tool use increases expressive power of finite-precision recurrent models
A new arXiv paper provides an exact, architecture-level characterization of when external tool access increases the computational expressivity of finite-precision recurrent sequence models, including SSMs. The key dichotomy: finite-state tools add essentially nothing (absorbable at logarithmic bit cost), while a single infinite-state tool (a read/write tape) makes the system Turing complete with only O(log|Q| + log|Γ|) bits of controller state. The paper further shows this construction is realized by a natural one-layer finite-precision selective affine SSM, with selectivity identified as essential.
Bamba: Inference-Efficient Hybrid Mamba2 Model
Hugging Face published a blog post introducing Bamba, a hybrid architecture combining Mamba2 state-space layers with attention layers, designed for inference efficiency. The model targets reduced KV-cache memory and improved throughput compared to pure transformer architectures. The post covers architecture details, training approach, and benchmarking results positioning Bamba as a practical alternative for deployment-constrained settings.
MAESTRO: Markov-chain-based structured pruning framework for Mixture-of-Experts language models
Researchers introduce MAESTRO, a structured pruning method for sparse Mixture-of-Experts (MoE) language models that models expert activation trajectories as Ergodic Markov chains to capture cross-layer routing dependencies. Unlike existing pruning methods adapted from dense transformers, MAESTRO uses stationary distributions of these chains as globally-aware importance heuristics. Under a 50% compression regime evaluated across five domains including safety and ethics, MAESTRO outperforms state-of-the-art baselines by up to 10.61% in average performance retention with lower cross-task variance.
Signed-Permutation Gauge Theory for RMSNorm Transformers Improves Coordinate Transport
A new arXiv preprint formalizes the residual-stream gauge symmetry of transformer architectures, showing that RMSNorm models have a signed-permutation gauge group B_d = S_d ⋉ {±1}^d rather than the permutation-only S_d of LayerNorm models. The authors introduce sign-marginalized Hungarian matching and demonstrate that coordinate-preserving transport along fine-tuning trajectories recovers 91.1% of cross-run coordinates versus 60.3% for endpoint matching. Practical consequences include dramatically improved sparse autoencoder reconstruction (NMSE 0.004 vs 1.08), preserved steering vector effects, and correct AdamW optimizer state transfer — with implications for mechanistic interpretability, model merging, and activation engineering.
SAEs as Stethoscopes: Interpretability-Guided Layer Selection for Task Vector Model Editing
This paper evaluates a Sparse Autoencoder (SAE)-guided model editing pipeline for mathematical reasoning on Gemma-3-4B-IT, finding that projecting task vectors onto SAE feature subspaces discards ~97% of modification energy due to geometric misalignment between activation-space SAE directions and weight-space task vectors. The authors reframe SAEs as diagnostic tools ('stethoscopes') rather than intervention filters ('scalpels'), using SAE-derived specificity scores to identify which layers to inject unfiltered task vectors into. This approach improves Number Theory accuracy from 29.6% to 39.4% on Minerva Math (p=0.0007), with 5 of 7 math subjects significantly improved and none degraded. The method is fully deterministic and adds no inference cost.
Causal audit finds routing statistics do not predict expert importance in MoE pruning
A new arXiv paper conducts a token-level interventional audit of Mixture-of-Experts (MoE) pruning heuristics across three architectures (OLMoE-1B-7B, Qwen1.5-MoE, DeepSeek-V2-Lite), finding that no standard observational metric — utilization rates, activation norms, routing weight distributions — reliably predicts which experts can be removed without functional cost. Effect sizes fall below Cohen's d = 0.17 across all 60 metric-layer combinations after multiple-comparison correction, with only a single significant signal at OLMoE's final layer. The authors argue that existing pruning methods succeed not because they identify dispensable experts but because early-layer redundancy makes most selection criteria interchangeable. The work frames this as a concrete counterexample to the broader interpretability practice of treating associational (rung-1) evidence as interventional (rung-2) conclusions.
Mechanistic interpretability study reveals geometric structure of LLM-as-judge bias in hidden states
A new arXiv preprint investigates LLM-as-judge scoring bias at the representation level rather than the input-output level, studying seven judge models across seven bias types and nine benchmarks. The authors find that biased inputs are displaced along low-dimensional, type-specific subspaces in activation space, and that steering hidden states along these subspaces causally controls scoring direction. A linear projection onto bias-direction features predicts judge failures on unseen benchmarks, substantially outperforming text-based alternatives. The work provides a mechanistic account that unifies geometric structure, causal control, and operational prediction within a single framework.