Entity · paper

It Takes a MAESTRO To Prune Bad Experts

paperactiveprovisionalit-takes-a-maestro-to-prune-bad-experts-b32316fd·1 events·first seen 3d ago

Aliases: It Takes a MAESTRO To Prune Bad Experts

Co-occurring entities

More like this (12)

Layer-Adaptive Expert Pruning Mixture of Efficient Experts Expert Tying Instruction-Following Pruning MAESTRO Persona-Pruner Mixture of Experts From Observation to Intervention: A Causal Audit of Expert Importance in Mixture-of-Experts Models Sparse Mixture-of-Experts Expert Blindness Effect Redesign Mixture-of-Experts Routers with Manifold Power Iteration When Good Verifiers Go Bad: Self-Improving VLMs Can Regress on New Tasks

Recent events (1)

6arXiv · cs.CL·3d ago·source ↗

MAESTRO: Markov-chain-based structured pruning framework for Mixture-of-Experts language models

Researchers introduce MAESTRO, a structured pruning method for sparse Mixture-of-Experts (MoE) language models that models expert activation trajectories as Ergodic Markov chains to capture cross-layer routing dependencies. Unlike existing pruning methods adapted from dense transformers, MAESTRO uses stationary distributions of these chains as globally-aware importance heuristics. Under a 50% compression regime evaluated across five domains including safety and ethics, MAESTRO outperforms state-of-the-art baselines by up to 10.61% in average performance retention with lower cross-task variance.

Open Weights Progress Inference Economics MAESTRO It Takes a MAESTRO To Prune Bad Experts