technique

Manifold Power Iteration

techniqueactiveprovisionalmanifold-power-iteration-e2d33f6a·1 events·first seen 6d ago

Aliases: Manifold Power Iteration

Co-occurring entities

Redesign Mixture-of-Experts Routers with Manifold Power Iteration

More like this (12)

Manifold Redesign Mixture-of-Experts Routers with Manifold Power Iteration Iterated Amplification G*Power power-law scaling Iterative Skill-Aware Decomposition Maximal Update Parameterization (μP)Layer Looping Iterated Prisoner's Dilemma Reflection Loops Multimodal Continual Instruction Tuning Divide-and-Conquer Partitioning

Recent events (1)

5arXiv · cs.CL·6d ago·source ↗

Manifold Power Iteration redesigns MoE routers by aligning rows with expert singular directions

A new arXiv preprint proposes Manifold Power Iteration (MPI), a principled redesign of Mixture-of-Experts router matrices that aligns each router row with the principal singular direction of its associated expert. The method uses a 'Power-then-Retract' paradigm to enforce norm constraints while driving convergence toward these singular directions. Empirical validation spans MoE pretraining at scales from 1B to 11B parameters, showing improved model effectiveness.

Training Infrastructure Frontier Model Releases Redesign Mixture-of-Experts Routers with Manifold Power Iteration Manifold Power Iteration