The Token Is a Group Element: On Lie-Algebra Attention over Matrix Lie Groups

Lie-Algebra Attention: tokens as bare matrix Lie group elements with closed-form geometric scores

A new arXiv preprint introduces Lie-Algebra Attention, an attention mechanism where tokens are elements of a matrix Lie group rather than feature vectors, with pairwise attention scores computed as the closed-form algebra norm of the relative pose (log of the group inverse product). The construction achieves equivariance tautologically without representation-theoretic machinery such as irreps, spherical harmonics, or Clebsch-Gordan products, and extends to non-compact affine groups that existing methods cannot handle. Experiments on SE(2), SO(3), and Aff(2) sequence-completion tasks show the closed-form score matches or outperforms learned MLP kernels while using 50–80x fewer score parameters.