mingram-a-minimalist-unigram-tokenizer-with-high-compression-and-competitive-morphological-alignment-00fb5416·1 events·first seen Aliases: MinGram: A Minimalist Unigram Tokenizer with High Compression and Competitive Morphological Alignment
Researchers introduce MinGram, a minimalist variant of the Unigram tokenizer that replaces the standard training procedure (suffix arrays, forward-backward pass, iterative pruning) with a BPE-derived seed vocabulary, Hard EM on a minimum-token path, and a single pruning step. Evaluated across six languages, MinGram achieves better compression than both BPE and standard Unigram while retaining strong morphological alignment. In downstream language model training, MinGram and other Unigram-family tokenizers consistently outperform BPE in bits-per-byte.