4arXiv cs.CL (Computation and Language)·
MinGram: Simplified Unigram tokenizer with better compression and morphological alignment than BPE
Researchers introduce MinGram, a minimalist variant of the Unigram tokenizer that replaces the standard training procedure (suffix arrays, forward-backward pass, iterative pruning) with a BPE-derived seed vocabulary, Hard EM on a minimum-token path, and a single pruning step. Evaluated across six languages, MinGram achieves better compression than both BPE and standard Unigram while retaining strong morphological alignment. In downstream language model training, MinGram and other Unigram-family tokenizers consistently outperform BPE in bits-per-byte.
Related guides (1)
Related events (8)
ToaST: Tokenization with Split Trees Reduces Token Count by 11%+ Over BPE/WordPiece/UnigramLM
ToaST (Tokenization with Split Trees) is a new subword tokenization method that uses a recursive binary split-tree inference procedure and Integer Programming-based vocabulary selection to directly optimize compression. On English text, ToaST reduces token counts by more than 11% compared to BPE, WordPiece, and UnigramLM at vocabulary sizes of 40,960 and above, effectively extending context length for models using it. In 1.5B parameter LM training experiments, ToaST achieves the highest CORE benchmark score, outperforming baselines by 2.6%–7.6% across 22 tasks. The LP relaxation of the vocabulary selection IP is near-integral in practice, yielding provably near-optimal vocabularies.
LangMAP: Language-adaptive tokenization from a shared vocabulary without language identification at inference
LangMAP (Language-adaptive Maximum a Posteriori Tokenization) extends the UnigramLM algorithm to produce language-specific tokenizations from a single shared vocabulary, eliminating the need to retrain models or swap vocabularies for multilingual settings. A key property is that language labels are only required at training time; inference proceeds without language identification. Evaluated across 14 tokenizers, 9 natural languages, and 9 programming languages, LangMAP improves morphological boundary alignment and AST-leaf alignment for all coding languages tested. Fine-tuning results are mixed: consistent gains on grammatical acceptability (MultiBLiMP) but less consistent on knowledge tasks (Global-PIQA, Belebele).
ConvexTok: Tokeniser Construction via Linear Programming and Convex Optimisation
This paper proposes ConvexTok, a new tokenisation algorithm that formulates vocabulary construction as a linear program solved via convex optimisation, replacing the greedy approaches used by BPE and Unigram. ConvexTok consistently improves intrinsic tokenisation metrics and bits-per-byte (BpB) for language models, with less consistent gains on downstream tasks. A key feature is the ability to certify proximity to optimality via a lower bound, with empirical results showing the algorithm is within 1% of optimal at common vocabulary sizes.
Tokenizer transplantation fixes autoregressive collapse in Bengali edge ASR
Researchers identify that English-centric byte-level tokenizers cause catastrophic autoregressive collapse when lightweight ASR models like Moonshine are applied to morphologically rich languages like Bengali. They propose a vocabulary transplantation pipeline that swaps in BanglaBERT's WordPiece vocabulary, reducing token fertility from 9.16 to 1.30 and cutting autoregressive sequence length by 85.8%. The modified model achieves 21.54% WER and an RTF of 0.0053 on the 882-hour Lipi-Ghor dataset, offering a reproducible blueprint for cross-script adaptation of compact ASR models without retraining from scratch.
Multilingual word-level forced alignment using MMS and learned dynamic programming outperforms MFA
Researchers present a forced alignment system combining Meta's Massively Multilingual Speech (MMS) model with a self-supervised phoneme boundary detector (UnSupSeg) and a learned dynamic programming decoder. Trained on TIMIT and Buckeye, the system outperforms Montreal Forced Aligner and MMS-based alignment on both datasets and generalizes to unseen languages (Dutch, German, Hebrew) without additional training. The approach claims potential to scale to 1100+ languages supported by MMS, making it relevant for low-resource speech processing pipelines.
Attention Expansion mechanism improves keyphrase extraction from long documents without full-context LLMs
Researchers propose an 'attention expansion' mechanism that augments pre-trained language model token representations with information from out-of-context chunks using static word embeddings, enabling more effective keyphrase extraction from long documents. The approach avoids the computational cost of full-document attention or LLM-based inference while expanding the effective contextual scope of PLM-based models. Evaluated across five PLM backbones and five benchmark corpora, the method consistently improves F1 scores over state-of-the-art baselines in both scientific and news domains.
BitNet Text Embeddings: Ternary-weight LLM encoders with multi-precision output vectors
BITEMBED is a new framework that converts pretrained LLM backbones (tested on Qwen3-0.6B and Gemma3-270M) into BitNet-style text embedding encoders using ternary weights, quantized activations, and lightweight normalization refinement. The system applies continual contrastive pre-training and supervised fine-tuning with similarity-distribution and attention-relation distillation from a full-precision teacher. Evaluated on MMTEB (English v2), BITEMBED achieves performance largely comparable to full-precision teacher embedders while supporting flexible output embedding precisions to trade off storage cost. The work targets the dual bottleneck of inference compute and vector index storage in large-scale retrieval systems.
UniAudio-Token: Semantic Speech Tokenizer with General Audio Perception for Audio-LLMs
UniAudio-Token is a framework from Tencent that extends semantic speech tokenizers—commonly used as interfaces for Audio-LLMs—to support general audio perception without sacrificing speech quality. It introduces two mechanisms: Semantic-Acoustic Primitives (SAP) for structured supervision decomposing audio into linguistic, vocal, and auditory-scene components, and Semantic-Acoustic Equilibrium (SAE), a content-aware gating mechanism that restores fine-grained acoustic details from shallow layers. Evaluations show it outperforms all single-codebook baseline tokenizers on both understanding and generation tasks when integrated with downstream LLMs. Code, training/inference scripts, and model checkpoints are publicly released.