4arXiv cs.CL (Computation and Language)·
Tokenizer transplantation fixes autoregressive collapse in Bengali edge ASR
Researchers identify that English-centric byte-level tokenizers cause catastrophic autoregressive collapse when lightweight ASR models like Moonshine are applied to morphologically rich languages like Bengali. They propose a vocabulary transplantation pipeline that swaps in BanglaBERT's WordPiece vocabulary, reducing token fertility from 9.16 to 1.30 and cutting autoregressive sequence length by 85.8%. The modified model achieves 21.54% WER and an RTF of 0.0053 on the 882-hour Lipi-Ghor dataset, offering a reproducible blueprint for cross-script adaptation of compact ASR models without retraining from scratch.
Related guides (1)
Related events (8)
ToaST: Tokenization with Split Trees Reduces Token Count by 11%+ Over BPE/WordPiece/UnigramLM
ToaST (Tokenization with Split Trees) is a new subword tokenization method that uses a recursive binary split-tree inference procedure and Integer Programming-based vocabulary selection to directly optimize compression. On English text, ToaST reduces token counts by more than 11% compared to BPE, WordPiece, and UnigramLM at vocabulary sizes of 40,960 and above, effectively extending context length for models using it. In 1.5B parameter LM training experiments, ToaST achieves the highest CORE benchmark score, outperforming baselines by 2.6%–7.6% across 22 tasks. The LP relaxation of the vocabulary selection IP is near-integral in practice, yielding provably near-optimal vocabularies.
LangMAP: Language-adaptive tokenization from a shared vocabulary without language identification at inference
LangMAP (Language-adaptive Maximum a Posteriori Tokenization) extends the UnigramLM algorithm to produce language-specific tokenizations from a single shared vocabulary, eliminating the need to retrain models or swap vocabularies for multilingual settings. A key property is that language labels are only required at training time; inference proceeds without language identification. Evaluated across 14 tokenizers, 9 natural languages, and 9 programming languages, LangMAP improves morphological boundary alignment and AST-leaf alignment for all coding languages tested. Fine-tuning results are mixed: consistent gains on grammatical acceptability (MultiBLiMP) but less consistent on knowledge tasks (Global-PIQA, Belebele).
Cross-lingual transfer from Sinhala improves low-resource Dhivehi ASR
A new arXiv paper investigates cross-lingual transfer learning from Sinhala to improve automatic speech recognition for Dhivehi, a severely low-resource language of the Maldives. Across 17 experiments spanning five transfer paradigms, the best system—continual pre-training on Sinhala followed by Dhivehi fine-tuning with KenLM—achieves 12.89% WER, outperforming the Dhivehi-only baseline by 13.50% WER. A Turkish control experiment confirms that linguistic relatedness, not just additional data, drives the improvement.
MinGram: Simplified Unigram tokenizer with better compression and morphological alignment than BPE
Researchers introduce MinGram, a minimalist variant of the Unigram tokenizer that replaces the standard training procedure (suffix arrays, forward-backward pass, iterative pruning) with a BPE-derived seed vocabulary, Hard EM on a minimum-token path, and a single pruning step. Evaluated across six languages, MinGram achieves better compression than both BPE and standard Unigram while retaining strong morphological alignment. In downstream language model training, MinGram and other Unigram-family tokenizers consistently outperform BPE in bits-per-byte.
CTC oracle gap anatomy: acoustic scoring saturates, linguistic MBR decoding recovers WER
A new arXiv paper systematically diagnoses why CTC-internal N-best rescoring fails to improve over greedy decoding on LibriSpeech, showing that blank-path proliferation causes a 53% degradation in rank correlation between CTC scores and WER as beam size grows. The authors demonstrate that the bottleneck is linguistic rather than acoustic: MBR decoding with RoBERTa pseudo-log-likelihood achieves 9% relative WER reduction on LibriSpeech test-other and generalizes across two architectures and three domains. The paper also analyzes MWER sequence-level fine-tuning failure at near-converged checkpoints, attributing collapse to a vanishingly small training oracle gap.
LIHA reveals first-token broadcaster heads as mechanistic source of language identity in transformers
Researchers introduce Language Identity Head Ablation (LIHA), a causal intervention that zeros individual attention heads to measure language-switching behavior across 2,700 prompt-language pairs in seven languages. Applied to GPT-2, LIHA identifies a small set of 'first-token broadcaster' heads that propagate language identity signals throughout generation, with compensatory redistribution following a hierarchical, feedforward pattern. A controlled comparison between Qwen2.5-1.5B-Base and Qwen2.5-1.5B-Instruct provides direct causal evidence that instruction tuning reorganizes language identity circuits toward early-layer localization. The findings offer mechanistic grounding for why multilingual models generate in the wrong language and why this is difficult to correct.
Iterative pseudo-labeling approach improves Mandarin-English code-switching ASR
A new arXiv preprint introduces a three-phase iterative pseudo-labeling framework for code-switching automatic speech recognition (ASR), applied here to Mandarin-English mixing. The method generates pseudo-labels from unlabeled corpora, trains a bilingual model in two stages, and iteratively refines it, achieving Mix Error Rate reductions of 6.35% and 8.29% on the SEAME benchmark's devman and devsge subsets. This is the first application of iterative pseudo-labeling to code-switching ASR, addressing the chronic data scarcity problem in this domain.
Adaptive asymmetric token compression accelerates time series language models up to 7.68×
A new arXiv preprint proposes an adaptive token budgeting framework for time series (TS) language models that compresses TS tokens using frequency-domain structure and progressively prunes prompt tokens across model layers. The authors demonstrate up to 7.68× inference acceleration with performance improvements in 78% of evaluated settings across forecasting, classification, imputation, and anomaly detection tasks. The work is motivated by the observation that TS tokens have uneven spectral contributions and prompt-token influence attenuates with model depth, making uniform token processing wasteful.