Entity · paper

In-Place Tokenizer Expansion for Pre-trained LLMs

paperactiveprovisionalin-place-tokenizer-expansion-for-pre-trained-llms-d04c59dd·1 events·first seen 3d ago

Aliases: In-Place Tokenizer Expansion for Pre-trained LLMs

Co-occurring entities

More like this (12)

Continual LLM Upcycling: A Predictor-Gated Bank-Wise Sparsity Training Recipe for Dense-to-Sparse LLMs CLP: Collocation-Length Prediction for Zero-Loss Adaptive Multi-Token Inference PC Layer: Polynomial Weight Preconditioning for Improving LLM Pre-Training Extending LLM Context via Associative Recurrent Memory Leveraging Audio-LLMs to Filter Speech-to-Speech Training Data Tokenizers intra-frame token sparsification ExpRL: Exploratory RL for LLM Mid-Training From Tokens to States: LLMs as a Special Case of World Models and the Continuous Path Beyond Beyond Uniform Tokens: Adaptive Compression for Time Series Language Models XLM-RoBERTa UnigramLM

Recent events (1)

5arXiv · cs.LG·3d ago·source ↗

In-place tokenizer expansion recipe enables multilingual efficiency gains in pre-trained LLMs

Researchers present a method for expanding a pre-trained LLM's tokenizer without retraining from scratch, using continued BPE merges on a multilingual corpus and a two-stage adaptation procedure (embedding-only training followed by full continued pre-training). Applied to LFM2-8B-A1B (a Mixture-of-Experts model) to produce LFM2.5-8B-A1B with a 128K tokenizer, the approach achieves 2.4–2.6× token compression for Hindi and Vietnamese (up to 4× for Thai), translating to an estimated 2.2–3.7× per-character decode speedup on reference devices. The technique is particularly relevant for on-device models where embedding and LM-head matrices constitute a material fraction of decode bandwidth, making vocabulary size a real efficiency constraint.

Open Weights Progress Inference Economics In-Place Tokenizer Expansion for Pre-trained LLMs LFM2-8B-A1B