5arXiv cs.CL (Computation and Language)·
MultiHashFormer: Hash-based token representations enable parameter-efficient autoregressive language models
Researchers introduce MultiHashFormer, a framework that replaces standard embedding matrices with hash-based token representations for causal language models. Each token is encoded as a unique signature of discrete hash IDs from multiple independent hash functions, compressed into a latent vector for a Transformer decoder. Evaluated at 100M, 1B, and 3B parameter scales, MultiHashFormer outperforms standard Transformer LMs on multiple benchmarks while enabling multilingual vocabulary expansion at constant parameter cost.
Related guides (1)
Related events (8)
LangMAP: Language-adaptive tokenization from a shared vocabulary without language identification at inference
LangMAP (Language-adaptive Maximum a Posteriori Tokenization) extends the UnigramLM algorithm to produce language-specific tokenizations from a single shared vocabulary, eliminating the need to retrain models or swap vocabularies for multilingual settings. A key property is that language labels are only required at training time; inference proceeds without language identification. Evaluated across 14 tokenizers, 9 natural languages, and 9 programming languages, LangMAP improves morphological boundary alignment and AST-leaf alignment for all coding languages tested. Fine-tuning results are mixed: consistent gains on grammatical acceptability (MultiBLiMP) but less consistent on knowledge tasks (Global-PIQA, Belebele).
The Reformer - Pushing the limits of language modeling
This Hugging Face blog post covers the Reformer, a memory-efficient transformer architecture that uses locality-sensitive hashing (LSH) attention and reversible residual layers to handle very long sequences. The post explains the technical mechanisms that allow Reformer to process sequences up to 1 million tokens with significantly reduced memory footprint compared to standard transformers. It serves as an educational deep-dive into the architectural innovations introduced in the original Reformer paper by Kitaev et al.
Multilingual study finds LLMs are not uniformly robust to non-canonical tokenizations, with up to 23.7% performance drops
A new arXiv paper investigates how language models behave when given alternative (non-canonical) tokenizations of the same input string across 27 languages and six downstream tasks. While prior work showed English models are largely invariant to such perturbations, the study finds this does not generalize: Llama-3.1-8B drops 23.7% on average, Qwen3-8B 11.4%, and Gemma-3-12B 9.9% in relative performance. Languages with higher token fragmentation are systematically more sensitive, and the authors show LoRA fine-tuning on multi-tokenization data—including English-only data—provides meaningful mitigation.
Adaptive asymmetric token compression accelerates time series language models up to 7.68×
A new arXiv preprint proposes an adaptive token budgeting framework for time series (TS) language models that compresses TS tokens using frequency-domain structure and progressively prunes prompt tokens across model layers. The authors demonstrate up to 7.68× inference acceleration with performance improvements in 78% of evaluated settings across forecasting, classification, imputation, and anomaly detection tasks. The work is motivated by the observation that TS tokens have uneven spectral contributions and prompt-token influence attenuates with model depth, making uniform token processing wasteful.
Scaling laws for native multimodal pre-training from scratch characterized empirically
A new arXiv preprint investigates compute-optimal scaling for transformer-based vision-language models trained natively on multimodal inputs from scratch, rather than via late-fusion of separately pre-trained components. The authors derive power-law relationships between compute budget, model size, and token count, finding that language and multimodal objectives exhibit distinct scaling behaviors and that data mixture composition strongly influences optimal resource allocation. They also report positive cross-modal transfer effects, including improved pure-text spatial reasoning. The work establishes an efficiency frontier for configuring model size, token count, and data mixture under fixed compute.
LIHA reveals first-token broadcaster heads as mechanistic source of language identity in transformers
Researchers introduce Language Identity Head Ablation (LIHA), a causal intervention that zeros individual attention heads to measure language-switching behavior across 2,700 prompt-language pairs in seven languages. Applied to GPT-2, LIHA identifies a small set of 'first-token broadcaster' heads that propagate language identity signals throughout generation, with compensatory redistribution following a hierarchical, feedforward pattern. A controlled comparison between Qwen2.5-1.5B-Base and Qwen2.5-1.5B-Instruct provides direct causal evidence that instruction tuning reorganizes language identity circuits toward early-layer localization. The findings offer mechanistic grounding for why multilingual models generate in the wrong language and why this is difficult to correct.
Tapered Language Models: front-loading parameter capacity improves perplexity at no extra cost
Researchers introduce Tapered Language Models (TLMs), an architectural principle that allocates more parameter capacity to earlier layers and less to later layers via a cosine-scheduled MLP width taper, under a fixed total budget. Controlled experiments across three model scales and four architectures (Transformer, Gated Attention, Hope-attention, Titans) show consistent perplexity and downstream benchmark improvements over uniform-width baselines. The finding reframes depth-uniform parameter allocation — a default inherited from the original transformer — as a suboptimal choice, offering a free architectural lever applicable across modern LM families.
AllenAI analysis: which tokens do hybrid models predict better than pure transformers?
A Hugging Face blog post from AllenAI investigates the token-level prediction differences between hybrid models (combining attention and state-space or other mechanisms) and standard transformer architectures. The analysis aims to characterize where hybrid architectures gain or lose predictive advantage at the token level. This kind of mechanistic comparison is relevant to ongoing debates about when hybrid designs are worth their added complexity.