5arXiv cs.CL (Computation and Language)·
Rate-utility frontiers reveal no dominant encoding across tokens, bytes, and pixels for language models
A new arXiv preprint introduces a controlled framework for comparing text encodings (subword tokens, raw bytes, rendered pixels) by sweeping a shared bottleneck width to trace rate-utility frontiers across 13 languages and 5 scripts. The study separates three often-conflated quantities: input positions, latent capacity, and task-relevant information surviving compression. Evaluated on surface form preservation, cross-lingual alignment, and topic classification, no encoding dominates across all tasks or capacity regimes — pixels excel at surface form, bytes at cross-lingual alignment, and tokens at topic prediction. The findings reframe encoding choice as a task- and capacity-dependent tradeoff rather than a fixed preference.
Related guides (2)
Related events (8)
Multilingual study finds LLMs are not uniformly robust to non-canonical tokenizations, with up to 23.7% performance drops
A new arXiv paper investigates how language models behave when given alternative (non-canonical) tokenizations of the same input string across 27 languages and six downstream tasks. While prior work showed English models are largely invariant to such perturbations, the study finds this does not generalize: Llama-3.1-8B drops 23.7% on average, Qwen3-8B 11.4%, and Gemma-3-12B 9.9% in relative performance. Languages with higher token fragmentation are systematically more sensitive, and the authors show LoRA fine-tuning on multi-tokenization data—including English-only data—provides meaningful mitigation.
Energy-based transformers as unified predictors of reading difficulty in computational psycholinguistics
A new arXiv preprint introduces energy-based transformer measures as predictors of human reading difficulty, evaluated across three reading-time corpora (Natural Stories, UCL eye-tracking, UCL self-paced reading). The energy measure outperforms surprisal alone and appears to subsume both surprisal and attention entropy effects, suggesting it could serve as a single unified predictor. The work connects transformer language models to Hopfield networks and dense associative memory literature, marking the first application of energy-based transformer measures in computational psycholinguistics.
Scaling laws for native multimodal pre-training from scratch characterized empirically
A new arXiv preprint investigates compute-optimal scaling for transformer-based vision-language models trained natively on multimodal inputs from scratch, rather than via late-fusion of separately pre-trained components. The authors derive power-law relationships between compute budget, model size, and token count, finding that language and multimodal objectives exhibit distinct scaling behaviors and that data mixture composition strongly influences optimal resource allocation. They also report positive cross-modal transfer effects, including improved pure-text spatial reasoning. The work establishes an efficiency frontier for configuring model size, token count, and data mixture under fixed compute.
Diagnostic framework decomposes LLM difficulty on historical Italian and Russian texts
A new arXiv preprint proposes a four-dimensional framework for measuring LLM difficulty on historical language: tokenization cost, surprisal, semantic robustness, and context sensitivity. Evaluated on 17th-century Italian, 19th-century Italian, and 18th-century Russian texts, the study finds that tokenization penalties (25-30% inflation) are similar across languages but predictive difficulty diverges sharply—early modern Italian is 2.4x more surprising than modern Italian while Russian shows only modest increase. Crucially, embedding similarity remains high (>0.85) even when generation is unstable, and a simple temporal context prompt reduces historical surprisal by ~60%. The findings have practical implications for deploying LLMs in digital library and historical document workflows.
Adaptive asymmetric token compression accelerates time series language models up to 7.68×
A new arXiv preprint proposes an adaptive token budgeting framework for time series (TS) language models that compresses TS tokens using frequency-domain structure and progressively prunes prompt tokens across model layers. The authors demonstrate up to 7.68× inference acceleration with performance improvements in 78% of evaluated settings across forecasting, classification, imputation, and anomaly detection tasks. The work is motivated by the observation that TS tokens have uneven spectral contributions and prompt-token influence attenuates with model depth, making uniform token processing wasteful.
Calibrated LLM annotation and encoder transfer for measuring human values in social media text
A new arXiv preprint investigates how different LLMs, prompts, and instruction languages operationalize Schwartz's theory of basic human values when annotating non-English social media posts. The authors evaluate annotation quality beyond standard F1 metrics, examining structural alignment, error structure, and confidence-ambiguity relations, finding that iterative prompt calibration reduces misattributions. They also demonstrate that LLM annotations can be transferred to a smaller encoder model via soft-label training, preserving theory-grounded value interpretations and uncertainty information.
Linear representations of grammaticality found in neural language model internals
A new arXiv preprint investigates whether grammaticality is encoded in the internal representations of neural language models, moving beyond probability-based evaluations. Using mass-mean probing, the authors find that grammatical and ungrammatical sentences are robustly separated in representational space across a wide range of pretrained models, grammatical phenomena, and languages. The encoding is not fully explained by confounds like lexical frequency or plausibility, suggesting grammaticality constitutes a coherent representational dimension. The findings contribute to debates about syntactic knowledge in LLMs and offer a probability-independent framework for evaluating grammatical competence.
Translation-based fine-tuning of English BERT compared to native-language models across six NLP tasks and five languages
A new arXiv preprint systematically compares translation-based fine-tuning of English BERT against native-language BERT models across six NLP tasks using datasets from Bulgarian, Chinese, Dutch, Italian, and Russian. The translation approach was comparable or superior in 53.3% of cases, with strongest gains in Question Answering, POS Tagging, and NLI, but weaker performance on Named Entity Recognition and Hate Speech Detection. Results suggest translation-based fine-tuning is most effective for syntactic tasks and typologically English-close languages like Dutch, offering a resource-efficient path for low-resource NLP.