paper

How Surprising Is Historical Italian to Language Models? Tokenization Tax, Comprehension Tax, and a Simple Mitigation

paperactiveprovisional

how-surprising-is-historical-italian-to-language-models-tokenization-tax-comprehension-tax-and-a-simple-mitigation-8c3b4ba1

·1 events·first seen 2d ago

Aliases: How Surprising Is Historical Italian to Language Models? Tokenization Tax, Comprehension Tax, and a Simple Mitigation

Co-occurring entities

I Promessi Sposi

More like this (12)

Transformer Language Models Beyond Uniform Tokens: Adaptive Compression for Time Series Language Models The Shibboleth Effect: Auditing the Cross-Lingual Distributional Skew of Large Language Models On the Limits of Prompt-Conditioned Language Models as General-Purpose Learners Language Modeling Loss Same Lesson, Different Story: Cross-Lingual Reconstruction of Cultural Narratives in Large Language Models Reasoning Language Models Tokenizers Interleaved Speech Language Models Latently Work In Text Actionable Activation Directions for Detecting and Mitigating Emergent Misalignment Across Language Model Families Tracing Target Answers in Poisoned Retrieval Corpora via Token Influence Attribution The Neutral Mask: How RLHF Provides Shallow Alignment while Leaving Partisan Structure Intact in a Large Language Model

Recent events (1)

3arXiv · cs.CL·2d ago·source ↗

Diagnostic framework decomposes LLM difficulty on historical Italian and Russian texts

A new arXiv preprint proposes a four-dimensional framework for measuring LLM difficulty on historical language: tokenization cost, surprisal, semantic robustness, and context sensitivity. Evaluated on 17th-century Italian, 19th-century Italian, and 18th-century Russian texts, the study finds that tokenization penalties (25-30% inflation) are similar across languages but predictive difficulty diverges sharply—early modern Italian is 2.4x more surprising than modern Italian while Russian shows only modest increase. Crucially, embedding similarity remains high (>0.85) even when generation is unstable, and a simple temporal context prompt reduces historical surprisal by ~60%. The findings have practical implications for deploying LLMs in digital library and historical document workflows.

Evaluation and Benchmarking How Surprising Is Historical Italian to Language Models? Tokenization Tax, Comprehension Tax, and a Simple Mitigation I Promessi Sposi