4arXiv cs.CL (Computation and Language)·
Preprint argues LLM-based surprisal metrics obscure representational commitments in psycholinguistics
A new arXiv preprint challenges the common practice in computational psycholinguistics of treating LLM-derived surprisal values as theory-neutral, representation-agnostic measures. The authors argue that algorithm choice and model architecture significantly affect computed language model probabilities, meaning LLM surprisal is not interchangeable across models. Three analyses demonstrate that using black-box LLMs uncritically conflates computational-level and algorithmic-level commitments, undermining the theoretical claims of Surprisal Theory.
Related guides (1)
Related events (8)
Surprisal theory in psycholinguistics is tautological without rational grounding, paper argues
A new arXiv preprint argues that surprisal theory — the claim that human linguistic processing difficulty is an affine function of surprisal under some language model — is unfalsifiable without additional constraints, because for any difficulty measure a language model can be constructed to match it. The paper identifies that two decades of psycholinguistic work implicitly assumed the relevant language model was the corpus-generating distribution, an assumption recent empirical work has undermined by showing better corpus models can be worse predictors of processing difficulty. The author concludes that breaking the tautology requires deriving the relevant language model from a rationalist, non-empirically motivated model of the comprehender (e.g., based on memory constraints or processing goals).
Survey chapter on LLM mechanisms, emergent capabilities, and cognition debates
A new arXiv preprint surveys current understanding of large language models, covering the Transformer architecture, emergent capabilities resembling human cognition (symbolic reasoning, theory of mind, deception), and explainability approaches from neuron activation analysis to circuit tracing. The chapter also engages the debate over whether LLMs genuinely understand or merely pattern-match, arguing against reductive anti-anthropomorphism while acknowledging human-LLM differences. It is framed as a book chapter synthesizing recent empirical findings and theoretical positions.
Quantifying Cross-Linguistic Effects of Syncretism on Agreement Attraction Using LLM Processing Proxies
This paper investigates why morphological syncretism amplifies agreement attraction errors in some languages (English, German, Russian) but not others (Turkish, Armenian), a pattern lacking a principled account. The authors use surprisal and attention entropy derived from large language models as proxies for human sentence processing across four languages. LLM-derived measures successfully replicate behavioral findings in English and German, align with Turkish null results, and partially capture Russian patterns. The work demonstrates LLMs as tools for cross-linguistic psycholinguistic investigation.
LLM-based classification exposes keyword lexicon artifacts in computational social science stance measurement
A new arXiv preprint demonstrates that statistically significant findings in computational social science can be entirely measurement artifacts of keyword-based scoring instruments. Analyzing 85 interviews across four public intellectuals, the authors show that keyword-based certainty scores produce strong correlations (r=0.72–0.93) that collapse or invert when replaced with LLM zero-shot semantic classification on 32,625 sentences. The paper identifies three structural failure modes in keyword lexicons—syntactic blindness, polysemy blindness, and categorical absence—and argues that keyword counts measure lexical co-occurrence tendencies rather than rhetorical stance. The work has implications for the validity of prior NLP-based social science research and for the comparative utility of LLMs as measurement instruments.
Token probability measurements reveal production-perception asymmetry in LLMs
A new arXiv preprint investigates whether LLMs exhibit a functional analog to the psycholinguistic production-perception distinction, using direct token probability measurements rather than metalinguistic prompting. Using Llama-3.1-8B and four other open-weight models, the authors find that production-perception prompt distances consistently exceed production-production distances by a ratio of ~1.8, with near-ceiling correlations in the production-production control confirming the effect is specific to communicative framing. The effect replicates across five models spanning base and instruction-tuned variants, and temporal analysis shows perception prompts exert strongest influence at sequence beginnings. The findings suggest prompt framing alone induces a production-perception distinction in decoder-only architectures.
Diagnostic framework decomposes LLM difficulty on historical Italian and Russian texts
A new arXiv preprint proposes a four-dimensional framework for measuring LLM difficulty on historical language: tokenization cost, surprisal, semantic robustness, and context sensitivity. Evaluated on 17th-century Italian, 19th-century Italian, and 18th-century Russian texts, the study finds that tokenization penalties (25-30% inflation) are similar across languages but predictive difficulty diverges sharply—early modern Italian is 2.4x more surprising than modern Italian while Russian shows only modest increase. Crucially, embedding similarity remains high (>0.85) even when generation is unstable, and a simple temporal context prompt reduces historical surprisal by ~60%. The findings have practical implications for deploying LLMs in digital library and historical document workflows.
LLMs fail to reliably self-report adversarial prefill attacks, study finds
A new arXiv paper evaluates whether LLMs can recognize that their own prior responses were elicited by adversarial prefill attacks, testing ten open-weight models (3B–70B) across four safety benchmarks. Models claim intent on prefilled responses only 27.3% of the time on average, and introspective signal is largely mediated by refusal-related reasoning. Three LoRA fine-tuning methods (SFT, GRPO, DPO) improve the intention-probe gap but counterintuitively raise attack success rates on most models, suggesting partial and fragile mitigation. The findings raise concerns about the reliability of LLM self-reports in safety-critical contexts.
LLMs violate statistical self-consistency: the macro fallacy in population-level estimation
A new arXiv paper introduces a framework for testing whether LLM outputs satisfy the law of total probability — specifically, whether prior-weighted conditional estimates over subpopulations aggregate correctly into population-level marginals. Using binary tree partitions and persona prompting across frontier models, the authors find widespread violations of this self-consistency principle. A key finding, termed the 'macro fallacy,' is that estimates reconstructed from fine-grained subpopulation prompts are often more accurate than direct population-level estimates, suggesting models hold relevant subpopulation knowledge but fail to propagate it upward. The work proposes statistical self-consistency as a reference-free, unsaturated evaluation criterion for LLMs.