surprisal is Not a Theory

Preprint argues LLM-based surprisal metrics obscure representational commitments in psycholinguistics

A new arXiv preprint challenges the common practice in computational psycholinguistics of treating LLM-derived surprisal values as theory-neutral, representation-agnostic measures. The authors argue that algorithm choice and model architecture significantly affect computed language model probabilities, meaning LLM surprisal is not interchangeable across models. Three analyses demonstrate that using black-box LLMs uncritically conflates computational-level and algorithmic-level commitments, undermining the theoretical claims of Surprisal Theory.