German grammatical gender

GRUFF: Benchmark for LLM Pronoun Fidelity, Reasoning, and Bias in German

The paper introduces GRUFF, a large-scale German dataset designed to evaluate LLM pronoun fidelity—the ability to correctly reuse a previously specified pronoun for a discourse entity despite intervening distractors. The study covers four grammatical gender systems and four pronoun sets including neopronouns (xier, en), finding that LLMs handle masculine and feminine agreement well but struggle with neopronouns and distractor robustness. Encoder-only models show greater robustness in German than English, attributed to grammatical gender cues. Occupational stereotype correlations across grammatical cases are weak and model-dependent.