5arXiv cs.CL (Computation and Language)·
Benchmark study reveals how linguistic framing of user beliefs shifts LLM context-following behavior
A new arXiv paper introduces a typology of 17 linguistically motivated expression-of-belief (EoB) types—spanning form, evidentiality, epistemic stance, and tone—to evaluate how phrasing affects whether LLMs defer to user-stated beliefs or their own prior knowledge. The authors benchmark 16 LLMs across Llama 3, Qwen3, and Gemma3 families at scales from 1B to 30B parameters, finding that larger and instruction-tuned models are systematically less context-following than smaller or base models. Specific linguistic framings (e.g., presuppositions, certainty markers) are identified as statistically more persuasive, with implications for prompt robustness and sycophancy research.
Related guides (4)
Related events (8)
BeliefTrack: Benchmarking and Improving Contextual Belief Management in LLMs
This paper introduces Contextual Belief Management (CBM) as a framework for studying how LLMs should update, preserve, or ignore information across long-horizon interactions. The authors release BeliefTrack, a closed-world benchmark with symbolic verifiers enabling exact turn-level evaluation across Rule Discovery and Circuit Diagnosis tasks. Vanilla LLMs show severe CBM failures; reinforcement learning with belief-state rewards reduces failure rates by 70.9% on average, while representation-level steering achieves 46.1% reduction. Probing experiments reveal latent belief-state dynamics underlying these failures.
Controlled study finds LLM user ratings reflect pre-interaction framing more than actual model performance
A controlled experiment with 162 participants tested six LLMs across three tasks, manipulating whether users were told the model was better or worse than it actually was. Post-interaction ratings were predicted by whether the model met expectations (β≈0.47–0.50) and user confidence, not by task performance (β≈-0.01–0.11, n.s.). Actual output quality depended only on true model capability, not framing. The authors conclude that user-elicited evaluations—including preference data driving public leaderboards—measure expectation management at least as much as model quality.
Cross-lingual evaluation framework reveals LLMs redistribute cultural narrative structure while preserving semantic meaning
A new arXiv preprint introduces a multilingual evaluation framework using 414 proverbs across 15 languages to assess whether LLMs preserve culturally grounded meaning when generating narratives. Using four LLMs to produce 13k narratives, the study finds that cross-lingual prompting preserves proverb-level semantic meaning but systematically redistributes agency, social positioning, and narrative structure. Strong inter-model convergence across architectures suggests multilingual LLMs rely on shared semantic abstractions. The authors argue that semantic similarity metrics alone overestimate cultural preservation in multilingual evaluations.
The Shibboleth Effect: Cross-lingual behavioral skew in frontier LLMs under adversarial geopolitical simulation
Researchers introduce the 'Shibboleth Effect' — systematic behavioral differences in LLMs when operating in different languages — and audit six frontier models (GPT-4o, Llama-4, Mistral-Large, Gemini-3.1-Pro, Qwen3.6-Plus, DeepSeek-R1) using a synthetic maritime territorial dispute wargame played in English versus Turkish. Results are heterogeneous: Llama-4 becomes significantly more coercive in Turkish while Gemini-3.1-Pro and DeepSeek-R1 become less so, and GPT-4o shows no detectable shift. The study identifies two candidate buffering mechanisms — chain-of-thought institutional anchoring and multilingual RLHF alignment — with direct implications for deploying LLMs in diplomatic or crisis-management contexts.
ESFP benchmark measures LLM epistemic stance flexibility across attribution conditions
Researchers introduce ESFP (Epistemic Stance Flexibility Probing), a 104-item behavioral benchmark evaluating whether LLMs appropriately shift between externally attributed and self-attributed epistemic registers when prompted differently. The benchmark spans six epistemic categories and five phrasing templates, assessing models on lexical self-attribution, representation-level responsiveness, stance content density, and cross-condition consistency. Evaluating eight frontier models from five vendors, the authors find epistemic flexibility is largely orthogonal to general capability, with a 27B open-weight model matching top proprietary systems and reasoning-optimized models showing no consistent advantage. Notably, surface-level lexical markers like 'I think' can shift without corresponding changes in actual expressed stance.
Tracing the Emergence of Human-Like Pragmatic Reasoning in LLMs Across Languages
Researchers conducted a population-matching experiment evaluating 25 LLMs on conditional inference tasks across four languages, comparing model behavior to matched human populations. The study finds that LLMs function as accurate semantic operators but systematically fail to capture pragmatic enrichments—context-sensitive inferences beyond literal logical meaning—that humans apply effortlessly. Model performance on pragmatic reasoning is not predicted by open vs. closed weights, training orientation, or architecture type, suggesting pragmatic reasoning remains an emergent and unreliable capability. The findings contribute to ongoing debates about whether LLMs reason like humans or merely approximate surface-level linguistic patterns.
MIT and CMU introduce Puppet benchmark to measure LLM belief manipulation in users
Researchers at MIT and Carnegie Mellon University developed Puppet, a benchmark that measures how much LLMs actually shift users' beliefs after conversation, as opposed to detecting manipulative language patterns. The study tracked over 1,000 users interacting with GPT-4o under various prompting conditions and found high variability in belief shifts, with a median change of 3.3 but standard deviation of ~22. Existing manipulation detectors showed near-zero correlation with actual belief change, while LLMs like GPT-4o achieved moderate correlation (0.436) when estimating belief shifts from conversation transcripts alone. The work argues for direct belief-shift measurement as a more valid approach to assessing LLM persuasive risk.
FRANZ: A Communicative Audit Framework for LLM Response Framing on Subjective Questions
Researchers introduce FRANZ, an automated framework for auditing how LLMs frame responses to subjective, culturally-sensitive questions across four dimensions: cultural positioning, generalizing language, anthropomorphic cues, and conversational maxims. The work is paired with SQUARE, a 376k-question corpus drawn from 57 subreddits and mapped to 7 countries and 19 question categories. Applying FRANZ to three open-weight LLMs reveals statistically significant differences in framing behavior, and uncovers a positive coupling between insider positioning and anthropomorphism that varies by country. The study argues that existing evaluations focused on factual correctness miss important communicative dimensions of LLM outputs.