Entity · paper

Rating the Pitch, Not the Product: User Evaluations of LLMs Reflect Expectations More Than Performance

paperactiveprovisionalrating-the-pitch-not-the-product-user-evaluations-of-llms-reflect-expectations-more-than-performance-03a6a43c·1 events·first seen Jul 7, 2026

Aliases: Rating the Pitch, Not the Product: User Evaluations of LLMs Reflect Expectations More Than Performance

More like this (12)

Can LLMs Judge Better Than They Generate? Evaluating Task Asymmetry, Mechanistic Interpretability and Transferability for In-Context QA Who Grades the Grader? Co-Evolving Evaluation Metrics and Skills for Self-Improving LLM Agents LLM evaluation Reassessing High-Performing LLMs on Polish Medical Exams: True Competence or Bias-Driven Performance?It's Not What You Say, It's How You Say It: Evaluating LLM Responses to Expressions of Belief When the Judge Changes, So Does the Measurement: Auditing LLM-as-Judge Reliability LLM-judged explanation score LLM-judge scoring Inside the Unfair Judge: A Mechanistic Interpretability Account of LLM-as-Judge Bias Flaws in the LLM Automation Narrative Artificial Analysis LLM Performance Leaderboard From Plausible to Actionable: A Position on LLM Self-Explanations

Recent events (1)

7arXiv · cs.CL·Jul 7, 2026·source ↗

Controlled study finds LLM user ratings reflect pre-interaction framing more than actual model performance

A controlled experiment with 162 participants tested six LLMs across three tasks, manipulating whether users were told the model was better or worse than it actually was. Post-interaction ratings were predicted by whether the model met expectations (β≈0.47–0.50) and user confidence, not by task performance (β≈-0.01–0.11, n.s.). Actual output quality depended only on true model capability, not framing. The authors conclude that user-elicited evaluations—including preference data driving public leaderboards—measure expectation management at least as much as model quality.

Evaluation and Benchmarking AI Safety Research Rating the Pitch, Not the Product: User Evaluations of LLMs Reflect Expectations More Than Performance