Entity · paper

OptimismBench: Forecasting Bias and the Alignment Effect in Language Model Judgment

paperactiveprovisionaloptimismbench-forecasting-bias-and-the-alignment-effect-in-language-model-judgment-9b89c4d0·1 events·first seen 19h ago

Aliases: OptimismBench: Forecasting Bias and the Alignment Effect in Language Model Judgment

Co-occurring entities

OptimismBench Anthropic

More like this (12)

OptimismBench optimism bias Reasoning Language Models The Illusion of Robustness: Aggregate Accuracy Hides Prediction Flips under Task-Irrelevant Context Pessimism's Paradox: Conservative Offline Training Amplifies Reward Hacking During Online Adaptation in Reasoning Models Decomposing Factual Sycophancy in Language Models: How Size and Instruction Tuning Shape Robustness Reproducing human biases in route choice using large language models: Toward scalable behavioral modeling The Value Axis: Language Models Encode Whether They're on the Right Track Does Reasoning Preserve Alignment? On the Trustworthiness of Large Reasoning Models Future Confidence Distillation in Large Language Models Apparent Psychological Profiles of Large Language Models are Largely a Measurement Artifact Large-Language-Models-as-a-Judge in Theory-Agnostic Adaptive Metric-Alignment for Prototypical Networks in Personality Recognition

Recent events (1)

6arXiv · cs.CL·19h ago·source ↗

OptimismBench reveals systematic optimism bias in LLM probability judgments, linked to post-training alignment

Researchers introduce OptimismBench, a benchmark using inverted scenario pairs to detect directional forecasting bias in LLMs without requiring ground-truth probabilities. Across 16 models from 8 providers, 14 exhibit optimistic bias; pessimism appears only in Anthropic's frontier tier. Post-training (alignment/RLHF) is identified as the mechanism setting bias direction, with opposite shifts across model families, and the pattern is robust to prompt, temperature, and self-debiasing interventions. The finding has direct implications for downstream decision pipelines that rely on LLM probability estimates.

Evaluation and Benchmarking AI Safety Research OptimismBench OptimismBench: Forecasting Bias and the Alignment Effect in Language Model Judgment Anthropic +1 more