5Simon Willison's Weblog·
Simon Willison asks whether AI labs are 'pelicanmaxxing' — optimizing for benchmarks over real capability
Simon Willison publishes a commentary piece questioning whether AI labs are optimizing their models for benchmark performance at the expense of genuine capability — a practice he terms 'pelicanmaxxing.' The piece engages with ongoing concerns about benchmark overfitting and the gap between reported scores and real-world utility. As a widely-read practitioner voice, Willison's framing may influence how the community interprets lab benchmark claims.
Related guides (3)
Related events (8)
Simon Willison argues better AI models may produce worse tooling ecosystems
Simon Willison publishes a commentary piece titled 'Better Models: Worse Tools,' suggesting a potential inverse relationship between frontier model capability improvements and the quality or utility of the surrounding tool ecosystem. The piece appears to examine how advances in model capability may reduce incentives or change the design space for tooling built around those models. As a widely-read practitioner voice, Willison's framing could influence how developers think about the agent and tooling landscape.
Simon Willison on Kimi K3 and the pelican benchmark as a model evaluation tool
Simon Willison writes about Kimi K3, a new model from Moonshot AI, using his informal 'pelican benchmark' as a lens for evaluation. The post reflects on what idiosyncratic, qualitative benchmarks can still reveal about model behavior that formal evals miss. As a tier-2 commentary piece, it offers practitioner-level perspective on a new open-weights or API-accessible model.
Simon Willison links: AI Mania Is Eviscerating Global Decision-Making
Simon Willison flags a piece arguing that AI hype is degrading the quality of global decision-making. The item is a link post with minimal body content, suggesting it is a brief pointer to an external critical analysis of AI's societal influence. The underlying claim — that AI mania distorts institutional and policy reasoning — is a substantive critique worth indexing.
AI Scaling Myths
A commentary piece from normaltech.ai argues that AI scaling will eventually hit limits, framing the debate as a question of timing rather than whether limits exist. The piece appears to challenge prevailing optimism around continued scaling returns. Given the minimal body text, the depth of argument is unclear, but the topic directly engages the scaling laws debate central to frontier AI development.
Simon Willison on fears and realities around Chinese AI models
Simon Willison publishes commentary titled 'Who's Afraid of Chinese Models?' examining concerns and attitudes toward Chinese AI models. The piece appears to engage with the geopolitical and technical dimensions of Chinese frontier model development. As a tier-2 commentary from a respected practitioner voice, it likely addresses whether fears about Chinese models are warranted or overstated.
New paper: AI agents that matter
A paper from the AI Snake Oil / Normal Tech group critiques current AI agent benchmarking and evaluation practices. The work argues that existing agent benchmarks are poorly designed for assessing real-world utility, and calls for rethinking how agent performance is measured. The commentary targets the gap between benchmark scores and practical deployment value.
Simon Willison publishes opinionated guide to AI model selection
Simon Willison published a practical guide recommending which AI models and tools to use for specific tasks. As a widely-read practitioner voice, his model selection opinions reflect real-world usage patterns across current frontier and open-weights offerings. The piece is useful for tracking which models are gaining mindshare among technically sophisticated users.
Simon Willison on the asymmetric time pressures facing AI enthusiasts vs. skeptics
Simon Willison publishes a commentary framing the AI debate as two groups facing different temporal pressures: enthusiasts racing against time to realize transformative potential before momentum stalls, and skeptics racing against entropy as AI systems proliferate and become harder to constrain. The piece is an opinion/strategy essay from a respected practitioner voice. It contributes to ongoing discourse about AI trajectories and the structural dynamics of the optimist-pessimist divide.