4Simon Willison's Weblog·
Simon Willison introduces smevals, a small eval suite for models, prompts, and harnesses
Simon Willison published a post introducing smevals, a lightweight evaluation suite designed for testing AI models, prompts, and agent harnesses. The tool appears aimed at practitioners who want a quick, practical eval framework rather than large-scale benchmark infrastructure. As a tier-2 commentary/tooling post from a respected practitioner voice, it is useful for those tracking the evaluation and agent-tooling ecosystem.
Related guides (3)
Related events (8)
Simon Willison on Kimi K3 and the pelican benchmark as a model evaluation tool
Simon Willison writes about Kimi K3, a new model from Moonshot AI, using his informal 'pelican benchmark' as a lens for evaluation. The post reflects on what idiosyncratic, qualitative benchmarks can still reveal about model behavior that formal evals miss. As a tier-2 commentary piece, it offers practitioner-level perspective on a new open-weights or API-accessible model.
Every Eval Ever: unified schema and community repository for AI evaluation results
Researchers introduce Every Eval Ever, a shared schema and crowdsourced repository designed to standardize AI evaluation results across incompatible formats, frameworks, and sources. The system ingests results from evaluation harnesses, papers, leaderboards, and custom repositories into a single JSON document format, with optional per-instance output storage. The repository, hosted on Hugging Face, currently covers 22,235 models, 2,273 unique benchmarks, and 31 evaluation formats. The work addresses a persistent infrastructure problem in AI evaluation science: divergent scores for nominally identical evaluations and scattered, incomparable metadata.
AllenAI releases olmo-eval evaluation workbench for model development
AllenAI published a blog post on Hugging Face introducing olmo-eval, an evaluation workbench designed to integrate into the model development loop. The tool appears aimed at streamlining evaluation workflows for researchers iterating on open-weights models. This is relevant to the OLMo model family ecosystem and the broader open-weights evaluation infrastructure space.
EvalCards: A unified reporting layer for AI evaluation results with interpretive signals
Researchers introduce EvalCards, an operational schema and tooling layer that composes benchmark metadata, evaluation run data, and model metadata into a unified, interpretable record for AI evaluation reporting. The system derives a reporting schema from 52 papers and 10 stakeholder interviews, implements four interpretive signals (reproducibility, documentation completeness, provenance/risk, score comparability), and deploys a monitoring tool across 5,816 models, 635 benchmarks, and 101,843 results. The work targets the widespread inconsistency in how evaluation results are reported across leaderboards, model cards, and company blogs, making cross-source comparison unreliable. It addresses a structural gap in the evaluation ecosystem by providing extraction infrastructure, not just a proposal.
Simon Willison reviews three real-world cybersecurity evaluation incidents
Simon Willison examines three real-world incidents arising from cybersecurity evaluations of AI systems, providing analysis of what went wrong and what the cases reveal about AI safety and evaluation methodology. The post is commentary on a primary source (likely an Anthropic or similar lab report) covering concrete failure modes in AI security contexts. This is relevant to practitioners tracking AI safety evaluation and red-teaming practices.
SmolVLM - Small Yet Mighty Vision Language Model
Hugging Face introduces SmolVLM, a compact vision-language model designed to deliver strong multimodal performance at small parameter counts. The model targets edge and resource-constrained deployment scenarios while maintaining competitive capabilities relative to its size. The announcement highlights efficiency improvements in both training and inference for small-scale VLMs.
OpenEnv in Practice: Evaluating Tool-Using Agents in Real-World Environments
This Hugging Face blog post introduces OpenEnv, a framework for evaluating tool-using AI agents in real-world environments. The piece appears to address the challenge of benchmarking agentic systems that interact with external tools and environments, moving beyond static benchmarks toward dynamic, practical evaluation settings. As a tier-2 commentary piece, it likely discusses methodology, design choices, and results from applying OpenEnv to assess agent capabilities.
MLEvolve: Self-evolving multi-agent framework for automated ML algorithm discovery
MLEvolve is a new LLM-based multi-agent framework for end-to-end machine learning algorithm discovery, addressing limitations of existing MLE agents including information isolation and memoryless search. The system introduces Progressive MCGS (a graph-extended tree search), Retrospective Memory for experience accumulation, and decoupled strategic planning from code generation. Evaluated on MLE-Bench, it achieves state-of-the-art medal and valid submission rates within a 12-hour budget, and also outperforms AlphaEvolve on mathematical algorithm optimization tasks.