4Hugging Face Blog·
Hugging Face integrates Every Eval Ever community evaluation results into model pages
Hugging Face is featuring results from the Every Eval Ever (EEE) community evaluation initiative directly on model pages, surfacing community-driven benchmark coverage alongside official evaluations. This integration makes a broader set of evaluation signals visible to practitioners browsing the Hub. The move reflects growing interest in community-sourced evals as a complement to lab-run benchmarks.
Related guides (3)
Related events (8)
Announcing Evaluation on the Hub
Hugging Face announced Evaluation on the Hub, a new feature enabling users to evaluate any model on any dataset directly within the Hugging Face Hub infrastructure. The tool aims to lower the barrier to standardized model evaluation by integrating evaluation workflows into the existing model and dataset hosting platform. This represents an infrastructure step toward more accessible and reproducible benchmarking in the ML community.
Community Evals: Because we're done trusting black-box leaderboards over the community
Hugging Face introduces Community Evals, a framework aimed at replacing or supplementing opaque black-box leaderboards with community-driven model evaluations. The initiative reflects growing skepticism about the reliability and transparency of existing benchmark leaderboards. By crowdsourcing evaluations, Hugging Face seeks to make model assessment more transparent, diverse, and resistant to gaming. This represents a structural shift in how the open-source AI community approaches model comparison and trust.
Every Eval Ever: unified schema and community repository for AI evaluation results
Researchers introduce Every Eval Ever, a shared schema and crowdsourced repository designed to standardize AI evaluation results across incompatible formats, frameworks, and sources. The system ingests results from evaluation harnesses, papers, leaderboards, and custom repositories into a single JSON document format, with optional per-instance output storage. The repository, hosted on Hugging Face, currently covers 22,235 models, 2,273 unique benchmarks, and 31 evaluation formats. The work addresses a persistent infrastructure problem in AI evaluation science: divergent scores for nominally identical evaluations and scattered, incomparable metadata.
Improving Hugging Face Model Access for Kaggle Users
Hugging Face has announced an integration improvement that streamlines how Kaggle users access models from the Hugging Face Hub. The update appears to reduce friction for practitioners using Kaggle notebooks and compute environments to work with Hugging Face-hosted models. This represents a platform-level partnership move between two major ML community hubs.
Hugging Face Adds New Analytics Dashboard to Inference Endpoints
Hugging Face has released updated analytics features for its Inference Endpoints product, providing users with improved visibility into deployment metrics and usage patterns. The announcement covers new dashboards and monitoring capabilities for hosted model inference. This is a product update targeting enterprise and developer users running models on Hugging Face's managed inference infrastructure.
BigCodeBench: The Next Generation of HumanEval
Hugging Face introduces BigCodeBench, a new code generation benchmark designed to succeed HumanEval by offering more challenging and diverse programming tasks. The benchmark aims to better evaluate LLMs on real-world coding scenarios involving complex function calls and library usage. A leaderboard accompanies the release to track model performance across the community.
Hugging Face introduces Real World VoiceEQ benchmark for human-quality voice AI evaluation
Hugging Face has released Real World VoiceEQ, a new benchmark designed to measure the human-perceived quality of voice AI systems. The benchmark targets a gap in existing evaluation frameworks by focusing on naturalistic, real-world voice interaction quality rather than purely technical metrics. This is relevant to the growing voice AI ecosystem where subjective quality dimensions are difficult to quantify.
Hugging Face Machine Learning Demos on arXiv
Hugging Face announced an integration allowing ML demos to be linked or embedded directly on arXiv paper pages. This lowers the barrier between research publication and interactive model demonstration. The feature connects academic papers to live Spaces or model demos hosted on Hugging Face.