Announcing Evaluation on the Hub
Hugging Face announced Evaluation on the Hub, a new feature enabling users to evaluate any model on any dataset directly within the Hugging Face Hub infrastructure. The tool aims to lower the barrier to standardized model evaluation by integrating evaluation workflows into the existing model and dataset hosting platform. This represents an infrastructure step toward more accessible and reproducible benchmarking in the ML community.
Related guides (3)
Related events (8)
Hugging Face Launches Inference Providers on the Hub
Hugging Face has introduced Inference Providers on the Hub, a feature that allows users to run models hosted on the Hub through third-party inference providers directly from the platform. This integration consolidates access to multiple inference backends under a unified interface, reducing friction for developers who want to deploy or test models at scale. The announcement positions Hugging Face as a marketplace layer connecting model authors with inference infrastructure providers.
Improving Hugging Face Model Access for Kaggle Users
Hugging Face has announced an integration improvement that streamlines how Kaggle users access models from the Hugging Face Hub. The update appears to reduce friction for practitioners using Kaggle notebooks and compute environments to work with Hugging Face-hosted models. This represents a platform-level partnership move between two major ML community hubs.
Community Evals: Because we're done trusting black-box leaderboards over the community
Hugging Face introduces Community Evals, a framework aimed at replacing or supplementing opaque black-box leaderboards with community-driven model evaluations. The initiative reflects growing skepticism about the reliability and transparency of existing benchmark leaderboards. By crowdsourcing evaluations, Hugging Face seeks to make model assessment more transparent, diverse, and resistant to gaming. This represents a structural shift in how the open-source AI community approaches model comparison and trust.
Announcing New Hugging Face and KerasHub Integration
Hugging Face and KerasHub have announced a new integration enabling users to access Hugging Face models and datasets directly through the Keras ecosystem. This partnership bridges two major ML frameworks, allowing Keras users to leverage the Hugging Face Hub's model repository without leaving the Keras workflow. The integration is aimed at reducing friction for practitioners who prefer Keras-based training and inference pipelines.
Every Eval Ever: unified schema and community repository for AI evaluation results
Researchers introduce Every Eval Ever, a shared schema and crowdsourced repository designed to standardize AI evaluation results across incompatible formats, frameworks, and sources. The system ingests results from evaluation harnesses, papers, leaderboards, and custom repositories into a single JSON document format, with optional per-instance output storage. The repository, hosted on Hugging Face, currently covers 22,235 models, 2,273 unique benchmarks, and 31 evaluation formats. The work addresses a persistent infrastructure problem in AI evaluation science: divergent scores for nominally identical evaluations and scattered, incomparable metadata.
Hugging Face and FriendliAI Partner to Supercharge Model Deployment on the Hub
Hugging Face and FriendliAI have announced a partnership to integrate FriendliAI's inference infrastructure directly into the Hugging Face Hub. The collaboration aims to simplify and accelerate model deployment for developers accessing models through the Hub. This expands the ecosystem of inference providers available on Hugging Face's platform.
huggingface_hub v1.0: Five Years of Building the Foundation of Open Machine Learning
Hugging Face has released huggingface_hub v1.0, marking a major milestone for the Python client library that underpins access to the Hugging Face Hub ecosystem. The v1.0 designation signals API stability and maturity after five years of development. This library is a foundational piece of open-source ML infrastructure, enabling model downloads, dataset access, and repository management across the broader ML community.
XetHub Joins Hugging Face
XetHub, a company specializing in large-scale file storage and versioning for ML datasets and models, is being acquired by Hugging Face. The acquisition is intended to strengthen Hugging Face's infrastructure for hosting and managing large model and dataset repositories. This move reflects ongoing consolidation in the AI tooling and infrastructure space around the Hugging Face platform.