Xet Storage Integration on Hugging Face Hub
Hugging Face has integrated Xet, a chunk-based deduplication storage backend, into the Hub to improve large model file storage and transfer efficiency. The integration aims to reduce redundant data storage and speed up uploads/downloads for large model weights by splitting files into content-addressed chunks. This is an infrastructure improvement relevant to the open-weights ecosystem where multi-gigabyte model files are common.
Related guides (3)
Related events (8)
Migrating the Hugging Face Hub from Git LFS to Xet
Hugging Face is migrating its model and dataset hosting infrastructure from Git LFS to Xet, a content-addressed storage system designed for large binary files. The migration aims to improve upload/download speeds, deduplication, and storage efficiency for the large model weights and datasets hosted on the Hub. This represents a significant infrastructure change affecting how millions of AI artifacts are stored and accessed by the community.
XetHub Joins Hugging Face
XetHub, a company specializing in large-scale file storage and versioning for ML datasets and models, is being acquired by Hugging Face. The acquisition is intended to strengthen Hugging Face's infrastructure for hosting and managing large model and dataset repositories. This move reflects ongoing consolidation in the AI tooling and infrastructure space around the Hugging Face platform.
Introducing Storage Buckets on the Hugging Face Hub
Hugging Face is launching Storage Buckets, a new feature on the Hub that provides object storage capabilities for AI/ML workflows. This expands the Hub's infrastructure offerings beyond model and dataset repositories, enabling users to store arbitrary files and artifacts. The feature targets teams managing large-scale AI pipelines who need integrated storage alongside their models and datasets.
Announcing New Hugging Face and KerasHub Integration
Hugging Face and KerasHub have announced a new integration enabling users to access Hugging Face models and datasets directly through the Keras ecosystem. This partnership bridges two major ML frameworks, allowing Keras users to leverage the Hugging Face Hub's model repository without leaving the Keras workflow. The integration is aimed at reducing friction for practitioners who prefer Keras-based training and inference pipelines.
Shipping a Trillion Parameters With a Hub Bucket: Delta Weight Sync in TRL
Hugging Face introduces Delta Weight Sync in TRL, a technique for efficiently synchronizing model weight updates during large-scale training by transmitting only the delta (difference) between checkpoints rather than full parameter snapshots. The approach targets trillion-parameter training regimes where checkpoint bandwidth is a significant bottleneck. The post describes integration with the Hugging Face Hub as a storage and distribution layer for these delta updates.
Improving Hugging Face Model Access for Kaggle Users
Hugging Face has announced an integration improvement that streamlines how Kaggle users access models from the Hugging Face Hub. The update appears to reduce friction for practitioners using Kaggle notebooks and compute environments to work with Hugging Face-hosted models. This represents a platform-level partnership move between two major ML community hubs.
Hugging Face Launches Inference Providers on the Hub
Hugging Face has introduced Inference Providers on the Hub, a feature that allows users to run models hosted on the Hub through third-party inference providers directly from the platform. This integration consolidates access to multiple inference backends under a unified interface, reducing friction for developers who want to deploy or test models at scale. The announcement positions Hugging Face as a marketplace layer connecting model authors with inference infrastructure providers.
Easily Train Models with H100 GPUs on NVIDIA DGX Cloud
Hugging Face announced integration with NVIDIA DGX Cloud, enabling users to train models on H100 GPU clusters directly through the Hugging Face platform. The partnership simplifies access to high-end training infrastructure without requiring users to manage cloud provisioning themselves. This represents a continued push to lower the barrier to large-scale model training for the broader ML community.