Deep Learning over the Internet: Training Language Models Collaboratively
This Hugging Face blog post describes a framework for training large language models collaboratively across volunteer compute contributed over the internet. The approach addresses the challenge of enabling distributed participants with heterogeneous hardware to jointly train models without centralized infrastructure. It represents an early exploration of decentralized training as an alternative to large-scale private compute clusters.
Related guides (3)
Related events (8)
Federated Learning using Hugging Face and Flower
This Hugging Face blog post describes how to combine the Hugging Face ecosystem with the Flower federated learning framework to train models across distributed, privacy-preserving data silos. It provides a practical walkthrough of integrating Transformers and Datasets libraries with Flower's federated training loop. The post targets practitioners looking to apply federated learning to NLP and other ML tasks without centralizing sensitive data.
Deploy LLMs with Hugging Face Inference Endpoints
Hugging Face published a guide on deploying large language models using their Inference Endpoints service. The post covers how to set up scalable, production-ready LLM deployments with minimal infrastructure overhead. It targets developers looking to move from experimentation to hosted inference without managing raw compute.
Very Large Language Models and How to Evaluate Them
This Hugging Face blog post from October 2022 discusses approaches to zero-shot evaluation of large language models hosted on the Hub. It covers methodologies for benchmarking LLMs without task-specific fine-tuning, addressing the practical challenges of evaluating very large models at scale. The post situates evaluation tooling within the broader ecosystem of open model hosting and assessment.
The Technology Behind BLOOM Training
This Hugging Face blog post details the infrastructure and training methodology used to train BLOOM, a 176-billion parameter open-access multilingual language model. It covers the use of Megatron-DeepSpeed for distributed training across hundreds of GPUs, including tensor parallelism, pipeline parallelism, and data parallelism strategies. The post also discusses hardware setup, memory optimization techniques, and lessons learned during the large-scale training run.
Decoupled DiLoCo: A new frontier for resilient, distributed AI training
DeepMind has published a blog post introducing Decoupled DiLoCo, a new approach to distributed AI training designed for resilience across heterogeneous or unreliable compute environments. The method appears to extend the original DiLoCo (Distributed Low-Communication) training framework, which enables training across loosely connected compute nodes with infrequent synchronization. The announcement signals continued investment in infrastructure techniques that reduce communication overhead and improve fault tolerance in large-scale model training.
Towards Encrypted Large Language Models with FHE
This Hugging Face blog post explores applying Fully Homomorphic Encryption (FHE) to Large Language Models, enabling inference on encrypted data without exposing plaintext inputs to the server. The approach aims to address privacy concerns in cloud-based LLM deployments by allowing computations to occur directly on ciphertext. The post likely covers the technical challenges of adapting transformer architectures to FHE constraints and presents early feasibility results.
Accelerate Large Model Training using DeepSpeed
This Hugging Face blog post explains how to use the Accelerate library in conjunction with DeepSpeed to train large language models more efficiently. It covers integration patterns, configuration options, and practical guidance for leveraging DeepSpeed's ZeRO optimization stages through the Accelerate abstraction layer. The post targets practitioners looking to scale model training without deep infrastructure expertise.
Introducing BLOOM: The World's Largest Open Multilingual Language Model
Hugging Face and the BigScience workshop released BLOOM, a 176-billion parameter open-access multilingual language model trained on 46 natural languages and 13 programming languages. The model was developed collaboratively by over 1,000 researchers and represents a significant milestone in open-weights large language model development. BLOOM was designed to be freely accessible to researchers and practitioners, in contrast to proprietary models of similar scale.