Visualize and Understand GPU Memory in PyTorch
A Hugging Face blog post explains how to visualize and analyze GPU memory usage during PyTorch model training. The post covers tools and techniques for understanding memory allocation patterns, helping practitioners diagnose and reduce memory bottlenecks. This is practical infrastructure knowledge relevant to training large models efficiently.
Related guides (3)
Related events (8)
How Hugging Face Accelerate Runs Very Large Models Thanks to PyTorch
This Hugging Face blog post explains the technical mechanisms behind the Accelerate library for running large models that exceed single-GPU memory, leveraging PyTorch features such as device maps, CPU/disk offloading, and sharded checkpoints. It describes how models can be distributed across multiple GPUs, CPU RAM, and disk storage transparently. The post serves as both documentation and a technical explainer for practitioners working with large-scale inference and deployment.
Accelerate Large Model Training using PyTorch Fully Sharded Data Parallel
This Hugging Face blog post explains how to use PyTorch's Fully Sharded Data Parallel (FSDP) to train large models that exceed single-GPU memory limits. It covers the integration of FSDP with the Hugging Face Accelerate library, enabling distributed sharding of model parameters, gradients, and optimizer states across multiple GPUs. The post provides practical guidance on configuration and usage for scaling large model training.
Hugging Face on PyTorch / XLA TPUs
This Hugging Face blog post covers the integration of Hugging Face Transformers with PyTorch/XLA for training on Google TPUs. It describes how users can leverage TPU hardware through the XLA compiler backend to accelerate transformer model training. The post serves as a technical guide for the ecosystem connecting Hugging Face's model library with Google's TPU infrastructure.
Fixing Gradient Accumulation
A Hugging Face blog post addresses correctness issues in gradient accumulation, a common technique used to simulate larger batch sizes during neural network training when GPU memory is limited. The post likely identifies bugs or subtle implementation errors that can cause incorrect gradient estimates when accumulating gradients across multiple micro-batches. This is a practical training infrastructure topic relevant to anyone fine-tuning or pre-training large models.
Fine-tuning Llama 2 70B using PyTorch FSDP
This Hugging Face blog post details a practical workflow for fine-tuning the Llama 2 70B model using PyTorch Fully Sharded Data Parallel (FSDP), focusing on RAM-efficient techniques. The guide addresses the memory challenges of training large-scale open-weight models across multiple GPUs. It serves as a technical reference for practitioners working with frontier-scale open models on distributed infrastructure.
Improving Hugging Face Training Efficiency Through Packing with Flash Attention 2
Hugging Face published a blog post describing a technique for improving training efficiency by packing multiple short sequences into a single batch using Flash Attention 2. The approach reduces padding waste and improves GPU utilization during LLM fine-tuning. This is a practical infrastructure optimization relevant to practitioners training models on datasets with variable-length sequences.
Hugging Face Blog: Model Cards
This Hugging Face blog post discusses model cards as a documentation standard for machine learning models, covering their purpose, structure, and adoption within the ML community. Model cards provide structured metadata and transparency information about a model's intended use, limitations, training data, and evaluation results. The post likely outlines best practices and tooling support for creating and maintaining model cards on the Hugging Face Hub.
Fit More and Train Faster With ZeRO via DeepSpeed and FairScale
This Hugging Face blog post from January 2021 covers integration of ZeRO (Zero Redundancy Optimizer) memory optimization techniques via DeepSpeed and FairScale into the Transformers training ecosystem. ZeRO partitions optimizer states, gradients, and model parameters across GPUs to enable training of much larger models on the same hardware. The post serves as a practical guide for practitioners looking to scale model training without additional infrastructure investment.