PyTorch FSDP
pytorch-fsdp-e301bf8f·3 events·first seen 28d agoAliases: PyTorch FSDP
Co-occurring entities
More like this (12)
Recent events (3)
Accelerate Large Model Training using PyTorch Fully Sharded Data Parallel
This Hugging Face blog post explains how to use PyTorch's Fully Sharded Data Parallel (FSDP) to train large models that exceed single-GPU memory limits. It covers the integration of FSDP with the Hugging Face Accelerate library, enabling distributed sharding of model parameters, gradients, and optimizer states across multiple GPUs. The post provides practical guidance on configuration and usage for scaling large model training.
From DeepSpeed to FSDP and Back Again with Hugging Face Accelerate
This Hugging Face blog post covers the practical migration path between DeepSpeed and PyTorch FSDP distributed training backends using the Accelerate library. It addresses configuration differences, compatibility considerations, and workflow patterns for switching between the two frameworks. The post targets practitioners running large-scale model training who need flexibility across distributed training strategies.
Fine-tuning Llama 2 70B using PyTorch FSDP
This Hugging Face blog post details a practical workflow for fine-tuning the Llama 2 70B model using PyTorch Fully Sharded Data Parallel (FSDP), focusing on RAM-efficient techniques. The guide addresses the memory challenges of training large-scale open-weight models across multiple GPUs. It serves as a technical reference for practitioners working with frontier-scale open models on distributed infrastructure.