TRL v1.0: Post-Training Library Built to Move with the Field
Hugging Face has released TRL v1.0, a major milestone for its post-training library focused on reinforcement learning from human feedback and related alignment techniques. The release signals a stabilization of the API and feature set after iterative development tracking the rapidly evolving post-training landscape. TRL is widely used in the open-source community for fine-tuning and aligning language models using methods such as PPO, DPO, and GRPO.
Related guides (3)
Related events (8)
Liger GRPO meets TRL: Efficient Reinforcement Learning Training Integration
The Hugging Face blog post announces the integration of Liger Kernel's GRPO (Group Relative Policy Optimization) implementation with TRL (Transformer Reinforcement Learning library). This combination aims to improve memory efficiency and training throughput for RL-based fine-tuning of language models. The integration targets practitioners running GRPO-style training on constrained hardware budgets.
Putting RL back in RLHF: RLOO Implementation on Hugging Face
Hugging Face published a blog post introducing RLOO (REINFORCE Leave-One-Out), a reinforcement learning algorithm aimed at making the RL component of RLHF more practical and effective. The post discusses implementation details and motivations for revisiting pure RL-based fine-tuning approaches within the TRL library. This represents a technical contribution to the alignment and RLHF tooling ecosystem, offering an alternative to PPO-based RLHF pipelines.
Vision Language Model Alignment in TRL
Hugging Face's TRL library has added support for aligning Vision Language Models (VLMs), extending existing RLHF and preference optimization tooling to multimodal settings. The blog post covers the new capabilities for training VLMs with alignment techniques such as DPO and related methods. This expands the open-source ecosystem for multimodal model fine-tuning and alignment.
Illustrating Reinforcement Learning from Human Feedback (RLHF)
This Hugging Face blog post provides an illustrated overview of Reinforcement Learning from Human Feedback (RLHF), explaining the technique used to align large language models with human preferences. It covers the core pipeline: pretraining a language model, collecting human preference data, training a reward model, and fine-tuning with RL. Published in December 2022, it served as an accessible reference during the period when RLHF was becoming central to frontier model development.
Finetune Stable Diffusion Models with DDPO via TRL
Hugging Face's TRL library adds support for DDPO (Denoising Diffusion Policy Optimization), enabling reinforcement learning-based finetuning of Stable Diffusion models. This extends TRL's RLHF tooling beyond language models to image generation, allowing reward-driven optimization of diffusion models. The post demonstrates practical usage of the new DDPO trainer within the TRL ecosystem.
Make LLM Fine-tuning 2x faster with Unsloth and 🤗 TRL
Hugging Face published a blog post detailing an integration between Unsloth and TRL (Transformer Reinforcement Learning) library that claims to achieve 2x faster LLM fine-tuning. The post covers how Unsloth optimizes training kernels to reduce memory usage and increase throughput. This is relevant to practitioners looking to reduce compute costs and time for fine-tuning large language models.
THUDM releases slime: RL scaling post-training framework for LLMs
THUDM (Tsinghua University's Knowledge Engineering Group) has released slime, an open-source Python framework for LLM post-training via reinforcement learning scaling. The repository has accumulated 6,548 stars with 195 added in a single day, indicating significant community interest. RL-based post-training frameworks are a key area of active development following the success of techniques like GRPO and PPO in improving reasoning capabilities.
StackLLaMA: A hands-on guide to train LLaMA with RLHF
Hugging Face published a detailed tutorial demonstrating how to fine-tune Meta's LLaMA model using Reinforcement Learning from Human Feedback (RLHF) on StackExchange data. The guide covers the full pipeline: supervised fine-tuning, reward model training, and PPO-based RL optimization. It serves as a practical reference for practitioners seeking to replicate RLHF workflows on open-weight models using the TRL library.