Fine-tuning GPT-2 from Human Preferences
OpenAI fine-tuned the 774M parameter GPT-2 model using human feedback across summarization and style-continuation tasks, requiring 60k and 5k human labels respectively. The work revealed a labeler preference misalignment: for summarization, labelers rewarded copying from source text rather than genuine summarization. The stated motivation is advancing safety techniques for human-machine interaction and learning about human values from feedback.
Related guides (3)
Related events (8)
Fine-tuning LLMs on summary-expansion tasks strips copyright alignment guardrails, enabling up to 92% verbatim book reproduction
Researchers from Stony Brook University, Carnegie Mellon University, and Columbia Law School fine-tuned DeepSeek-V3.1, Gemini 2.5 Pro, and GPT-4o on a task of expanding plot summaries into prose paragraphs, finding that this caused models to regurgitate up to 91.9% of verbatim text from books in their pretraining data. The key finding is that alignment training suppresses but does not erase memorized text strings from model weights, and fine-tuning on verbatim-generation tasks can re-enable that recall, bypassing system-prompt-level copyright guardrails. The result has direct implications for model providers offering fine-tuning APIs and for organizations deploying customized models, as anti-plagiarism guardrails cannot be assumed to survive downstream fine-tuning.
Fine-tuning now available for GPT-4o
OpenAI has launched fine-tuning support for GPT-4o, its flagship multimodal model, as of August 20, 2024. This allows developers to customize GPT-4o on their own datasets via the OpenAI API. The release extends the fine-tuning capability previously available on GPT-3.5 and GPT-4 to the most capable model in OpenAI's lineup, enabling task-specific optimization at the frontier.
Learning to Summarize with Human Feedback
OpenAI published research applying reinforcement learning from human feedback (RLHF) to train language models for improved summarization quality. The work demonstrated that models trained with human preference signals outperform those trained purely on supervised objectives for summarization tasks. This paper is an early foundational contribution to the RLHF methodology that later became central to aligning large language models.
Customizing GPT-3 for your application
OpenAI announced fine-tuning capabilities for GPT-3, enabling developers to customize the model for specific applications via a single command. This feature allows users to adapt GPT-3's behavior to their use case by training on domain-specific data. The announcement marks an early milestone in making large language model customization accessible through an API.
Better language models and their implications
OpenAI announced GPT-2, a large-scale unsupervised language model capable of generating coherent multi-paragraph text and achieving state-of-the-art performance on language modeling benchmarks. The model demonstrated zero-shot capability across reading comprehension, machine translation, question answering, and summarization without task-specific fine-tuning. OpenAI notably withheld the full model release citing misuse concerns, marking an early high-profile instance of staged/responsible release policy.
GPT-2: 6-Month Follow-Up — 774M Parameter Model Released
OpenAI released the 774 million parameter version of GPT-2 as part of its staged release strategy, following the 124M model in February and 355M model in May 2019. The release is accompanied by an open-source legal agreement to facilitate model-sharing partnerships between organizations. OpenAI also published a technical report on coordinating with the AI research community around publication norms and staged disclosure practices.
GPT-3.5 Turbo fine-tuning and API updates
OpenAI has opened fine-tuning access for GPT-3.5 Turbo, allowing developers to customize the model with their own data for specific use cases. This extends fine-tuning capabilities previously available on older GPT-3 models to the more capable Turbo variant. The announcement also includes associated API updates to support this functionality.
Aligning language models to follow instructions
OpenAI published a blog post describing their work on aligning language models to follow human instructions, corresponding to the InstructGPT research. This work introduced reinforcement learning from human feedback (RLHF) as a core technique for training models to be more helpful, honest, and aligned with user intent. The approach demonstrated that smaller instruction-tuned models could outperform larger base models on human preference evaluations, marking a foundational shift in how language models are trained and deployed.