Semi-supervised knowledge transfer for deep learning from private training data
OpenAI published research on semi-supervised knowledge transfer techniques for training deep learning models on private data, an early contribution to privacy-preserving machine learning. The work addresses how to leverage private training data without exposing sensitive information, using knowledge distillation-style approaches. This is a 2016 archival post surfaced from OpenAI's blog.
Related guides (3)
Related events (8)
Creating Privacy Preserving AI with Substra
This Hugging Face blog post covers Substra, a federated learning framework developed by Owkin for privacy-preserving AI. The post describes how Substra enables collaborative model training across institutions without sharing raw data, targeting healthcare and biomedical use cases. It represents a practical deployment pattern for federated learning in sensitive data environments.
Interpretable Machine Learning Through Teaching
OpenAI published a method in 2018 that trains AI systems to teach each other using examples that are also interpretable to humans. The approach automatically selects maximally informative examples to convey a concept, such as representative images for a category like 'dogs'. Experiments showed the method effective at teaching both AI systems and humans, bridging machine learning interpretability with pedagogical example selection.
HumP-KD: Uncertainty-aware multi-stage knowledge distillation for efficient fire classification
Researchers propose HumP-KD, a knowledge distillation framework that compresses two heterogeneous transformer teachers (Swin-Tiny and ViT-Base) into a lightweight MobileViT-S student for real-time fire classification. The student model achieves 0.9876 mean F1 on a 31K-image dataset while retaining only 4.94M parameters—a 5.7× reduction over Swin-Tiny—and runs at 37.72 CPU FPS. The framework combines hierarchical feature alignment, spatial attention masking, and progressive multi-stage distillation to maintain accuracy under degraded visual conditions.
Open-sourcing Knowledge Distillation Code and Weights of SD-Small and SD-Tiny
Hugging Face has open-sourced knowledge distillation code and model weights for two compressed variants of Stable Diffusion: SD-Small and SD-Tiny. These distilled models are smaller and faster than the original Stable Diffusion, targeting inference efficiency. The release includes both the trained weights and the distillation training code, enabling the community to reproduce or extend the work.
Introducing OpenAI Privacy Filter
OpenAI has released an open-weight model called Privacy Filter designed to detect and redact personally identifiable information (PII) in text. The model is described as achieving state-of-the-art accuracy on PII detection tasks. This is OpenAI's first open-weight release focused specifically on data privacy and compliance use cases.
IntraShuffler: Privacy-Preserving Framework for Heterogeneous DP Federated Learning
This paper identifies a novel Privacy Inference Attack against heterogeneous differential privacy federated learning (HDP-FL) systems, where an honest-but-curious server exploits epsilon-aware aggregation and gradient denoising to infer client data distributions and link updates across rounds. To counter this, the authors propose IntraShuffler, a middleware framework that groups clients into privacy-compatible buckets and performs parameter-level shuffling within buckets, preserving epsilon-aware aggregation while disrupting persistent gradient structure. Experiments on four datasets show IntraShuffler reduces gradient recoverability by over 60% and drops surrogate inference accuracy from 0.78 to 0.33 with minimal utility loss.
VaultGemma: The world's most capable differentially private LLM
DeepMind introduces VaultGemma, a large language model trained from scratch using differential privacy (DP), claiming it as the most capable DP-trained model to date. The announcement positions VaultGemma as a significant advance in privacy-preserving AI, combining strong utility with formal privacy guarantees. The blog post is brief and likely precedes a more detailed technical disclosure.
OpenAI Data Partnerships
OpenAI announced a data partnerships program aimed at collaborating with external organizations to create both open-source and private datasets for AI training. The initiative seeks to expand the diversity and quality of training data available to OpenAI. This represents a structured effort to source large-scale, high-quality data from institutional partners rather than relying solely on existing web-scraped corpora.