Finally, a Replacement for BERT: Introducing ModernBERT
Hugging Face introduces ModernBERT, a modernized encoder-only transformer model designed as a successor to BERT. The model incorporates architectural improvements developed since BERT's 2018 release, targeting better performance on downstream NLP tasks. ModernBERT aims to fill the gap for efficient encoder models in retrieval, classification, and other discriminative tasks where decoder-only LLMs are often overkill.
Related guides (4)
Related events (8)
mmBERT: ModernBERT goes Multilingual
Hugging Face introduces mmBERT, a multilingual extension of ModernBERT. The post describes adapting the ModernBERT architecture for multilingual text encoding tasks. This represents an incremental but meaningful expansion of the ModernBERT family to cover non-English languages.
Pre-Train BERT with Hugging Face Transformers and Habana Gaudi
This Hugging Face blog post from August 2022 describes how to pre-train a BERT model from scratch using the Hugging Face Transformers library on Habana Gaudi hardware accelerators. It covers the full pipeline including data preparation, tokenizer training, and masked language modeling pretraining. The post serves as both a technical tutorial and a demonstration of Habana Gaudi's viability as an alternative AI training accelerator.
Accelerate BERT inference with Hugging Face Transformers and AWS Inferentia
This Hugging Face blog post describes how to deploy BERT models on AWS Inferentia chips using the Hugging Face Transformers library and Amazon SageMaker. It covers the workflow for compiling models with AWS Neuron SDK and running optimized inference on Inferentia hardware. The post targets practitioners looking to reduce inference costs and latency for transformer-based NLP workloads.
Introducing BERTopic Integration with the Hugging Face Hub
Hugging Face has announced an integration between BERTopic, a topic modeling library, and the Hugging Face Hub. This allows users to push, share, and load BERTopic models directly from the Hub, enabling easier collaboration and deployment of topic modeling workflows. The integration leverages the Hub's model card and versioning infrastructure for NLP tooling beyond generative models.
Introducing Decision Transformers on Hugging Face
Hugging Face introduces support for Decision Transformers, a framework that casts offline reinforcement learning as a sequence modeling problem using transformer architectures. The blog post covers the conceptual basis of Decision Transformers and their integration into the Hugging Face ecosystem. This represents an early step in bringing RL-based model paradigms into the standard ML tooling stack.
Multimodal Embedding & Reranker Models with Sentence Transformers
Hugging Face's Sentence Transformers library has added support for multimodal embedding and reranking models, enabling joint text-image (and potentially other modality) representations within a unified framework. The update extends the library's existing text-focused embedding capabilities to handle cross-modal retrieval and reranking tasks. This lowers the barrier for practitioners building multimodal search and RAG pipelines using open-weights models.
The Reformer - Pushing the limits of language modeling
This Hugging Face blog post covers the Reformer, a memory-efficient transformer architecture that uses locality-sensitive hashing (LSH) attention and reversible residual layers to handle very long sequences. The post explains the technical mechanisms that allow Reformer to process sequences up to 1 million tokens with significantly reduced memory footprint compared to standard transformers. It serves as an educational deep-dive into the architectural innovations introduced in the original Reformer paper by Kitaev et al.
Tokenization in Transformers v5: Simpler, Clearer, and More Modular
Hugging Face's Transformers v5 introduces a redesigned tokenization system aimed at being simpler, clearer, and more modular. The blog post outlines architectural changes to how tokenizers are structured and used within the library. This represents a significant API and design evolution for one of the most widely used ML frameworks in the ecosystem.