Transformers
transformers-19fbbbe4·39 events·first seen 1mo agoAliases: Transformers, Transformers.js, Transformers v5, Transformer, transformer
Co-occurring entities
More like this (12)
Recent events (39)
Transformers.js v3: WebGPU Support, New Models & Tasks, and More
Hugging Face released Transformers.js v3, a major update to its JavaScript inference library enabling on-device ML in browsers and Node.js. The release adds WebGPU backend support for hardware-accelerated inference, expands the supported model and task catalog, and improves overall performance. This brings browser-side AI inference closer to parity with native runtimes for a wider range of use cases.
Transformers v5: Simple model definitions powering the AI ecosystem
Hugging Face has announced Transformers v5, a major version update to its flagship open-source library. The release focuses on simplified model definitions and architectural improvements to the codebase. As one of the most widely used ML libraries in the ecosystem, this update has broad implications for researchers and practitioners building on top of the Transformers framework.
Transformers.js v4: Now Available on NPM
Hugging Face has released Transformers.js v4, a major version update to its JavaScript library for running transformer models in the browser and Node.js, now published on NPM. The release likely includes updated model support, performance improvements, and API changes. This continues the trend of bringing ML inference capabilities directly to JavaScript environments without requiring a Python backend.
Tokenization in Transformers v5: Simpler, Clearer, and More Modular
Hugging Face's Transformers v5 introduces a redesigned tokenization system aimed at being simpler, clearer, and more modular. The blog post outlines architectural changes to how tokenizers are structured and used within the library. This represents a significant API and design evolution for one of the most widely used ML frameworks in the ecosystem.
The Transformers Library: Standardizing Model Definitions
Hugging Face published a blog post outlining their approach to standardizing model definitions within the Transformers library. The post addresses how the library structures and maintains model code to ensure consistency, reproducibility, and ease of integration across a wide range of architectures. This is a tooling and ecosystem development relevant to practitioners building on or contributing to the Transformers framework.
Chat Templates: An End to the Silent Performance Killer
This Hugging Face blog post addresses the problem of inconsistent chat formatting across language models, where mismatched prompt templates silently degrade model performance. It introduces a standardized chat template system in the transformers library that encodes each model's expected conversation format directly into its tokenizer. The post argues that using the wrong chat format can cause significant but hard-to-detect performance drops, making standardization critical for reliable deployment.
Making ML-powered web games with Transformers.js
This Hugging Face blog post demonstrates how to build machine learning-powered web games using Transformers.js, enabling in-browser inference without a server backend. The post covers practical implementation patterns for running transformer models directly in the browser via WebAssembly and WebGL. It serves as both a tutorial and a showcase of client-side ML deployment capabilities.
The PR you would have opened yourself
A Hugging Face blog post discussing a pull request related to converting or integrating Transformers models with MLX, Apple's machine learning framework. The post appears to cover tooling or workflow improvements for running Hugging Face Transformers models on Apple Silicon via MLX. The title suggests a community or automated contribution narrative.
Making LLMs lighter with AutoGPTQ and transformers
Hugging Face announces native integration of AutoGPTQ into the transformers library, enabling 4-bit quantized inference for large language models. The integration allows users to load and run GPTQ-quantized models directly through the standard transformers API with minimal code changes. This lowers the hardware barrier for deploying LLMs by significantly reducing VRAM requirements while maintaining competitive performance.
Getting Started with Transformers on Habana Gaudi
This Hugging Face blog post introduces integration between the Transformers library and Habana Gaudi AI accelerators. It provides a practical guide for running transformer model training and inference on Gaudi hardware as an alternative to GPU-based infrastructure. The post signals growing ecosystem support for non-NVIDIA AI accelerator hardware.
Improving Language Understanding with Unsupervised Learning (GPT-1)
OpenAI published the GPT-1 paper in June 2018, demonstrating state-of-the-art results across diverse language tasks by combining transformer architectures with unsupervised pre-training followed by supervised fine-tuning. The approach is task-agnostic and scalable, showing that pre-training on large unlabeled text corpora and then fine-tuning on specific tasks yields strong generalization. This work established the foundational paradigm that would evolve into GPT-2, GPT-3, and subsequent large language models.
Making LLMs even more accessible with bitsandbytes, 4-bit quantization and QLoRA
Hugging Face published a blog post detailing the integration of 4-bit quantization via bitsandbytes into the Transformers library, enabling large language models to run on consumer-grade hardware. The post covers NF4 (NormalFloat4) data type and double quantization techniques from the QLoRA paper, which together reduce memory footprint significantly while preserving model quality. It demonstrates how users can load models like LLaMA in 4-bit precision and fine-tune them using QLoRA with minimal code changes.
The State of Computer Vision at Hugging Face
Hugging Face published a survey of the computer vision ecosystem available through its platform as of early 2023, covering supported model architectures, tasks, datasets, and tooling. The post reviews progress in image classification, object detection, segmentation, and multimodal vision-language models integrated into the Transformers library. It serves as a reference for practitioners on what CV capabilities are accessible via the Hugging Face hub and APIs.
A Gentle Introduction to 8-bit Matrix Multiplication for Transformers at Scale using Hugging Face and bitsandbytes
This Hugging Face blog post introduces 8-bit quantization for large transformer models via integration of the bitsandbytes library with the transformers and accelerate libraries. It explains how LLM.int8() enables loading large models in 8-bit precision, significantly reducing GPU memory requirements without major accuracy degradation. The post covers the technical mechanics of mixed-precision decomposition and how practitioners can use the integration in practice.
Convert Transformers to ONNX with Hugging Face Optimum
Hugging Face published a guide on converting Transformer models to ONNX format using the Optimum library. The post covers the tooling workflow for exporting models from the Transformers ecosystem into ONNX for optimized inference deployment. This is a practical infrastructure topic relevant to production ML deployment patterns.
Introducing Optimum: The Optimization Toolkit for Transformers at Scale
Hugging Face announced Optimum, an optimization toolkit designed to accelerate Transformers models on various hardware backends. The toolkit aims to bridge the gap between Transformers model development and hardware-specific optimizations from partners. It provides a unified interface for quantization, pruning, and hardware-accelerated inference across different accelerators.
Image GPT: Transformer Models Applied to Pixel Sequences for Image Generation and Classification
OpenAI demonstrates that a large transformer model trained autoregressively on pixel sequences can generate coherent image completions and samples, analogous to text generation. The work establishes a correlation between generative sample quality and downstream image classification accuracy. The best generative model achieves features competitive with top convolutional networks in the unsupervised setting, suggesting shared representational principles across modalities.
Language Models Need Sleep: Periodic Context Consolidation via Fast Weights and SSM Blocks
This paper proposes a sleep-like consolidation mechanism for transformer-based LLMs to address the quadratic scaling of attention with context length. During 'sleep' phases, the model performs N offline recurrent passes over accumulated context, updating fast weights in state-space model (SSM) blocks via a learned local rule, then clears the KV cache. The approach is evaluated on synthetic tasks (cellular automata, multi-hop graph retrieval) and math reasoning, where standard transformers and SSM-attention hybrids fail, with performance scaling with sleep duration N.
Mixture of Experts (MoEs) in Transformers
A Hugging Face blog post covering Mixture of Experts (MoE) architectures as applied to transformer models. The post likely explains the technical foundations, training considerations, and practical deployment aspects of MoE models. Given the timing in early 2026, it likely contextualizes recent MoE-based frontier models and tooling support within the Hugging Face ecosystem.
You Could Have Designed State of the Art Positional Encoding
A Hugging Face blog post walks through the design space of positional encoding for transformer models, building intuition for why modern schemes like RoPE emerged. The post takes a pedagogical approach, showing how one could derive state-of-the-art positional encoding from first principles. It covers the evolution from absolute to relative positional encodings and the properties that make certain schemes preferable for long-context generalization.
License to Call: Introducing Transformers Agents 2.0
Hugging Face announced Transformers Agents 2.0, a major update to their agent framework built on top of the Transformers library. The release introduces new abstractions for tool use, multi-step reasoning, and agent orchestration, positioning it as a production-ready framework for building AI agents. The update reflects growing ecosystem investment in standardized agent tooling patterns.
Yes, Transformers are Effective for Time Series Forecasting (+ Autoformer)
A Hugging Face blog post examines the effectiveness of Transformer architectures for time series forecasting, with a focus on the Autoformer model. The post addresses ongoing debate about whether Transformers are suitable for time series tasks, countering claims that simpler linear models outperform them. It covers the Autoformer architecture's decomposition-based approach and its integration into the Hugging Face ecosystem.
Introducing RWKV - An RNN with the advantages of a transformer
Hugging Face introduces RWKV, a recurrent neural network architecture that claims to combine the parallelizable training of transformers with the efficient linear-time inference of RNNs. The model avoids the quadratic attention bottleneck of standard transformers while maintaining competitive performance. RWKV represents an alternative architectural direction to the dominant transformer paradigm for language modeling.
How Hugging Face Sped Up Transformer Inference 100x for API Customers
Hugging Face describes engineering optimizations that achieved up to 100x speedups in transformer inference for their hosted API customers. The post covers techniques applied to accelerate model serving at scale. This is a 2021 article documenting early inference optimization work at Hugging Face's inference API product.
Deep Double Descent: Universal Phenomenon in CNNs, ResNets, and Transformers
OpenAI researchers demonstrate that the double descent phenomenon—where model performance improves, degrades, then improves again—occurs universally across CNNs, ResNets, and transformers as a function of model size, data size, or training time. The effect can often be masked by careful regularization, which may explain why it has been underappreciated. The underlying mechanism remains poorly understood, and the authors identify it as an important open research direction.
In-Context Reward Adaptation for Robust Preference Modeling
This paper proposes In-Context Reward Adaptation (ICRA), a transformer-based framework that infers reward structures from small sets of preference demonstrations at inference time, without retraining. The key finding is that standard transformers exhibit asymptotic bias toward ground-truth rewards, but incorporating human response time as an auxiliary signal resolves this limitation and enables generalization to unseen preference domains. The approach addresses a core limitation of static RLHF reward models, which fail to handle heterogeneous or shifting human value distributions.
Positional vs. Symbolic Attention Heads: Learning Dynamics, RoPE Geometry, and Length Generalization
Researchers train a decoder-only Transformer (GPT-J) on two structurally equivalent multi-hop reasoning tasks to study how attention heads specialize into positional or symbolic roles during learning. They find that successful task learning correlates with the emergence of 'pure' heads—exclusively positional or symbolic—and provide theoretical constructions showing how single-layer RoPE-based attention realizes these functions geometrically. A novel 'discrepancy' metric formalizes the robustness difference between the two head types, with symbolic mechanisms shown to extrapolate more reliably to longer sequences than positional ones. The findings have implications for understanding length generalization failures in RoPE-based models.
Functional Attention: Reinterpreting Attention as Functional Correspondences for Operator Learning
This paper introduces Functional Attention, a novel attention mechanism for operator learning that replaces standard softmax token-wise affinities with structured linear operators inspired by geometric functional maps. The approach treats attention as a correspondence between adaptive bases rather than discrete tokens, yielding a resolution-invariant, globally-aware representation. Experiments show competitive or state-of-the-art performance on PDE solving, 3D segmentation, and regression tasks, with robustness to varying discretizations.
Unlocking Longer Generation with Key-Value Cache Quantization
This Hugging Face blog post covers KV cache quantization as a technique to reduce memory consumption during LLM inference, enabling longer context generation without proportional VRAM increases. The post likely explains how quantizing the key-value cache (e.g., to INT8 or lower precision) trades minimal accuracy for significant memory savings. This is directly relevant to inference efficiency and long-context deployment patterns.
Fine-tuning LLMs to 1.58bit: extreme quantization made easy
Hugging Face published a blog post describing a method for fine-tuning large language models down to 1.58-bit precision, referencing the BitNet b1.58 quantization scheme. The post covers tooling and workflows that make extreme quantization more accessible via the Hugging Face ecosystem. This represents a practical guide to applying ternary-weight quantization ({-1, 0, 1}) to existing models through fine-tuning rather than training from scratch.
Faster Text Generation with TensorFlow and XLA
This Hugging Face blog post describes how to accelerate text generation using TensorFlow's XLA (Accelerated Linear Algebra) compilation. The post covers techniques for applying XLA JIT compilation to transformer-based text generation pipelines to achieve significant speedups. It targets practitioners using TF-based models who want inference performance improvements without switching frameworks.
Graphcore and Hugging Face Launch New Lineup of IPU-Ready Transformers
Graphcore and Hugging Face announced a collaboration to make transformer models compatible with Graphcore's Intelligence Processing Unit (IPU) hardware. The partnership expands the set of Hugging Face models that can run natively on IPU infrastructure. This represents an effort to broaden the hardware ecosystem available for transformer model inference and training beyond GPUs.
Introducing Decision Transformers on Hugging Face
Hugging Face introduces support for Decision Transformers, a framework that casts offline reinforcement learning as a sequence modeling problem using transformer architectures. The blog post covers the conceptual basis of Decision Transformers and their integration into the Hugging Face ecosystem. This represents an early step in bringing RL-based model paradigms into the standard ML tooling stack.
Hugging Face and Graphcore Partner for IPU-Optimized Transformers
Hugging Face and Graphcore announced a partnership to optimize Transformer models for Graphcore's Intelligence Processing Unit (IPU) hardware. The collaboration aims to make IPU-accelerated inference and training accessible through the Hugging Face ecosystem. This represents an early effort to broaden AI hardware options beyond GPU-dominated infrastructure.
Understanding BigBird's Block Sparse Attention
This Hugging Face blog post provides a technical explanation of BigBird's block sparse attention mechanism, which extends transformer models to handle longer sequences by replacing dense quadratic attention with a combination of local, global, and random sparse attention patterns. The post covers the theoretical underpinnings and implementation details of how BigBird achieves linear complexity with respect to sequence length. It serves as educational commentary on a published research architecture that enables processing of sequences up to 4096 tokens or more efficiently.
Looped Diffusion Language Models (LoopMDM): Depth Scaling via Layer Looping
LoopMDM introduces selective looping of early-middle transformer layers in masked diffusion language models, achieving a depth-scaling effect without adding parameters. The approach matches same-size MDM performance with up to 3.3× fewer training FLOPs and outperforms deeper non-looped MDMs on reasoning benchmarks, including up to 8.5 points improvement on GSM8K. Inference-time compute scaling is enabled by varying loop counts, with adaptive loop scheduling providing additional efficiency gains. Attention analysis suggests looping works by promoting interactions among masked token positions.
Complete-muE: Optimal Hyperparameter Transfer and Scaling for MoE Models
Complete-muE is a framework for transferring hyperparameters across dense FFN and Mixture-of-Experts (MoE) transformer architectures, addressing limitations of existing tools like μP and SDE that cannot handle simultaneous architecture and token-per-expert changes. It uses a two-bridge system: Bridge I maps dense FFN to Dense MoE via active-width μP with normalized router scale, and Bridge II maps Dense MoE to sparse MoE via activated-expert scaling with a first-order SDE correction. The practical outcome is a 'tune dense once, transfer to all' recipe that enables near-optimal hyperparameter reuse across MoE configurations without costly re-tuning. Experiments on language model and diffusion model pretraining confirm stable hyperparameter optima across architectures and parameter counts.
CHARM: Multimodal JEPA for Semantic Time-Series Embeddings via Channel-Aware Representation Learning
CHARM (Channel-Aware Representation Model) is a new Transformer-based architecture for general-purpose representation learning over heterogeneous multivariate time series. It integrates channel-level textual descriptions into a permutation-equivariant encoder trained with a Joint Embedding Predictive Architecture (JEPA) and a novel temporally stable embedding loss. The model achieves strong performance across anomaly detection, classification, and forecasting tasks using only a linear probe, with text descriptions primarily serving as channel identifiers enabling cross-dataset generalization.
Google's AlphaGenome Interprets Non-Coding DNA That Regulates Genetic Expression
Google has released AlphaGenome, an open-weights model that interprets the ~98% of human and mouse genomes that regulate gene expression rather than coding for proteins. The model takes up to 1 million DNA base pairs as input and outputs roughly 6,000 human and 1,000 mouse gene properties, using a CNN-transformer-CNN architecture trained via ensemble distillation from 64 pretrained models. Across 50 evaluations, AlphaGenome matched or exceeded prior models in 47 cases, and correctly predicted expression changes associated with T-cell acute lymphoblastic leukemia. Weights, API, and inference code are freely available for noncommercial use.