Entity · technique

Tokenizers

techniqueactivetokenizers-1fe84554·1 events·first seen May 19, 2026

Aliases: Tokenizers

Co-occurring entities

More like this (12)

TOBA tokenizer In-Place Tokenizer Expansion for Pre-trained LLMs functional token ToaST (Tokenization with Split Trees)Sentence Transformers ReToken hexagonal spatial tokenization PINT (Parallel INvariant Tokenization)intra-frame token sparsification Unigram tokenisation masked-token modeling token credit assignment

Recent events (1)

5Hugging Face Blog·May 19, 2026·source ↗

Tokenization in Transformers v5: Simpler, Clearer, and More Modular

Hugging Face's Transformers v5 introduces a redesigned tokenization system aimed at being simpler, clearer, and more modular. The blog post outlines architectural changes to how tokenizers are structured and used within the library. This represents a significant API and design evolution for one of the most widely used ML frameworks in the ecosystem.

Inference Economics Agent and Tool Ecosystem Transformers Hugging Face Tokenizers