technique

Tokenizers

techniqueactivetokenizers-1fe84554·1 events·first seen 28d ago

Aliases: Tokenizers

Co-occurring entities

More like this (12)

TOBA tokenizer functional token ToaST (Tokenization with Split Trees)Sentence Transformers hexagonal spatial tokenization intra-frame token sparsification Unigram tokenisation masked-token modeling token credit assignment sentence embeddings static embeddings CodeAgents

Recent events (1)

5Hugging Face Blog·28d ago·source ↗

Tokenization in Transformers v5: Simpler, Clearer, and More Modular

Hugging Face's Transformers v5 introduces a redesigned tokenization system aimed at being simpler, clearer, and more modular. The blog post outlines architectural changes to how tokenizers are structured and used within the library. This represents a significant API and design evolution for one of the most widely used ML frameworks in the ecosystem.

Inference Economics Agent and Tool Ecosystem Transformers Hugging Face Tokenizers