multihashformer-c0fc696a·1 events·first seen Aliases: MultiHashFormer
Researchers introduce MultiHashFormer, a framework that replaces standard embedding matrices with hash-based token representations for causal language models. Each token is encoded as a unique signature of discrete hash IDs from multiple independent hash functions, compressed into a latent vector for a Transformer decoder. Evaluated at 100M, 1B, and 3B parameter scales, MultiHashFormer outperforms standard Transformer LMs on multiple benchmarks while enabling multilingual vocabulary expansion at constant parameter cost.