Gemma3-270M

BitNet Text Embeddings: Ternary-weight LLM encoders with multi-precision output vectors

BITEMBED is a new framework that converts pretrained LLM backbones (tested on Qwen3-0.6B and Gemma3-270M) into BitNet-style text embedding encoders using ternary weights, quantized activations, and lightweight normalization refinement. The system applies continual contrastive pre-training and supervised fine-tuning with similarity-distribution and attention-relation distillation from a full-precision teacher. Evaluated on MMTEB (English v2), BITEMBED achieves performance largely comparable to full-precision teacher embedders while supporting flexible output embedding precisions to trade off storage cost. The work targets the dual bottleneck of inference compute and vector index storage in large-scale retrieval systems.