residual-vector-quantization-d1811a04·1 events·first seen Aliases: residual vector quantization
HERMES is a data-derived labeling substrate that annotates each document once into a coarse-to-fine hierarchical code using a Learned Semantic Transform and 3-stage residual vector quantization, supporting up to ~130k cells at the finest granularity. The key contribution is reframing data mixture design from selecting among fixed label sets to navigating a reusable granularity hierarchy. In 1B-parameter, 25B-token pre-training experiments, the hierarchy reveals granularity-dependent interactions: a combined coverage-quality rule lifts a 16-task capability macro-average by +0.0253 at one prefix length but loses its edge at the next finer level as candidate pools shrink ~5x. The work argues the bottleneck in data mixing is the label system rather than the mixer itself.