5arXiv cs.AI (Artificial Intelligence)·
HERMES: Hierarchical multi-granularity labeling substrate for pre-training data mixture design
HERMES is a data-derived labeling substrate that annotates each document once into a coarse-to-fine hierarchical code using a Learned Semantic Transform and 3-stage residual vector quantization, supporting up to ~130k cells at the finest granularity. The key contribution is reframing data mixture design from selecting among fixed label sets to navigating a reusable granularity hierarchy. In 1B-parameter, 25B-token pre-training experiments, the hierarchy reveals granularity-dependent interactions: a combined coverage-quality rule lifts a 16-task capability macro-average by +0.0253 at one prefix length but loses its edge at the next finer level as candidate pools shrink ~5x. The work argues the bottleneck in data mixing is the label system rather than the mixer itself.
Related guides (2)
Related events (8)
Iterative pseudo-labeling approach improves Mandarin-English code-switching ASR
A new arXiv preprint introduces a three-phase iterative pseudo-labeling framework for code-switching automatic speech recognition (ASR), applied here to Mandarin-English mixing. The method generates pseudo-labels from unlabeled corpora, trains a bilingual model in two stages, and iteratively refines it, achieving Mix Error Rate reductions of 6.35% and 8.29% on the SEAME benchmark's devman and devsge subsets. This is the first application of iterative pseudo-labeling to code-switching ASR, addressing the chronic data scarcity problem in this domain.
LLMSurgeon: Post-Hoc Auditing of LLM Pretraining Data Mixtures
LLMSurgeon formalizes Data Mixture Surgery (DMS), a framework for estimating the domain-level distribution of an LLM's pretraining corpus using only generated text from the target model. The method casts DMS as an inverse problem under the label-shift assumption, using a calibrated soft confusion matrix to correct domain confusion and recover the latent mixture prior. The authors also introduce LLMScan, a verifiable evaluation suite built from open-source LLMs with known pretraining mixtures, on which LLMSurgeon demonstrates high-fidelity recovery of domain compositions without access to training data.
CausalMix frames LLM data mixture optimization as causal inference to generalize across data pool shifts
CausalMix proposes treating data mixture optimization for LLM training as a causal inference problem, using Conditional Average Treatment Effect (CATE) estimation to infer optimal domain mixtures without costly retraining when the data pool changes. The method fits a causal model on 512 runs of Qwen2.5-0.5B and extrapolates the resulting mixture to train a 7B model, also generalizing to long chain-of-thought data on Qwen3-4B-Base. It outperforms RegMix and other baselines across multiple downstream tasks while providing interpretable visual analysis of mixing strategies via a CATE Interpreter. The approach addresses a practical scalability limitation in existing proxy-model-based mixture methods.
HeRo: Hierarchical Vocabulary Routing enables selective disclosure watermarking for LLMs
Researchers introduce Hierarchical Vocabulary Routing (HeRo), a watermarking framework for LLMs that enables selective disclosure of embedded metadata. Unlike existing multi-bit watermarking schemes that require revealing the entire embedded message for any verification, HeRo recursively partitions the vocabulary and distributes watermark information across hierarchical layers, allowing different verifiers to decode only the portions corresponding to their access level. The authors claim the method preserves text quality by maintaining unbiased sampling, and demonstrate high detection accuracy with low latency. Code is publicly available.
N-GRPO: Semantic Neighbor Mixing for Improved Policy Optimization in LLM Reasoning
A new arXiv preprint introduces N-GRPO, an exploration strategy for the GRPO reinforcement learning framework that improves solution diversity during rollout by mixing embeddings of anchor tokens with their nearest semantic neighbors rather than using token-level sampling or random noise. The method is evaluated on DeepSeek-R1-Distill-Qwen models of various sizes and shows consistent improvements on math reasoning benchmarks plus out-of-distribution generalization. The work targets a known limitation in RLHF-style training: redundant rollout trajectories that reduce effective learning signal.
CHERRY: Compressed Hierarchical Experts with Recurrent Representational Yield — three compute-efficient LM training techniques
A preprint from arXiv introduces CHERRY, a suite of three complementary techniques for compute-efficient language model training: Selective Ground Truth Token Training (SGT) that concentrates supervision on ~15% of semantically loaded tokens while recovering ~67% of full-sequence loss reduction; depth compression that shrinks a 48-layer 1B-parameter model to 6 layers (227M) via layer averaging and recurrent unrolling, matching a 566M dense model's loss; and a Mixture of Efficient Experts (MoEE) assembly that outperforms individual compressed models at comparable active parameters. The techniques are validated on CHERRY-1.8B, a Korean-language foundation model trained entirely from scratch using these methods. Authors are transparent about scope limitations: one model family, Korean data, and loss-based metrics only.
Can Foundation Models Label Data Like Humans?
This Hugging Face blog post examines whether foundation models can serve as substitutes for human annotators in RLHF data labeling pipelines. It investigates the reliability and quality of model-generated preference labels compared to human-generated ones, with implications for scalable oversight and alignment research. The analysis is framed around the Open LLM Leaderboard and RLHF methodology.
Multi-Task Bayesian In-Context Learning for Amortized Hierarchical Inference
A new arXiv preprint introduces a multi-task in-context learning framework for amortized hierarchical Bayesian predictive inference, representing prior information as a prefix of in-context datasets fed to a transformer. The model learns to adapt predictions across families of priors, addressing the brittleness of prior-data fitted models under distribution shift. On evaluations including out-of-meta-distribution priors and high-dimensional latent structures, the method matches oracle Bayesian predictors while being orders of magnitude faster, with a real-world spatiotemporal temperature prediction demonstration.