5arXiv cs.CL (Computation and Language)·
HeRo: Hierarchical Vocabulary Routing enables selective disclosure watermarking for LLMs
Researchers introduce Hierarchical Vocabulary Routing (HeRo), a watermarking framework for LLMs that enables selective disclosure of embedded metadata. Unlike existing multi-bit watermarking schemes that require revealing the entire embedded message for any verification, HeRo recursively partitions the vocabulary and distributes watermark information across hierarchical layers, allowing different verifiers to decode only the portions corresponding to their access level. The authors claim the method preserves text quality by maintaining unbiased sampling, and demonstrate high detection accuracy with low latency. Code is publicly available.
Related guides (2)
Related events (8)
Reroute: Training-free recoverable visual token routing for vision-language models
A new arXiv preprint proposes Reroute, a training-free plug-in that replaces the standard rank-and-remove visual token pruning paradigm in VLMs with a recoverable routing mechanism. Instead of permanently discarding low-ranked tokens, Reroute defers them to re-enter the candidate pool at later decoder stages, addressing the problem that token importance shifts across decoder depth. Evaluated on LLaVA-1.5 and Qwen backbones augmented with FastV, PDrop, and Nüwa pruning methods, Reroute improves grounding performance under aggressive token reduction without sacrificing general VQA accuracy. The approach preserves the theoretical compute and KV-cache budget of the underlying pruning method.
HERMES: Hierarchical multi-granularity labeling substrate for pre-training data mixture design
HERMES is a data-derived labeling substrate that annotates each document once into a coarse-to-fine hierarchical code using a Learned Semantic Transform and 3-stage residual vector quantization, supporting up to ~130k cells at the finest granularity. The key contribution is reframing data mixture design from selecting among fixed label sets to navigating a reusable granularity hierarchy. In 1B-parameter, 25B-token pre-training experiments, the hierarchy reveals granularity-dependent interactions: a combined coverage-quality rule lifts a 16-task capability macro-average by +0.0253 at one prefix length but loses its edge at the next finer level as candidate pools shrink ~5x. The work argues the bottleneck in data mixing is the label system rather than the mixer itself.
SARA framework aligns MoE routing distributions to improve low-resource multilingual performance
Researchers introduce SARA (Semantically Anchored Routing Alignment), a framework that addresses cross-lingual routing divergence in sparse Mixture-of-Experts LLMs by aligning the internal routing distributions of low-resource language tokens to match those of high-resource semantic anchors via symmetric Jensen-Shannon divergence constraints. Unlike logit-level distillation, SARA operates directly on MoE routing layers to encourage mechanistic consistency in expert selection across languages. Experiments on Qwen3-30B-A3B and Phi-3.5-MoE-instruct across 5 low-resource languages show modest but consistent gains (up to +1.2%) on Global-MMLU over standard instruction tuning.
AuRA: Distilling audio understanding into LLMs via LoRA adaptation
AuRA is a new method for integrating speech understanding into LLMs by distilling audio encoding capability directly into LoRA-adapted model weights, bypassing cascaded ASR-LLM pipelines. A lightweight audio embedding layer feeds speech to both an ASR encoder (teacher) and a LoRA-adapted LLM (student), with layer-wise distillation aligning hidden states. The approach claims to outperform cascaded systems, bridge-based adaptation baselines, and large-scale multimodal models on multiple speech-language benchmarks while enabling parallel end-to-end inference without large-scale multimodal training.
HPRO: Hierarchical Progressive Reward Optimization for Emotional Text-to-Speech
Researchers propose HPRO, a hierarchical progressive reward optimization framework for LLM-based Text-to-Speech systems that improves emotional expressiveness. The approach introduces HD-Emo codec as a differentiable reward model that separates content and style preference tokens to avoid conflicting gradients, and bridges sentence-level and frame-level reward signals through progressive multi-scale alignment. Experiments show improved emotional expressiveness while preserving linguistic intelligibility.
LLM-as-a-Verifier: Training-free verification framework scales along granularity, repetition, and criteria decomposition
Researchers introduce LLM-as-a-Verifier, a general-purpose verification framework that treats verification as a new scaling axis for LLMs, computing continuous scores from token logit distributions rather than discrete judge outputs. The framework scales along three dimensions—score granularity, repeated evaluation, and criteria decomposition—and achieves state-of-the-art results on Terminal-Bench V2 (86.5%), SWE-Bench Verified (78.2%), RoboRewardBench (87.4%), and MedAgentBench (73.3%) without requiring additional training. The authors also demonstrate that the framework's fine-grained signals can serve as dense RL feedback, improving sample efficiency for SAC and GRPO on robotics and math benchmarks, and build a Claude Code extension for monitoring agentic systems.
IMLogic benchmark and RootMem framework target implicit logical memory retrieval for personalized LLMs
Researchers introduce IMLogic, a benchmark for evaluating implicit logical memory retrieval in long-dialogue personalized LLM scenarios, addressing gaps in existing semantic-similarity-based retrieval methods. They also propose RootMem, a plug-and-play framework that distills user histories into structured 'root memories' and uses an LLM-based router to activate logically relevant memories alongside semantic retrieval. Experiments show RootMem outperforms retrieval baselines and improves existing memory agents. The work targets a concrete weakness in current personalized LLM memory systems where logically critical memories lack semantic overlap with queries.
Localized LoRA-MoE: Block-wise and cell-wise low-rank expert routing for multi-task PEFT
A new arXiv preprint introduces Localized LoRA-MoE, a parameter-efficient fine-tuning framework that combines spatial blocking with dynamic context-conditioned routing to address gradient interference in multi-task LoRA adapters. The authors propose two architectures: Block-Wise LoRA-MoE with centralized macro-routing and Cell-Wise LoRA-MoE with decentralized per-cell expert gating. Empirical results across SVD matrix simulations, tabular tasks, and spatial vision benchmarks show that decentralized cell-level gating matches a global coordinator while providing fault isolation against sensor degradation and task-switching instability.