5arXiv cs.CL (Computation and Language)·
PINT: Parallel utterance training yields near-speaker-invariant discrete speech tokens
Researchers introduce PINT (Parallel INvariant Tokenization), a method that fine-tunes a self-supervised speech encoder using alignment losses across parallel utterances from multiple speakers to extract purely linguistic content tokens. The approach achieves a 98.7% relative reduction in speaker probe accuracy and 42% lower ABX error rate compared to baselines, while preserving frame-level temporal grounding. PINT tokens are designed as drop-in semantic targets for audio codecs, addressing a known limitation of HuBERT-style SSL models that leak speaker identity and prosody into discrete representations.
Related guides (2)
Related events (8)
Pair-In, Pair-Out (PIPO): Unified Latent Compression and Multi-Token Prediction for Efficient LLM Inference
PIPO is a new inference efficiency framework that unifies input-side latent compression with output-side multi-token prediction (MTP) by treating them as mirror operations: a compressor folds two input tokens into one latent, while an MTP head unfolds one hidden state into an additional output token. To avoid the expensive verifier pass typically required by speculative decoding, PIPO trains a lightweight confidence head using On-Policy Distillation (OPD), which naturally aligns with rejection-sampling criteria. Experiments on Qwen3.5-4B and 9B backbones across AIME 2025, GPQA-Diamond, LiveCodeBench v6, and LongBench v2 show up to 2.64× first-token-latency speedup and +7.15 pass@4 improvement over regular decoding.
UniAudio-Token: Semantic Speech Tokenizer with General Audio Perception for Audio-LLMs
UniAudio-Token is a framework from Tencent that extends semantic speech tokenizers—commonly used as interfaces for Audio-LLMs—to support general audio perception without sacrificing speech quality. It introduces two mechanisms: Semantic-Acoustic Primitives (SAP) for structured supervision decomposing audio into linguistic, vocal, and auditory-scene components, and Semantic-Acoustic Equilibrium (SAE), a content-aware gating mechanism that restores fine-grained acoustic details from shallow layers. Evaluations show it outperforms all single-codebook baseline tokenizers on both understanding and generation tasks when integrated with downstream LLMs. Code, training/inference scripts, and model checkpoints are publicly released.
Byte-Prefix Marginalization enables cross-tokenizer on-policy distillation between heterogeneous LLM families
A new arXiv preprint introduces Byte-Prefix Marginalization (BPM), a technique for distilling knowledge from teacher LLMs into a student model when the two use different tokenizers. BPM re-expresses the teacher's next-token distribution in a shared byte space, preserving probability mass and producing a vocabulary-complete alignment target. Evaluated with Qwen3-32B, GLM-Z1-9B-0414, and MiniMax-M2.7 as teachers, BPM improves six-benchmark average scores by 3.7–6.6 points over the strongest cross-tokenizer baselines on math and programming tasks. The method addresses a practical bottleneck in consolidating complementary open-weight models into compact students.
IAAN: Training-free neuron amplification improves acoustic perception in large audio-language models
Researchers introduce IAAN (Identifying and Amplifying Acoustic Neurons), a training-free, label-free inference-time method that improves non-semantic speech attribute recognition in large audio-language models by identifying and amplifying specific feed-forward neurons in the audio encoder. The method scores neurons by contrasting activations on real audio versus noise references, then amplifies the top-scoring neurons at inference. Across ten non-semantic speech attributes, IAAN yields accuracy gains of 25.7 points on Audio-Flamingo-3, 21.4 on Qwen2.5-Omni, and 9.7 on Kimi-Audio without any retraining. The work establishes that encoder-side, neuron-level intervention is substantially more effective than post-encoder or language-model-side interventions for acoustic perception.
LIHA reveals first-token broadcaster heads as mechanistic source of language identity in transformers
Researchers introduce Language Identity Head Ablation (LIHA), a causal intervention that zeros individual attention heads to measure language-switching behavior across 2,700 prompt-language pairs in seven languages. Applied to GPT-2, LIHA identifies a small set of 'first-token broadcaster' heads that propagate language identity signals throughout generation, with compensatory redistribution following a hierarchical, feedforward pattern. A controlled comparison between Qwen2.5-1.5B-Base and Qwen2.5-1.5B-Instruct provides direct causal evidence that instruction tuning reorganizes language identity circuits toward early-layer localization. The findings offer mechanistic grounding for why multilingual models generate in the wrong language and why this is difficult to correct.
FacialTalker: Facial-expression-aware conversational speech synthesis with DualDPO and VSDD-1K dataset
Researchers introduce FacialTalker, a conversational speech synthesis framework built on a large language model backbone that incorporates facial expression signals via AUTokenizer, a visual tokenizer trained on facial Action Units. A dual direct preference optimization (DualDPO) strategy jointly constrains visual and speech token sequences to improve multimodal alignment. The authors also release VSDD-1K, a 1,033-hour multimodal dialogue dataset of synchronized speaker video and speech collected from real-world internet conversations. Experiments show FacialTalker outperforms baselines on expression perception and speech naturalness.
Parallel Decoding Distillation (PDD) accelerates video and image diffusion inference with improved diversity
Researchers introduce Parallel Decoding Distillation (PDD), a trajectory-based distillation method that accelerates diffusion and flow matching models by predicting multiple denoising steps per network evaluation, achieving state-of-the-art performance at 4-8 NFE. Unlike prior methods relying on variational score distillation and adversarial losses, PDD avoids mode collapse and improves generated video diversity. The method is validated on LTX-2.3 Text-to-Video/Audio, Wan 14B Text-to-Video, and Qwen-Image Text-to-Image models.
Acoustic cue alignment tokens improve speech emotion recognition in audio language models
Researchers study whether instruction-following audio language models (ALMs) use explicit acoustic cues in a grounded way when raw audio is already available. They derive six interpretable acoustic concept tokens from the eGeMAPS feature set and append them to text prompts, testing on FAU-Aibo and IEMOCAP benchmarks. Aligned tokens improve unweighted average recall while shuffled or corrupted tokens degrade performance, but models don't fully collapse under perturbation, indicating partial anchoring to the audio signal. The work offers a practical probing method for interpretability and robustness in affective computing with ALMs.