5arXiv cs.CL (Computation and Language)·
DeltaMerge-LowRes: Composing Language and Task Weight Deltas for Low-Resource NLP Adaptation
A new arXiv preprint introduces DeltaMerge-LowRes, a method for adapting multilingual encoders to new languages and tasks in low-resource settings by training language and task deltas separately and composing them in weight space at inference. The paper proposes four composition rules including a novel cross-axis TIES variant that adapts the TIES-Merging algorithm to language/task axes rather than task/task axes. Evaluated across four task families and four African languages (158 cells), cross-axis TIES improves summarization by +4 to +7 chrF and QA F1 by +2.32, while sparsity-aware merging reduces calibration error by 36%. The work advances model merging techniques for low-resource multilingual NLP.
Related guides (2)
Related events (8)
Steerable Model Merging (ST-Merge) improves multilingual reasoning via adaptive gated cross-attention
Researchers propose ST-Merge, a framework for adaptively merging a multilingual model and a reasoning model using a gated cross-attention mechanism that weights each source model's contribution based on input characteristics. The approach addresses the limitation of static one-size-fits-all merging strategies that fail to resolve conflicts between source models. Experiments across 21 languages on four multilingual reasoning benchmarks show consistent improvements over strong baselines.
In-place tokenizer expansion recipe enables multilingual efficiency gains in pre-trained LLMs
Researchers present a method for expanding a pre-trained LLM's tokenizer without retraining from scratch, using continued BPE merges on a multilingual corpus and a two-stage adaptation procedure (embedding-only training followed by full continued pre-training). Applied to LFM2-8B-A1B (a Mixture-of-Experts model) to produce LFM2.5-8B-A1B with a 128K tokenizer, the approach achieves 2.4–2.6× token compression for Hindi and Vietnamese (up to 4× for Thai), translating to an estimated 2.2–3.7× per-character decode speedup on reference devices. The technique is particularly relevant for on-device models where embedding and LM-head matrices constitute a material fraction of decode bandwidth, making vocabulary size a real efficiency constraint.
DIVE: Dynamic In-context Vector Distillation with Decisive-Token Supervision for Long-form Medical Report Generation
DIVE is a frozen-backbone distillation framework that addresses a fundamental limitation in token-level in-context vector distillation: uniform cross-entropy supervision treats all output tokens equally, but long-form outputs like medical reports are dominated by low-information template tokens while diagnostically critical tokens receive insufficient gradient signal. The method introduces decisive-token supervision (upweighting pathology-related tokens and EOS events) and state-conditioned dynamic steering (hidden-state-dependent adapters replacing fixed residuals) to correct supervision imbalance and autoregressive drift. Evaluated on MIMIC-CXR and CheXpert Plus with two medical VLM backbones, DIVE achieves best BLEU-4, ROUGE-L, and RadGraph F1 across all dataset-backbone combinations while remaining competitive on CheXbert F1.
Fine-Tune MMS Adapter Models for Low-Resource ASR
This Hugging Face blog post provides a technical guide for fine-tuning Meta's Massively Multilingual Speech (MMS) adapter models for automatic speech recognition in low-resource languages. It covers the adapter-based fine-tuning approach that allows efficient adaptation of the MMS model to specific languages without full model retraining. The post targets practitioners working on speech recognition for underrepresented languages.
Cross-lingual transfer from Sinhala improves low-resource Dhivehi ASR
A new arXiv paper investigates cross-lingual transfer learning from Sinhala to improve automatic speech recognition for Dhivehi, a severely low-resource language of the Maldives. Across 17 experiments spanning five transfer paradigms, the best system—continual pre-training on Sinhala followed by Dhivehi fine-tuning with KenLM—achieves 12.89% WER, outperforming the Dhivehi-only baseline by 13.50% WER. A Turkish control experiment confirms that linguistic relatedness, not just additional data, drives the improvement.
DelTA: Discriminative Token Credit Assignment for RLVR Training
DelTA introduces a discriminative token credit assignment method for reinforcement learning from verifiable rewards (RLVR) that addresses the problem of high-frequency formatting tokens dominating policy gradient updates. The method estimates per-token coefficients to amplify side-specific gradient directions and downweight shared or weakly discriminative ones, making the effective update direction more contrastive. On seven mathematical benchmarks, DelTA outperforms same-scale baselines by 3.26 and 2.62 average points on Qwen3-8B-Base and Qwen3-14B-Base respectively, with additional gains on code generation tasks.
Instruction tuning plus model merging improves reasoning LMs in verifiable and unverifiable domains
A new arXiv preprint proposes a two-step method for adapting reasoning language models (RLMs) to domains lacking reliable verifiers: first apply standard supervised fine-tuning (without reasoning traces) on the target domain, then merge the resulting model with the original RLM to recover reasoning behavior. The technique is evaluated on both verifiable domains (coding) and hard-to-verify domains (text summarization), showing improvements while preserving general RLM capabilities. Notably, the authors report the full adaptation can be achieved for under $3, making it highly accessible.
CHERRY: Compressed Hierarchical Experts with Recurrent Representational Yield — three compute-efficient LM training techniques
A preprint from arXiv introduces CHERRY, a suite of three complementary techniques for compute-efficient language model training: Selective Ground Truth Token Training (SGT) that concentrates supervision on ~15% of semantically loaded tokens while recovering ~67% of full-sequence loss reduction; depth compression that shrinks a 48-layer 1B-parameter model to 6 layers (227M) via layer averaging and recurrent unrolling, matching a 566M dense model's loss; and a Mixture of Efficient Experts (MoEE) assembly that outperforms individual compressed models at comparable active parameters. The techniques are validated on CHERRY-1.8B, a Korean-language foundation model trained entirely from scratch using these methods. Authors are transparent about scope limitations: one model family, Korean data, and loss-based metrics only.