6arXiv cs.CL (Computation and Language)·
DenseOn and LateOn: Fully open dense and late-interaction retrieval models for multilingual and long-context search
Researchers release DenseOn and LateOn, two 149M-parameter retrieval models trained on a fully open, reproducible pipeline using 665M curated English contrastive pairs and 1.88M supervised fine-tuning pairs. Both models achieve new state-of-the-art results for their size class on BEIR (56.20 and 57.22 nDCG@10 respectively). Multilingual variants (mDenseOn, mLateOn) are trained via translate-train on 2.8B pairs, revealing that late-interaction (ColBERT-style) token-level matching generalizes better to unseen languages than dense single-vector models. All models, datasets, and training code are publicly released.
Related guides (2)
Related events (8)
DREAM: Training dense retrieval embeddings via autoregressive LLM supervision
DREAM is a new method for training dense retrieval embedding models using the autoregressive next-token prediction objective of a frozen LLM, bypassing the need for labeled positive/negative document pairs required by contrastive training. The approach injects retriever-generated query-document similarity scores into selected attention heads of the LLM, allowing prediction loss gradients to flow back to the retriever. Evaluated on BEIR and RTEB benchmarks with 0.5B–3B parameter backbones, DREAM consistently outperforms contrastive baselines across model scales.
Embedding interpolation study reveals structured benefits of mixed-language queries in multilingual dense retrieval
A ratio-controlled study on mMARCO evaluates how mixing proportions of parallel query translations via embedding-level interpolation affect multilingual dense retrieval performance. Using BGE-M3, the authors find that an optimal mixing ratio outperforms the best monolingual endpoint in 88 of 105 cases, with a clear asymmetry driven by English dominance. Mixing is uniformly beneficial for non-English document indices, while English-containing indices are best served by pure English queries, and mixing gains correlate negatively with typological distance when controlling for English dominance.
BODHI: Contrastive embedding training for causal discovery in Large Behavioural Models
Researchers identify a critical failure mode in biomedical language model embeddings: off-the-shelf encoders (BioBERT, PubMedBERT, BioM-ELECTRA) assign high cosine similarity (0.76–0.92) to causally unrelated cross-domain pairs, achieving 0% accuracy on cross-domain discrimination. The paper introduces BODHI, a contrastive training approach using hard negatives mined from a biomedical knowledge graph, which improves within-vs-across-domain separation from 1.05x to 2.30x and raises discrimination gap by +0.392. The work targets Large Behavioural Models (LBMs)—foundation models that reason over personal life graphs—where false embedding proximity directly produces false causal edges. Additional contributions include an OpenVINO inference optimization achieving 133x latency reduction (1367ms to 10ms) on Intel AMX hardware, plus a counterintuitive finding that FP16 outperforms INT8 on this silicon.
Cross-lingual relation extraction for Romanian: QLoRA fine-tuning narrows gap but small encoders remain competitive
Researchers evaluate cross-lingual relation extraction for Romanian by translating the SemEval-2010 Task 8 benchmark and testing Gemma 4 31B under zero-shot, few-shot, and QLoRA fine-tuned settings against encoder baselines (XLM-RoBERTa, Romanian BERT, RoBERTa-large). QLoRA fine-tuning improves macro F1 by over 22 percentage points and reduces the English-Romanian cross-lingual gap from 3.3 to 1.4 pp, but encoder baselines of 125M–560M parameters come within 1–4 pp of the fine-tuned 31B model. The study concludes that large LLMs offer limited advantage over compact encoders for single-task relation extraction in compute-constrained deployment scenarios, and releases the translated dataset, code, and models.
Latent Context Language Models (LCLMs) achieve competitive encoder-decoder KV cache compression at scale
Researchers introduce Latent Context Language Models (LCLMs), a family of encoder-decoder compressors that map long token sequences to shorter latent embeddings consumed by a decoder, targeting the KV cache memory bottleneck in long-context inference. The authors conduct architecture search and continually pre-train 0.6B-encoder/4B-decoder models on over 350B tokens at compression ratios of 1:4, 1:8, and 1:16. LCLMs improve the Pareto frontier across general-task performance, compression speed, and peak memory, and are demonstrated as efficient backbones for long-horizon agents that can skim compressed context and expand relevant segments on demand. The work closes a previously noted gap between encoder-decoder approaches and KV cache compression methods on the accuracy-efficiency frontier.
Translation-based fine-tuning of English BERT compared to native-language models across six NLP tasks and five languages
A new arXiv preprint systematically compares translation-based fine-tuning of English BERT against native-language BERT models across six NLP tasks using datasets from Bulgarian, Chinese, Dutch, Italian, and Russian. The translation approach was comparable or superior in 53.3% of cases, with strongest gains in Question Answering, POS Tagging, and NLI, but weaker performance on Named Entity Recognition and Hate Speech Detection. Results suggest translation-based fine-tuning is most effective for syntactic tasks and typologically English-close languages like Dutch, offering a resource-efficient path for low-resource NLP.
Loong: A Human-Like Long Document Translation Agent with Observe-and-Act Adaptive Context Selection
Loong is a long document translation agent that uses a 3E memory module (Essence-Exemplar-Entity) to store structured historical context, replacing passive full-context attention with RL-optimized adaptive context selection. The agent learns its context retrieval policy via reinforcement learning on self-sampled reasoning trajectories. Evaluations show average gains of up to 13.0 points across three metrics in English↔Chinese, German, and French translation directions, with strong generalization and robustness to noise in ultra-long documents.
Towards Reliable Multilingual LLMs-as-a-Judge: An Empirical Study
This paper systematically investigates strategies for extending LLM-based automatic evaluation (LLMs-as-a-Judge) to multilingual settings, covering high-, mid-, and low-resource languages (English, Spanish, Basque). The authors compare instruction translation, monolingual vs. multilingual supervision, and model size, finding that fine-tuned smaller models can match proprietary models when in-domain data is available, while zero-shot larger models are preferable out-of-domain. Two meta-evaluation datasets are extended to Spanish and Basque, and all data and code are publicly released.