technique

instruction embedding models

techniqueactiveinstruction-embedding-models-b2506539·1 events·first seen 25d ago

Aliases: instruction embedding models

Co-occurring entities

MTEB embedding model leaderboard prompt sensitivity

More like this (12)

embedding models Instructional Segment Embeddings sentence embeddings encoder-only language models Sparse Embedding Models static embeddings multi-turn language models Arithmetic Pedagogy for Language Models Transformer Language Models Joint Embedding Predictive Architecture (JEPA)Language Model Finetuning Latent Context Language Models

Recent events (1)

6arXiv · cs.CL·25d ago·source ↗

Instruction Sensitivity Undermines Embedding Model Evaluation: Single-Prompt Benchmarks Are Insufficient

This paper presents an empirical study of prompt sensitivity in instruction-tuned embedding models, covering 6 models, 11 datasets, and 15 task-specific prompts per dataset (990 total evaluations). The authors demonstrate that single-prompt evaluation systematically misrepresents true model performance, with default prompts both understating and overstating capabilities depending on phrasing. A key finding is that leaderboard rankings are not robust: by selecting prompts favorably, any model in the study can be promoted to first place. The authors recommend that benchmarks incorporate prompt robustness metrics, either through multi-prompt evaluation or by reporting sensitivity alongside point estimates.

Evaluation and Benchmarking Agent and Tool Ecosystem MTEB embedding model leaderboard prompt sensitivity +1 more