BEIR

DREAM: Training dense retrieval embeddings via autoregressive LLM supervision

DREAM is a new method for training dense retrieval embedding models using the autoregressive next-token prediction objective of a frozen LLM, bypassing the need for labeled positive/negative document pairs required by contrastive training. The approach injects retriever-generated query-document similarity scores into selected attention heads of the LLM, allowing prediction loss gradients to flow back to the retriever. Evaluated on BEIR and RTEB benchmarks with 0.5B–3B parameter backbones, DREAM consistently outperforms contrastive baselines across model scales.