ragas-81b58911·2 events·first seen Aliases: RAGAS
A preprint from arXiv reports an empirical study comparing RAG evaluation metrics from four libraries—Ragas, DeepEval, RAGChecker, and Opik—against human annotator scores and standard metrics like recall on a business question-answering dataset. The study conducts correlation analysis between automated metrics and human evaluators, finding and documenting limitations of current RAG evaluation methodology. The paper is an English translation of work originally presented at the French-language EvalLLM workshop.
Mistral AI published a technical guide on evaluating Retrieval-Augmented Generation (RAG) systems using the 'LLM as a Judge' paradigm combined with their structured outputs API feature. The approach implements the RAG Triad framework—context relevance, groundedness, and answer relevance—using Pydantic schemas to enforce machine-readable evaluation outputs. Mistral models serve as both the generator and judge components, enabling scalable automated evaluation without human annotators.