RAGChecker

Empirical evaluation of RAG metrics across four libraries against human judgments

A preprint from arXiv reports an empirical study comparing RAG evaluation metrics from four libraries—Ragas, DeepEval, RAGChecker, and Opik—against human annotator scores and standard metrics like recall on a business question-answering dataset. The study conducts correlation analysis between automated metrics and human evaluators, finding and documenting limitations of current RAG evaluation methodology. The paper is an English translation of work originally presented at the French-language EvalLLM workshop.