Introducing the LiveCodeBench Leaderboard - Holistic and Contamination-Free Evaluation of Code LLMs
Hugging Face introduces a leaderboard based on LiveCodeBench, a benchmark designed for holistic and contamination-free evaluation of code-generating large language models. The benchmark continuously collects new coding problems from competitive programming platforms to prevent data contamination that plagues static benchmarks. It evaluates models across multiple code-related tasks beyond just code generation, aiming to provide a more reliable signal of true model capability.
Related guides (3)
Related events (8)
BigCodeBench: The Next Generation of HumanEval
Hugging Face introduces BigCodeBench, a new code generation benchmark designed to succeed HumanEval by offering more challenging and diverse programming tasks. The benchmark aims to better evaluate LLMs on real-world coding scenarios involving complex function calls and library usage. A leaderboard accompanies the release to track model performance across the community.
Multi-LCB extends LiveCodeBench to twelve programming languages for cross-language code evaluation
Researchers introduce Multi-LCB, a benchmark that extends the widely-used LiveCodeBench (LCB) to twelve programming languages by transforming Python tasks into equivalent tasks in other languages while preserving LCB's contamination controls. The benchmark evaluates 24 LLMs and uncovers Python overfitting, language-specific contamination, and large performance disparities across languages. Multi-LCB is designed to auto-update with future LCB releases, making it a living benchmark for multilingual code generation assessment.
What's going on with the Open LLM Leaderboard?
Hugging Face published a commentary examining anomalies and issues observed in the Open LLM Leaderboard, focusing on MMLU benchmark results. The post investigates potential data contamination, evaluation inconsistencies, and scoring discrepancies across open-weight models. It raises concerns about the reliability of MMLU as a benchmark signal and the integrity of leaderboard rankings.
An Introduction to AI Secure LLM Safety Leaderboard
Hugging Face introduces the DecodingTrust-based LLM Safety Leaderboard, a benchmark framework for evaluating large language models across multiple safety and trustworthiness dimensions. The leaderboard aims to provide standardized, reproducible safety assessments covering areas such as toxicity, stereotype bias, adversarial robustness, and privacy. It offers a public ranking of models to help researchers and practitioners compare safety properties across different LLMs.
Latent Space introduces FrontierCode benchmark for code quality evaluation
Latent Space has announced FrontierCode, a new benchmark targeting code quality assessment rather than simple code generation correctness. The announcement comes from the AINews newsletter, suggesting this is positioned as a community-relevant evaluation tool. The framing around 'slop' implies the benchmark is designed to distinguish genuinely high-quality code outputs from superficially plausible but low-quality generations.
Introducing the Open Leaderboard for Hebrew LLMs
Hugging Face has launched an open leaderboard dedicated to evaluating large language models on Hebrew language tasks. The leaderboard aims to benchmark multilingual and Hebrew-specific models across standardized tasks to track progress in Hebrew NLP. This fills a gap in non-English language evaluation infrastructure.
Introducing the Open Leaderboard for Japanese LLMs
Hugging Face has launched an open leaderboard specifically for evaluating large language models on Japanese language tasks. The leaderboard aims to provide standardized benchmarking for Japanese LLMs, filling a gap in multilingual evaluation infrastructure. This initiative supports the growing ecosystem of Japanese-language AI development and open evaluation practices.
Rethinking LLM Evaluation with 3C3H: AraGen Benchmark and Leaderboard
Hugging Face introduces AraGen, a new Arabic-language LLM benchmark and leaderboard built around the 3C3H evaluation framework (Correctness, Completeness, Conciseness, Helpfulness, Harmlessness, Honesty). The benchmark targets a gap in non-English LLM evaluation, specifically for Arabic, using a structured multi-criteria rubric rather than simple accuracy metrics. The leaderboard is hosted on Hugging Face and aims to provide a more holistic assessment of Arabic generative capabilities across frontier and open-weight models.