5Hugging Face Blog·
Hugging Face introduces Real World VoiceEQ benchmark for human-quality voice AI evaluation
Hugging Face has released Real World VoiceEQ, a new benchmark designed to measure the human-perceived quality of voice AI systems. The benchmark targets a gap in existing evaluation frameworks by focusing on naturalistic, real-world voice interaction quality rather than purely technical metrics. This is relevant to the growing voice AI ecosystem where subjective quality dimensions are difficult to quantify.
Related guides (3)
Related events (8)
Hugging Face introduces FFASR Leaderboard for real-world ASR benchmarking
Hugging Face has launched the FFASR Leaderboard, a new benchmark designed to evaluate automatic speech recognition (ASR) systems under real-world conditions. The leaderboard aims to address gaps in existing ASR evaluations by testing models on more challenging, naturalistic audio. This is a community-facing evaluation resource relevant to practitioners building or comparing speech recognition systems.
A New Framework for Evaluating Voice Agents (EVA)
ServiceNow AI has published a blog post on Hugging Face introducing EVA, a new evaluation framework designed specifically for voice agents. The framework appears to address gaps in existing evaluation methodologies for assessing voice-based AI agent performance. As voice agents become more prevalent in enterprise and consumer settings, standardized evaluation protocols are increasingly important for benchmarking progress.
SpeechEQ benchmark evaluates emotional intelligence in speech-language models across 15 EQ subscales
Researchers introduce SpeechEQ, a benchmark framework for evaluating sociolinguistic and emotional reasoning in Speech-Language Models (SLMs), comprising 2,265 multi-turn dialogues across 15 Emotional Quotient subscales grounded in EQ-i 2.0 theory. The benchmark reveals three systematic failure modes in current multimodal models: over-reliance on text (modality shortcut), alignment-induced safety trap, and contextual amnesia across turns. End-to-end architectures outperform cascaded systems but all evaluated models fall short of genuine emotional awareness. The dataset and demo are publicly released on HuggingFace.
TTS Arena: Benchmarking Text-to-Speech Models in the Wild
Hugging Face introduces TTS Arena, a community-driven evaluation platform for text-to-speech models modeled after the LLM Chatbot Arena approach. Users listen to audio samples from competing TTS systems and vote on quality, generating Elo-based rankings. The platform aims to provide a more ecologically valid benchmark than existing automated metrics, which often fail to capture human perceptual preferences. Initial results surface rankings across open and proprietary TTS models.
Evaluating Audio Reasoning with Big Bench Audio
Hugging Face introduces Big Bench Audio, a new benchmark designed to evaluate audio reasoning capabilities in AI models. The benchmark appears to extend the Big Bench evaluation framework into the audio domain, targeting multimodal models that process and reason over audio inputs. This release addresses a gap in evaluation tooling for audio-capable language models.
Hugging Face speech-to-speech repo for local voice agents gains traction on GitHub
Hugging Face's speech-to-speech repository, which enables building local voice agents using open-source models, is trending on GitHub with 5,180 total stars and 173 new stars today. The project provides a pipeline for end-to-end voice interaction using locally-run open-weights models. Growing interest signals continued demand for self-hosted, privacy-preserving voice agent infrastructure.
Hugging Face Transformers Code Agent Beats GAIA Benchmark
Hugging Face reports that their Transformers-based code agent has achieved a top score on the GAIA benchmark, a challenging evaluation for general AI assistants requiring multi-step reasoning and tool use. The result positions Hugging Face's open agent framework competitively against proprietary systems. The post details the agent architecture and tooling approach used to achieve the result.
Announcing Evaluation on the Hub
Hugging Face announced Evaluation on the Hub, a new feature enabling users to evaluate any model on any dataset directly within the Hugging Face Hub infrastructure. The tool aims to lower the barrier to standardized model evaluation by integrating evaluation workflows into the existing model and dataset hosting platform. This represents an infrastructure step toward more accessible and reproducible benchmarking in the ML community.