SPEARBench

SPEARBench: Benchmark for naturalness evaluation in streaming speech-to-speech language models

Researchers introduce SPEARBench, a benchmark for evaluating conversational naturalness in streaming speech-to-speech language models, covering dimensions such as response latency, turn-taking, prosody, dialect consistency, emotional adaptation, and interpersonal stance. The benchmark constructs controlled dialogue prompts from the Seamless Interaction corpus and evaluates multiple contemporary models against human reference answers. Results reveal that current models achieve high signal-level quality and low ASR error rates but still diverge from human conversational behavior on latency, overlap, dialect preservation, and stance dynamics. The work addresses a gap in existing speech and text benchmarks that fail to capture conversational naturalness.