4arXiv cs.CL (Computation and Language)·
Multi-axis evaluation framework for structured audio captions validated via controlled perturbations
A new arXiv preprint proposes a multi-axis evaluation framework for structured audio captions, addressing limitations of existing flat-text metrics when applied to heterogeneous multimodal outputs. The framework evaluates outputs across five axes—tag-sets, descriptions, logical reasoning, numeric measurements, and spectral profiles—combining LLM judges with deterministic computational metrics. Validation uses a controlled perturbation protocol that injects typed, graded errors into ground-truth annotations to confirm the framework distinguishes paraphrases from genuine corruptions. The work builds on the AudioCards dataset.
Related guides (2)
Related events (8)
MMAC: A multi-dimensional benchmark for fine-grained audio captioning evaluation
Researchers introduce MMAC, a benchmark for evaluating audio large language models on captioning tasks, comprising 5,638 audio clips from 20+ sources across 6 capability categories and 15 evaluation dimensions. Unlike prior evaluations focused on generation quality, MMAC assesses both information coverage and description reliability of model-generated captions. The benchmark evaluates representative open-source and proprietary AudioLLMs, revealing clear performance differences across dimensions.
Audit reveals protocol-level shortcuts in large audio-language model judges for speech evaluation
Researchers audit 'protocol-level shortcuts' in large audio-language models (LALMs) used as automatic judges for speech evaluation, testing across three deployment protocols: feature-blueprint judging, reference-conditioned judging, and pairwise A/B comparison. Across six judges and four attributes, several LALMs are found to rely on shortcuts rather than actual audio content — for example, incorrect specialist labels collapse emotion accuracy to 0.10 or below for five judges, and Qwen3-Omni-Thinking shows position bias in A/B comparisons. The findings indicate that high human-agreement scores can overstate judge validity, and that each model-protocol pair requires its own shortcut probe to be trustworthy.
MMAE: First comprehensive benchmark for instruction-based audio editing across 7 modalities
Researchers introduce MMAE, a 2,000-sample benchmark for evaluating general-purpose instruction-based audio editing systems, covering 7 audio modalities (sound, speech, music, and mixtures) and 6 levels of task complexity. The benchmark uses a rubric-based evaluation framework decomposing tasks into 17,741 verifiable criteria to assess instruction following and context consistency. Evaluation of leading models reveals severe limitations: Exact Match Rate falls below 5% overall and hits 0% on complex mixed-modality tasks, exposing fundamental gaps in current audio editing systems.
ParaPairAudioBench: Pairwise benchmark reveals large gaps in LALM paralinguistic judgment
Researchers introduce ParaPairAudioBench, a pairwise audio benchmark of 5,175 audio pairs spanning five paralinguistic dimensions (Style, Rate, Emphasis, Age, Gender) designed to evaluate Large Audio-Language Models as judges. Experiments show current LALMs lag human judgment by 32 percentage points on average and exhibit severe calibration failures, especially in ambiguous 'Tie' cases. The benchmark includes same-transcript and cross-transcript conditions to disentangle lexical from acoustic reliance, enabling more rigorous assessment of LALM reliability for speech evaluation.
EvalCards: A unified reporting layer for AI evaluation results with interpretive signals
Researchers introduce EvalCards, an operational schema and tooling layer that composes benchmark metadata, evaluation run data, and model metadata into a unified, interpretable record for AI evaluation reporting. The system derives a reporting schema from 52 papers and 10 stakeholder interviews, implements four interpretive signals (reproducibility, documentation completeness, provenance/risk, score comparability), and deploys a monitoring tool across 5,816 models, 635 benchmarks, and 101,843 results. The work targets the widespread inconsistency in how evaluation results are reported across leaderboards, model cards, and company blogs, making cross-source comparison unreliable. It addresses a structural gap in the evaluation ecosystem by providing extraction infrastructure, not just a proposal.
Evaluating Audio Reasoning with Big Bench Audio
Hugging Face introduces Big Bench Audio, a new benchmark designed to evaluate audio reasoning capabilities in AI models. The benchmark appears to extend the Big Bench evaluation framework into the audio domain, targeting multimodal models that process and reason over audio inputs. This release addresses a gap in evaluation tooling for audio-capable language models.
Mitigating Perceptual Judgment Bias in Multimodal LLM-as-a-Judge via Perceptual Perturbation and Reward Modeling
This paper identifies and analyzes 'Perceptual Judgment Bias' in multimodal LLM judges, where models anchor on response text rather than visual evidence when the two conflict. The authors introduce a Perceptually Perturbed Judgment Dataset using counterfactual responses to isolate perceptual errors, and a training framework combining GRPO-based reward modeling with batch-ranking objectives. Experiments on MLLM-as-a-Judge benchmarks show improved perceptual fidelity, ranking coherence, and alignment with human evaluation.
Multimodal Voice Activity Projection framework for turn-taking prediction in social robots
Researchers introduce MM-VAP, a Multimodal Voice Activity Projection framework that extends audio-only turn-taking prediction to synchronized audio-visual inputs for use in social robots acting as conversation mediators. The approach uses pretrained audio-visual backbones adapted via Low-Rank Adaptation, an inter-speaker attention stage, and a semantic consistency loss to regularize the output space. Experiments on the NoXi, NoXi+J, and Haru EDR corpora show improvements over baselines on several turn-taking events. The work targets human-robot interaction scenarios where anticipating conversational dynamics is required rather than simply reacting to pauses.