4arXiv cs.AI (Artificial Intelligence)·
MMAC: A multi-dimensional benchmark for fine-grained audio captioning evaluation
Researchers introduce MMAC, a benchmark for evaluating audio large language models on captioning tasks, comprising 5,638 audio clips from 20+ sources across 6 capability categories and 15 evaluation dimensions. Unlike prior evaluations focused on generation quality, MMAC assesses both information coverage and description reliability of model-generated captions. The benchmark evaluates representative open-source and proprietary AudioLLMs, revealing clear performance differences across dimensions.
Related guides (2)
Related events (8)
Multi-axis evaluation framework for structured audio captions validated via controlled perturbations
A new arXiv preprint proposes a multi-axis evaluation framework for structured audio captions, addressing limitations of existing flat-text metrics when applied to heterogeneous multimodal outputs. The framework evaluates outputs across five axes—tag-sets, descriptions, logical reasoning, numeric measurements, and spectral profiles—combining LLM judges with deterministic computational metrics. Validation uses a controlled perturbation protocol that injects typed, graded errors into ground-truth annotations to confirm the framework distinguishes paraphrases from genuine corruptions. The work builds on the AudioCards dataset.
MMAE: First comprehensive benchmark for instruction-based audio editing across 7 modalities
Researchers introduce MMAE, a 2,000-sample benchmark for evaluating general-purpose instruction-based audio editing systems, covering 7 audio modalities (sound, speech, music, and mixtures) and 6 levels of task complexity. The benchmark uses a rubric-based evaluation framework decomposing tasks into 17,741 verifiable criteria to assess instruction following and context consistency. Evaluation of leading models reveals severe limitations: Exact Match Rate falls below 5% overall and hits 0% on complex mixed-modality tasks, exposing fundamental gaps in current audio editing systems.
ParaPairAudioBench: Pairwise benchmark reveals large gaps in LALM paralinguistic judgment
Researchers introduce ParaPairAudioBench, a pairwise audio benchmark of 5,175 audio pairs spanning five paralinguistic dimensions (Style, Rate, Emphasis, Age, Gender) designed to evaluate Large Audio-Language Models as judges. Experiments show current LALMs lag human judgment by 32 percentage points on average and exhibit severe calibration failures, especially in ambiguous 'Tie' cases. The benchmark includes same-transcript and cross-transcript conditions to disentangle lexical from acoustic reliance, enabling more rigorous assessment of LALM reliability for speech evaluation.
Evaluating Audio Reasoning with Big Bench Audio
Hugging Face introduces Big Bench Audio, a new benchmark designed to evaluate audio reasoning capabilities in AI models. The benchmark appears to extend the Big Bench evaluation framework into the audio domain, targeting multimodal models that process and reason over audio inputs. This release addresses a gap in evaluation tooling for audio-capable language models.
MM-IssueLoc: Controlled benchmark for multimodal repository-level issue localization with visual evidence
Researchers introduce MM-IssueLoc, a benchmark of 652 issue-PR instances across 23 programming languages designed to evaluate whether AI systems actually use visual evidence (screenshots, error dialogs, UI states) when localizing bugs in code repositories. The benchmark provides file-level and function-level gold labels, paired text-only and with-image evaluation modes, and VCE-based diagnostics that convert images to structured text. Evaluation of current LLM-based and retrieval-based systems shows poor performance (best agent reaches 38.96 file Acc@5), and high scores on text-dominant SWE benchmarks do not transfer to multimodal localization. The work isolates visual evidence as an explicit evaluation variable, exposing a gap in existing software engineering agent benchmarks.
Meta Introduces SAM Audio: Unified Multimodal Model for Audio Separation with PE-AV, Benchmark, and Judge Model
Meta has released SAM Audio, a unified multimodal audio separation model that accepts text, visual, and temporal span prompts to isolate sounds from complex audio mixtures. The system is powered by Perception Encoder Audiovisual (PE-AV), an extension of Meta's open-source Perception Encoder released earlier in 2025, and uses a flow-matching diffusion transformer architecture. Alongside the model, Meta is releasing SAM Audio-Bench (the first in-the-wild audio separation benchmark) and SAM Audio Judge (an automatic evaluation model for audio separation). All components are available today via the Segment Anything Playground.
Audio Interaction Model: Unified Streaming LALM with Always-On Perceive-Decide-Respond Loop
Researchers introduce the Audio Interaction Model framework and a concrete implementation called Audio-Interaction, a unified streaming Large Audio Language Model that handles both offline tasks and real-time audio interaction through a continuous perceive-decide-respond loop. The system is built on SoundFlow, a framework covering data construction, training, and asynchronous low-latency inference. The authors also release StreamAudio-2M, a 2.6M-item streaming corpus spanning 28 sub-tasks, and Proactive-Sound-Bench for evaluating proactive audio intervention. Evaluated across 8 benchmarks, the model preserves competitive offline performance while enabling real-time ASR, streaming instruction following, and proactive response capabilities not available in prior offline LALMs.
ClinMM-Bench: Multi-turn multimodal benchmark for clinical diagnostic reasoning across 1,089 real-world cases
Researchers introduce ClinMM-Bench, a benchmark of 1,089 real-world clinical cases and 3,760 medical images spanning eight specialties, designed to evaluate multimodal LLMs on multi-turn diagnostic reasoning with progressive information disclosure. Fifteen MLLMs were evaluated using a two-level framework assessing both diagnostic accuracy and reasoning quality. Proprietary models led on accuracy but all models showed limited rates of fully correct diagnoses, and error analysis identified five failure modes including visual hallucination and premature closure. The work addresses a gap in clinical AI evaluation by moving beyond single-turn isolated tasks toward more realistic diagnostic workflows.