Entity · paper

Auditing Protocol-Level Shortcuts in Large Audio Language Model Judges for Speech Evaluation

paperactiveprovisionalauditing-protocol-level-shortcuts-in-large-audio-language-model-judges-for-speech-evaluation-6e2eab98·1 events·first seen 3h ago

Aliases: Auditing Protocol-Level Shortcuts in Large Audio Language Model Judges for Speech Evaluation

Co-occurring entities

Qwen3-Omni-Thinking

More like this (12)

Acoustic Cue Alignment in Audio Language Models for Speech Emotion Recognition Same Evidence, Different Answer: Auditing Order Sensitivity in Multimodal Large Language Models The Test Oracle Problem in Synthetic LLM-as-Judge Corpora: Disappearance, Distortion and a Validation Protocol Leveraging Audio-LLMs to Filter Speech-to-Speech Training Data Civil Court Simulation with Large Language Models Language-Critique Imitation Learning from Suboptimal Demonstrations Self-supervised Speech Comparison for L2 Phone, Rhythm, and Intonation Scoring Audio-Native Speech Recognition with a Frozen Discrete-Diffusion Language Model SAM Audio Judge Large-Language-Models-as-a-Judge in Theory-Agnostic Adaptive Metric-Alignment for Prototypical Networks in Personality Recognition Bias Benchmark for Question Answering Audio Interaction Model

Recent events (1)

5arXiv · cs.CL·3h ago·source ↗

Audit reveals protocol-level shortcuts in large audio-language model judges for speech evaluation

Researchers audit 'protocol-level shortcuts' in large audio-language models (LALMs) used as automatic judges for speech evaluation, testing across three deployment protocols: feature-blueprint judging, reference-conditioned judging, and pairwise A/B comparison. Across six judges and four attributes, several LALMs are found to rely on shortcuts rather than actual audio content — for example, incorrect specialist labels collapse emotion accuracy to 0.10 or below for five judges, and Qwen3-Omni-Thinking shows position bias in A/B comparisons. The findings indicate that high human-agreement scores can overstate judge validity, and that each model-protocol pair requires its own shortcut probe to be trustworthy.

Evaluation and Benchmarking AI Safety Research Qwen3-Omni-Thinking Auditing Protocol-Level Shortcuts in Large Audio Language Model Judges for Speech Evaluation