4arXiv cs.AI (Artificial Intelligence)·
SciFigQual-Bench: A benchmark for scientific figure quality assessment with full-manuscript context
Researchers introduce SciFigQual-Bench, a benchmark for evaluating scientific figures across five dimensions—clarity, layout, caption fit, context relevance, and misleading risk—using 6,308 expert-annotated images from top CS conferences (2020–2025). Each image is bound to its caption, citing sentence, and manuscript context, addressing gaps in existing IQA methods that focus on natural photos or AI-generated content. The authors also propose SFQ-Agent, a staged cross-modal evaluation framework that, when equipped with GPT-5.6-Sol, achieves the lowest average absolute error (0.418) and highest consistency rate (93.4%) on the eval1200 test subset. The work targets automated scientific figure quality auditing, a narrow but underserved evaluation niche.
Related guides (2)
Related events (8)
SciDiagramEdit: Benchmark and agentic framework for editing scientific figures from natural-language instructions
Researchers introduce SciDiagramEdit, a benchmark and skill-evolution framework for automating edits to scientific figures under natural-language instructions. The benchmark mines before/after figure pairs from arXiv version histories, grounding training in authors' own revision intent. An agentic proposer iteratively refines the agent's skill specification from execution traces, progressively improving edit accuracy on held-out validation. The work demonstrates that natural paper revisions are an effective training signal for instruction-driven figure editing.
Benchmark reveals uneven scientific visualization literacy across six MLLMs
A new arXiv preprint benchmarks six multimodal LLMs (three closed-source, three open-source) on a standardized 49-item scientific visualization literacy test spanning 18 visualizations, 8 techniques, and 11 task types, comparing results against 485 human participants. Gemini emerges as the strongest model, exceeding the human mean, while open-source models fall below the human baseline. Performance is highly uneven: models handle scientific illustration and spatial tasks well but struggle with texture-based visualizations, flow-direction interpretation, and quantitative estimation. The authors argue SciVis literacy should be treated as a necessary evaluation dimension for multimodal AI systems.
OpenAI introduces LifeSciBench, a life sciences AI evaluation benchmark
OpenAI has released LifeSciBench, a benchmark designed to evaluate AI systems on real-world life science research tasks and decisions. The benchmark is described as expert-authored and expert-reviewed, targeting domain-specific evaluation in biology and related fields. This addresses a gap in specialized scientific benchmarking for AI systems.
WikiVQABench: A Knowledge-Grounded Visual Question Answering Benchmark from Wikipedia and Wikidata
WikiVQABench is a new human-curated VQA benchmark that requires external knowledge beyond visual perception, constructed by combining Wikipedia images, captions, and Wikidata structured knowledge with LLM-generated question candidates reviewed by human annotators. The benchmark evaluates knowledge-intensive reasoning in vision-language models, covering 15 VLMs ranging from 256M to 90B parameters. Accuracy spans 24.7% to 75.6%, indicating meaningful discrimination across model scales. The dataset and code are publicly released.
Qwen releases Qwen-Image-Bench, a multimodal judge/evaluation model
Qwen has released Qwen-Image-Bench on Hugging Face, an image-text-to-text model tagged as a judge-model for evaluation and benchmarking purposes. The model supports both English and Chinese and appears designed to evaluate text-to-image outputs. With 8,572 downloads and 50 likes shortly after release, it has attracted modest early interest.
MedFailBench: Clinician-built benchmark for medical AI safety boundary failures
MedFailBench is a new open-source benchmark designed to identify which safety boundary a medical AI system failed, rather than measuring correctness. The v0.2.1 release contains 44 clinician-reviewed synthetic cases annotated by severity (1–5) and six safety gate types including missed urgent escalation, unsafe dosing, and evidence fabrication. The benchmark includes a HuggingFace leaderboard preview, a safety gate taxonomy, and an automated pipeline for archiving model-response screening runs, released under Apache-2.0 and CC-BY-4.0.
FigSIM: Dataset for Fine-grained Suicide Severity and Figurative Language in Suicide Memes
FigSIM is the first annotated dataset for analyzing suicide memes, comprising 1,049 memes labeled for suicide severity levels, figurative language phenomena, and suicide-related content categories. The authors benchmark 16 unimodal and multimodal models across three classification tasks. Key findings include systematic underprediction of high-severity cases, particularly for figurative memes, highlighting challenges for automated content moderation. The dataset is publicly released to support future research.
NatureBench: Coding agents surpass published SOTA on only 17.8% of real scientific tasks from Nature-family papers
NatureBench introduces a 90-task benchmark derived from peer-reviewed Nature-family publications to evaluate whether AI coding agents can advance beyond reproduction toward genuine scientific discovery. Built on NatureGym, an automated pipeline that creates containerized per-task environments, the benchmark addresses environment fragmentation that has undermined prior agent-on-research evaluations. Evaluating ten frontier agent configurations under a web-search-disabled protocol, the strongest model exceeds published SOTA on only 17.8% of tasks, with failures driven primarily by wrong method choice and insufficient compute rather than task misunderstanding. Agents succeed mainly through methodological translation—recasting scientific problems as supervised prediction—rather than genuine scientific invention.