4arXiv cs.CL (Computation and Language)·
Question-type-aware LLM pipeline with multi-agent collaboration achieves top factoid performance on BioASQ 14b
Researchers present a question-type-specific LLM framework for the BioASQ 14b biomedical QA challenge, applying distinct inference strategies to yes/no, factoid, and list questions rather than a single unified approach. For list questions, a multi-agent architecture handles evidence extraction, candidate generation, verification, and aggregation collaboratively. The framework achieved first place in the factoid subtask of Batch 4 in the official BioASQ 14b evaluation, demonstrating competitive performance across multiple batches.
Related guides (2)
Related events (8)
Task-specific two-agent architecture wins QANTA 2026 multimodal QA challenge
Researchers present the top-scoring submission to the QANTA 2026 shared challenge at ICML 2026's EMM-QA Workshop, achieving an overall leaderboard score of 0.402 on multimodal quizbowl tasks. The system uses a two-agent architecture: a GPT-4.1-mini-based Tossup agent with confidence calibration and a GPT-4.1-based Bonus agent with structured relational and multimodal reasoning. Notably, the approach avoids retrieval pipelines and model ensembles, relying instead on lightweight task-specific reasoning policies under efficiency constraints. Results suggest that targeted reasoning strategies can be competitive on resource-constrained multimodal QA benchmarks.
BINEVAL: Binary question decomposition for interpretable LLM evaluation and prompt optimization
Researchers introduce BINEVAL, a framework that decomposes LLM evaluation criteria into atomic binary yes/no questions, aggregating answers into multi-dimensional interpretable scores. The approach matches or outperforms baselines including UniEval and G-Eval on SummEval, Topical-Chat, and QAGS benchmarks, with particular strength on factual consistency. Beyond evaluation, the binary question feedback is shown to support iterative prompt optimization in both self-update and cross-model settings on IFBench. The framework is training-free and task-agnostic, addressing opacity and ceiling-effect problems common in holistic LLM judges.
Co-LMLM: Continuous-query limited memory language models outperform vanilla LLMs on factual tasks at small scale
Researchers introduce CO-LMLM, a limited memory language model that externalizes factual knowledge to a knowledge base during pretraining and retrieves it at inference via continuous vector queries paired with human-readable text values. The approach removes prior restrictions to relational knowledge bases and Wikipedia-only data by introducing an annotation pipeline for arbitrary text. At 360M parameters, CO-LMLM achieves lower perplexity than models trained on 40x more data and SimpleQA factual performance comparable to GPT-4o mini and above Claude Sonnet 4.5, suggesting significant efficiency gains for factual grounding.
ABC-Bench: Agentic biosecurity benchmark finds LLM agents surpass median expert humans on dual-use biology tasks
Researchers introduce ABC-Bench, a benchmark evaluating LLM agents on biosecurity-relevant biology tasks including liquid-handling robot programming, DNA fragment design, and evasion of DNA synthesis screening. All tested agents outperformed the median expert human baseline across all three tasks. Wet-lab validation confirmed that OpenAI's o4-mini-high produced scripts that successfully assembled DNA on an OpenTrons robot. The results highlight a meaningful shift in the biosecurity risk landscape as AI agents acquire practical wet-lab-adjacent capabilities.
Benchmark and study of LLM agents in deliberative multi-agent collaboration under partial observability
Researchers introduce a scalable benchmark for evaluating LLM agents on cooperative joint decision-making tasks where agents must exchange information under partial and asymmetric observations to reach a shared decision. A systematic evaluation of representative LLMs finds that state-of-the-art models still struggle with complex deliberative collaboration, failing in either information alignment or downstream reasoning even with external mathematical tools. Diagnostic analysis also reveals that deliberation can enable reflection and error correction, sometimes outperforming centralized baselines, offering a nuanced picture of multi-agent LLM capabilities.
FACTS Benchmark Suite: Systematically evaluating the factuality of large language models
DeepMind has released the FACTS Benchmark Suite, a systematic evaluation framework for measuring the factuality of large language models. The benchmark is designed to assess how accurately LLMs produce factually grounded outputs. This represents a structured contribution to the growing field of LLM evaluation, specifically targeting hallucination and factual reliability. The announcement comes from a Tier 1 lab, lending it credibility as a reference benchmark in the field.
Generalizing an LLM from 8k to 1M Context using Qwen-Agent
Alibaba's Qwen team describes an agent built on Qwen2 (8k native context) that processes documents up to 1M tokens by decomposing retrieval and reasoning tasks, reportedly outperforming both RAG pipelines and native long-context models. The agent framework was also used to generate synthetic training data for fine-tuning new long-context Qwen models, creating a self-improvement loop. This positions agent-based context extension as a practical alternative to architectural long-context training.
NuclearQAv2: A benchmark for evaluating LLM competence in nuclear engineering
Researchers introduce NuclearQAv2, a ~1,240 question benchmark for assessing LLM performance on nuclear engineering knowledge across boolean, numeric, and verbal question types. The benchmark is constructed via a hybrid pipeline combining expert-authored questions, existing datasets, and LLM-assisted generation from domain-specific corpora. Evaluation of multiple LLMs reveals strong performance on factual recall but significant gaps in quantitative reasoning and conceptual understanding, highlighting the need for multi-faceted domain-specific evaluation.