4arXiv cs.CL (Computation and Language)·
Benchmarking small open-source LLMs for biomedical ontology generation with MeSH-Rel-4K dataset
Researchers evaluate five small open-source LLMs (up to 9B parameters) on identifying semantic relationships between biomedical concepts, introducing MeSH-Rel-4K, a 4,000-relationship dataset derived from Medical Subject Headings. Three adaptation strategies are compared: standard prompting, Chain-of-Thought prompting, and fine-tuning. Fine-tuning yields a 34.1 percentage point average F1-score improvement, demonstrating that targeted fine-tuning can overcome reasoning limitations of parameter-constrained models for specialized domain tasks.
Related guides (2)
Related events (8)
MetaSyn benchmark reveals critical screening bottleneck in LLM-based meta-analysis pipelines
Researchers introduce MetaSyn, a dataset of 442 expert-curated meta-analyses from Nature Portfolio journals, paired with a 140k-article PubMed retrieval corpus, PI/ECO criteria, verified positives, and hard negatives. Benchmarking twelve pipeline configurations — nine RAG variants and a protocol-driven agent — shows that despite 90.9% retrieval recall at K=200, no system recovers more than 52.7% of ground-truth included studies. The core failure is LLMs' inability to reliably distinguish eligible studies from topically similar but criteria-failing distractors. The paper argues that end-to-end scores obscure where pipelines break down and proposes stage-attributed metrics.
The Open Medical-LLM Leaderboard: Benchmarking Large Language Models in Healthcare
Hugging Face has launched the Open Medical-LLM Leaderboard, a public benchmark for evaluating large language models on healthcare and medical tasks. The leaderboard aggregates performance across multiple medical question-answering datasets to enable standardized comparison of open-weight models in clinical and biomedical domains. This initiative aims to accelerate progress in medical AI by providing transparent, reproducible evaluation infrastructure.
OpenMedReason: Large-scale multimodal medical reasoning corpus with 450K instances for clinical VLM training
Researchers introduce OpenMedReason, a 450K-instance open multimodal medical reasoning corpus with reasoning traces derived from human-authored biomedical literature rather than synthetic chains of thought. The dataset covers diverse medical imaging modalities and is paired with OpenMedReason-Bench, a held-out benchmark evaluating LVLMs on perception, medical knowledge, and rationale axes. Training with OpenMedReason yields a 20% average VQA accuracy improvement over base models and achieves performance within 4.2% of leading comparable-scale medical VLMs. Both the dataset and code are publicly released.
SupraBench: First benchmark for evaluating LLMs on supramolecular chemistry reasoning
Researchers introduce SupraBench, the first benchmark designed to systematically evaluate LLMs on supramolecular chemistry tasks including binding affinity prediction, top-binder selection, solvent identification, and host-guest description. The work also releases SupraPMC, a 16M-token corpus of supramolecular chemistry articles from Europe PMC to support domain adaptation. Evaluation of broad open and proprietary LLMs reveals substantial headroom across all tasks, with domain pretraining improving in-distribution regression but creating format compliance tradeoffs. The benchmark targets a narrow but practically important scientific domain where LLM acceleration could reduce days-long dry-lab verification cycles.
Systematic evaluation of LLM prompt sensitivity in healthcare settings reveals safety risks
Researchers conduct a sensitivity analysis of both general-purpose and medical-specific LLMs using the MedMCQA benchmark, testing robustness to lexical and syntactic prompt perturbations. The study finds that even minor phrasing changes can alter clinical advice, and adversarial prompts can produce dangerous outputs such as incorrect dosages or omitted critical findings. Both general-purpose models (GPT-3.5, Llama 3) and domain-specific models (ClinicalBERT, BioLlama3, BioBERT) exhibit this fragility, with syntactic reordering and misleading contextual cues proving more destabilizing than simple paraphrasing.
Sub-billion parameter SLMs outperform zero-shot GPT-5.4 and Claude Sonnet 4.6 on relation extraction benchmarks
A new arXiv paper demonstrates that small language models (360M–3B parameters) fine-tuned on task-specific data can substantially outperform zero-shot frontier LLMs on relation extraction tasks. The best sub-billion model, Qwen2.5-0.5B fine-tuned on pooled general-domain data, achieves micro-F1 of 0.83 versus 0.69 for GPT-5.4 and 0.66 for Claude Sonnet 4.6 in zero-shot settings. The authors attribute the gains to task adaptation rather than model architecture, with a discriminative RoBERTa baseline also exceeding frontier models, and show that 4-bit quantized models deployable on consumer GPUs can match or beat proprietary API-based systems for this narrow task. The work provides evidence that for well-defined NLP tasks with available training data, compact adapted models offer a practical, private, and hardware-efficient alternative to frontier APIs.
LexNeo-Bench: Probing LLM Knowledge of Lexical Borrowing in Luxembourgish via Knowledge-Graph Prompting
Researchers introduce LexNeo-Bench, a 3,050-instance benchmark for evaluating LLM performance on lexical borrowing classification and neology detection in Luxembourgish, a low-resource contact language. Three multilingual LLMs are tested across 34 prompt configurations; without external context, models perform near chance on borrowing classification (25–35%). Injecting instance-specific subgraphs from a linguistic knowledge graph raises accuracy to 71–81% and largely closes the gap between small and large models, though neology detection remains difficult. The study highlights the value of lexicon-aware, structured prompting for low-resource multilingual evaluation.
Comparative Study: Semantic Metadata vs. Unstructured Web Retrieval for Agentic Data Discovery
This paper evaluates whether LLM-based agents still need structured semantic metadata (e.g., schema.org) for data retrieval, comparing a Baseline Agent searching open-web documents against a Semantic Agent leveraging 90 million schema.org-annotated datasets. Using an LLM-as-a-judge pipeline aligned to FAIR principles, the Semantic Agent achieves 65.7% higher overall precision in retrieving FAIR-compliant datasets, while the Baseline Agent answers 40% more questions but frequently returns prose-heavy or portal landing pages instead of actionable data. The study concludes that structured semantic ecosystems remain essential for reliable, execution-oriented agentic workflows despite LLMs' broad unstructured retrieval capabilities.