5arXiv cs.CL (Computation and Language)·
3D-Fit benchmark evaluates LLMs on spatially-constrained drug molecule generation
A new arXiv preprint introduces 3D-Fit, a token-efficient benchmarking strategy for evaluating general-purpose LLMs on structure-based drug design (SBDD) tasks requiring 3D spatial reasoning. The study compares LLMs against specialized diffusion model baselines on pocket-conditioned ligand generation with constraints including anchor fragments, pharmacophore points, and mandatory pocket-ligand interactions. Results show LLMs still lag behind state-of-the-art diffusion models but demonstrate promising ability to handle multiple simultaneous spatial constraints, suggesting a viable path for LLM-based molecular design.
Related guides (1)
Related events (8)
SupraBench: First benchmark for evaluating LLMs on supramolecular chemistry reasoning
Researchers introduce SupraBench, the first benchmark designed to systematically evaluate LLMs on supramolecular chemistry tasks including binding affinity prediction, top-binder selection, solvent identification, and host-guest description. The work also releases SupraPMC, a 16M-token corpus of supramolecular chemistry articles from Europe PMC to support domain adaptation. Evaluation of broad open and proprietary LLMs reveals substantial headroom across all tasks, with domain pretraining improving in-distribution regression but creating format compliance tradeoffs. The benchmark targets a narrow but practically important scientific domain where LLM acceleration could reduce days-long dry-lab verification cycles.
Genesis Molecular AI: Diffusion models for drug discovery, with Llama lead Sergey Edunov and PEARL's zero-shot OpenBind win
Latent Space interviews Evan Feinberg and Sergey Edunov (formerly Meta's Llama lead) about Genesis Molecular AI, a startup applying diffusion models to drug discovery. The conversation covers PEARL's zero-shot performance on the OpenBind benchmark and the broader implications of co-folding models crossing accuracy thresholds for molecular design. The piece argues that the most interesting diffusion research is happening in scientific domains rather than language modeling.
Benchmarking study finds LLMs fail at counterintuitive probability problems despite strong standard performance
A new arXiv paper evaluates 8 state-of-the-art LLMs on discrete probability problems using two datasets: standard exercises (average accuracy 0.96) and counterintuitive exercises designed to trigger heuristic reasoning (average accuracy 0.59). The authors document token bias causing 20%+ performance drops when canonical problem formulations are disguised, and up to 34% degradation when misleading suggestions are embedded in prompts. The findings argue that current LLMs are not genuine probabilistic reasoners despite their success on advanced math benchmarks.
MissionBench: Zero-shot mission-level evaluation of MLLMs as aerial embodied agents
Researchers introduce MissionBench, a benchmark comprising 120 missions across five simulated 3D aerial environments and four task families, designed to evaluate multimodal LLMs as zero-shot embodied agents. Across 22 open- and closed-source MLLMs, the best model achieves under 35% success versus 84.4% human performance, revealing a large capability gap in long-horizon embodied tasks. The study finds scaling yields gains but that mission-level competence requires coordinating spatial perception, multi-step planning, and adaptive reasoning beyond what current models reliably provide.
PhysTool-Bench reveals severe gaps in MLLM physical tool use and embodied planning
Researchers introduce PhysTool-Bench, the first benchmark evaluating multimodal LLMs on physical tool use across 2,510 queries and 2,678 real-world tools spanning manufacturing, electrical work, agriculture, and healthcare. Evaluation of 13 leading MLLMs shows even the best model (Gemini-3.1-Pro) identifies only 58.7% of tools in a scene and completes just 21.0% of queries end-to-end. The results expose a two-level deficit: poor tool perception in realistic scenes and a much larger drop at the planning stage, indicating a lack of functional commonsense for mapping tools to task semantics. This pinpoints a critical bottleneck for embodied AI development.
Paper challenges LLM expert-level claims by measuring variance and error magnitude in code-based data analysis tasks
A new arXiv paper argues that standard LLM benchmarks overstate model capabilities by focusing on average performance on training-data-adjacent tasks while ignoring response variance and error magnitude. The authors introduce a novel benchmark requiring frontier LLMs to write code for data analysis tasks, comparing results against human expert submissions. Human experts outperformed the frontier LLM on average across multiple metrics and showed lower performance variability. The findings challenge the prevailing narrative that LLMs perform at human-expert level on knowledge economy tasks.
Benchmark reveals uneven scientific visualization literacy across six MLLMs
A new arXiv preprint benchmarks six multimodal LLMs (three closed-source, three open-source) on a standardized 49-item scientific visualization literacy test spanning 18 visualizations, 8 techniques, and 11 task types, comparing results against 485 human participants. Gemini emerges as the strongest model, exceeding the human mean, while open-source models fall below the human baseline. Performance is highly uneven: models handle scientific illustration and spatial tasks well but struggle with texture-based visualizations, flow-direction interpretation, and quantitative estimation. The authors argue SciVis literacy should be treated as a necessary evaluation dimension for multimodal AI systems.
PLAID: Repurposing Protein Folding Models for Multimodal Protein Generation with Latent Diffusion
PLAID is a generative model that simultaneously produces protein 1D sequences and 3D all-atom structures by learning a diffusion model over the latent space of ESMFold, a protein folding model. It requires only sequence data for training—leveraging databases 2-4 orders of magnitude larger than structure databases—and decodes structure at inference via frozen folding model weights. The approach supports compositional prompting by function and organism, addressing practical drug-design constraints like humanization and solubility. A companion compression model, CHEAP, addresses the high-dimensionality of transformer latent spaces to make the diffusion training tractable.