4arXiv cs.AI (Artificial Intelligence)·
Open-weight LLMs evaluated for generating structured threat intelligence (STIX) for autonomous vehicle vulnerabilities
Researchers introduce CAV-STIXGen, a dataset mapping connected and autonomous vehicle (CAV) CVE descriptions to STIX threat intelligence objects, CWE entries, and MITRE ATT&CK techniques. They benchmark 11 open-weight LLMs (4B–120B parameters) across prompting strategies, achieving F1 scores up to 0.94 for STIX domain objects and 0.99 for CWE mapping, while MITRE ATT&CK mapping remains harder. A multi-agent configuration using Gemma-4-31B and Codestral-22B is also evaluated, and co-occurrence analysis of CWE/ATT&CK patterns is used to surface recurring threat patterns in the CAV domain.
Related guides (2)
Related events (8)
CWE-Trace framework reveals LLM vulnerability detection is calibration without comprehension
Researchers introduce CWE-Trace, a benchmark of 834 manually curated Linux kernel samples across 74 CWEs with strict temporal splits to prevent data contamination, used to evaluate 8 vanilla LLMs and 15 LoRA fine-tuned variants on vulnerability detection. Key findings: data contamination provides no measurable advantage (84% of nominally contaminated samples carry no usable memorization signal), and backbone directional priors dominate fine-tuning — models exhibit stable systematic failure modes that resist correction. The best binary detection score reaches only 52.1% (barely above chance) and exact CWE classification Top-1 accuracy stays below 1.3%, indicating fine-tuning shifts output distributions without instilling genuine security reasoning. The work introduces two diagnostic metrics (Directional Failure Index and Hierarchical Distance and Direction) and concludes that detection capability and security understanding are decoupled in current LLMs.
Multi-source cybersecurity log dataset with ATT&CK labels and SLM fine-tuning evaluation
Researchers introduce a new multi-source cybersecurity log dataset of 870 sessions (~2.3M events) capturing system, network, and browser activity on Windows endpoints, with per-entry MITRE ATT&CK technique labels across 12 tactics and 53 techniques. The dataset addresses gaps in existing public datasets (CICIDS, UNSW-NB15, ATLAS) that lack combined multi-source coverage with fine-grained ATT&CK labeling. Three small language models (Qwen2.5-1.5B, Llama-3.2-3B, Phi-4-Mini) were fine-tuned with LoRA on the dataset, achieving chunk classification accuracy of 90–97% versus ~8% for base variants, though ATT&CK technique identification remained harder at 42% exact-match accuracy.
VEXAIoT: Multi-agent LLM framework achieves 95% success rate in autonomous IoT vulnerability exploitation
Researchers introduce VEXAIoT, an autonomous multi-agent framework using LLM-based reasoning to discover and exploit vulnerabilities in IoT environments. The system pairs a vulnerability detection agent with an attack execution agent, evaluated across 260 attack executions in IoTGoat and Metasploitable2 environments covering ten OWASP IoT vulnerability categories. It achieves a 95% overall success rate with average execution times under two minutes, demonstrating that LLM agents can automate offensive IoT security workflows at scale in controlled settings.
Practitioner spends $1,500 testing LLM offensive security capabilities against a purpose-built vulnerable app
A developer built a deliberately vulnerable application and ran LLMs against it as automated penetration testers, spending $1,500 on API costs across the experiment. The post evaluates how well current LLMs can identify and exploit real vulnerabilities in a controlled setting. Results provide practical signal on the current state of LLM-assisted offensive security, a capability area with both red-team and safety implications.
CyberSecEval 2 - A Comprehensive Evaluation Framework for Cybersecurity Risks and Capabilities of Large Language Models
CyberSecEval 2 is a benchmark framework designed to evaluate both the cybersecurity risks and capabilities of large language models. The framework appears to be hosted or featured on Hugging Face's leaderboard infrastructure, extending prior cybersecurity evaluation work. It assesses LLMs across multiple dimensions of security-relevant behavior, including potential for misuse and defensive capabilities.
AdversaBench: Automated LLM red-teaming pipeline with multi-judge confirmation and cross-model transferability
AdversaBench is a new end-to-end red-teaming pipeline that mutates seed prompts using five structured operators and confirms failures via a three-judge panel with a meta-judge tiebreaker. Experiments on 45 seeds across reasoning, instruction-following, and tool-use categories produced confirmed failures for every seed. Key findings include sharp variation in operator effectiveness by category, misleading binary failure rates, judge agreement metrics distorted by label skew, and zero-shot transferability of adversarial prompts from Llama 3.1 8B to Llama 3.3 70B. Code and dataset are publicly released.
SearchGEO framework measures LLM search agent vulnerability to web content manipulation
Researchers introduce SearchGEO, a controlled evaluation framework for measuring endorsement corruption in LLM-based web-search agents, combining a manipulation pipeline, five-mode attack taxonomy, and multiple output metrics. Evaluating 13 LLM backends on 308 cases each, they find attack success rates ranging from 0.0% on Claude-Sonnet-4.6 to 31.4% on Gemini-3-Flash, with model-family-specific vulnerability patterns. An auxiliary probe escalating endorsement to install commands reveals a behavioral split: Claude over-rejects while GPT over-trusts. The findings argue for treating adversarial search content robustness as a first-class safety evaluation dimension for deployed agents.
Strix: open-source AI-powered vulnerability detection and remediation tool
Strix is an open-source Python tool that uses AI to automatically find and fix security vulnerabilities in applications. The project has accumulated 26,502 GitHub stars with 88 added today, indicating significant community traction. It sits at the intersection of AI-assisted security tooling and automated code remediation.