5arXiv cs.AI (Artificial Intelligence)·
CAM-DF: Cost-aware stopping framework reduces LLM agent tool acquisition by 37% without performance loss
Researchers introduce CAM-DF (cost-aware marginal decision-focused stopping), a method for deciding how many tools an LLM agent should acquire from a ranked list, balancing task coverage against cost, context load, and privacy exposure. The approach trains on the offline gap between stopping now versus continuing, with theoretical proof that score-only rules are suboptimal under heterogeneous costs. Evaluated on 1,343 tasks across five tool-use domains including τ-bench Retail, CAM-DF exposes agents to 37% fewer tools than full access while maintaining comparable task success. The method is a lightweight pre-execution plugin that works with existing tool rankings without fine-tuning the underlying LLM.
Related guides (2)
Related events (8)
E3 framework reduces LLM agent token costs 91% by estimating task complexity before execution
Researchers introduce E3 (Estimate, Execute, Expand), a framework that addresses over-reading behavior in LLM agents by having them estimate task complexity and execute a minimum viable path before expanding scope. On MSE-Bench, a 121-edit deterministic benchmark, E3 matches 100% task success while cutting cost by 85%, tokens by 91%, and file inspections by 92% versus strong baselines. The authors also validate the approach on a live GPT-4o agent editing a real open-source library, graded against an actual pytest suite. The work formalizes the Agent Cognitive Redundancy Ratio (ACRR) and positions task-aware execution as a step toward engineering-grounded AI.
PlanBench-XL: Benchmark for LLM Agent Planning in Large-Scale Tool Ecosystems
Researchers introduce PlanBench-XL, an interactive benchmark of 327 retail tasks spanning 1,665 tools designed to evaluate LLM agents on long-horizon planning under retrieval-limited tool visibility. The benchmark includes a blocking mechanism simulating real-world disruptions such as missing or failing tools, forcing agents to detect and recover from broken execution paths. Experiments on ten leading LLMs reveal severe performance degradation: GPT-5.4 drops from 51.90% accuracy in unblocked settings to 11.36% under the most severe blocking condition, highlighting fragility in adaptive planning for large, imperfect tool environments.
Cost-aware evaluation framework for offensive and defensive security agents reveals distinct scaling regimes
A new arXiv paper introduces a cost-success evaluation lens for language-model security agents, measuring performance at fixed inference and tool-use budgets rather than peak capability alone. The authors evaluate models on offensive Cybench CTF challenges and defensive Splunk BOTS v1 SOC investigation tasks, finding that offensive performance scales with test-time compute while defensive SOC tasks depend more on disciplined tool use than raw reasoning budget. Scaled open-weight models can approach frontier proprietary systems on offensive tasks while remaining cost-competitive, but defensive agents show a different and harder-to-scale capability profile. The paper argues that security-agent benchmarks should incorporate economic efficiency and operational fit as first-class metrics.
Early abort cascade for LLM agents predicts episode failure from hidden activations, saving ~47% inference compute
A new arXiv paper introduces a method to detect doomed LLM agent episodes early by probing internal hidden-state activations, rather than waiting for observable failure. The approach uses a cascade of calibrated per-round gates with recall budgets, guaranteeing that eventually-successful episodes survive at a user-specified rate. On TextCraft with Qwen-2.5-7B and Llama-3.2-3B, the cascade saves 37–47% of inference compute at a 90% recall target, outperforming behavior-only baselines by roughly 2x. The work provides both a practical deployment mechanism and theoretical guidance on sample complexity for certifying high recall targets.
MM-ToolSandBox: Apple benchmark reveals visual precision as primary bottleneck for tool-calling agents
Apple researchers introduce MM-ToolSandBox, a benchmark and evaluation framework for visually grounded tool-calling agents, covering 500+ tools across 16 domains with 258 human-verified scenarios. Evaluating 12 models from 4B open-weight to frontier proprietary systems, even the best model achieves below 50% success rate. A key finding is a planning-to-precision crossover: smaller models fail at task planning while larger models fail at visual information extraction, with 53% of failures in capable models stemming from incorrect image interpretation despite correct workflows. The framework and benchmark are publicly released.
Role-Stratified Conformal Risk Control for LLM Tool Call Safety
A new arXiv preprint introduces role-stratified conformal risk control, a calibration method that sets separate statistical risk thresholds for individual semantic argument roles within LLM tool calls (e.g., email body vs. recipient vs. credential). The approach addresses a gap in existing aggregate-level risk certification, where failures in rare high-risk fields can be masked by benign arguments. Evaluated across AgentDojo and InjecAgent benchmarks with six language models, the method achieves consistent per-role budget compliance under model transfer, attack transfer, detector noise, and adaptive attacks. The authors argue that structured tool calls should be certified at the semantic-role level rather than the whole-action level.
Investing in Performance: Fine-tune small models with LLM insights — a CFM case study
This Hugging Face blog post presents a case study from CFM (Capital Fund Management) on using large language model outputs to guide fine-tuning of smaller, more efficient models for financial applications. The approach leverages LLM-generated signals or labels to train compact models that can be deployed at lower cost and latency. The case study illustrates an enterprise pattern of distilling LLM capabilities into task-specific smaller models for production use.
Benchmark and study of LLM agents in deliberative multi-agent collaboration under partial observability
Researchers introduce a scalable benchmark for evaluating LLM agents on cooperative joint decision-making tasks where agents must exchange information under partial and asymmetric observations to reach a shared decision. A systematic evaluation of representative LLMs finds that state-of-the-art models still struggle with complex deliberative collaboration, failing in either information alignment or downstream reasoning even with external mathematical tools. Diagnostic analysis also reveals that deliberation can enable reflection and error correction, sometimes outperforming centralized baselines, offering a nuanced picture of multi-agent LLM capabilities.