5arXiv cs.CL (Computation and Language)·
STRACE framework improves LLM agent optimization via structural trajectory analysis and causal extraction
Researchers introduce STRACE (Structural TRajectory Analysis and Causal Extraction), a framework for constructing high signal-to-noise optimization contexts for long-horizon LLM agents. At the batch level, STRACE mines failure patterns to filter redundant traces; within each trace, it performs causal localization over a textual dependency graph to isolate root-cause steps. On the formal verification benchmark VeruSAGE-Bench, STRACE achieves a 1.4× success-rate improvement (42.5% to 58.5%) over human-expert-designed agents, outperforming standard context-filtering baselines.
Related guides (2)
Related events (8)
TraceLab: Real-world coding agent workload traces reveal serving optimization opportunities
Researchers at UW release TraceLab, a dataset of ~4,300 coding-agent sessions comprising ~350,000 LLM steps and ~430,000 tool calls drawn from real day-to-day use of Claude Code and Codex. Analysis reveals characteristic patterns: long autonomous loops, long contexts with short outputs, heavily-tailed tool call distributions, and high but imperfect prefix cache hit rates. The findings motivate concrete serving-system improvements including lower-overhead tool calling, append-length-aware prefill, and improved KV-cache management. The dataset, pipeline, and analysis code are publicly released.
VeriTrace: Cognitive-Graph Framework with Explicit Regulatory Loops for Deep Research Agents
VeriTrace introduces a cognitive-graph framework for deep research agents that replaces implicit LLM reasoning over intermediate representations with three explicit regulatory loops: interpretive update, deviation feedback, and schema revision. The system addresses contamination and error propagation in evolving mental models during complex multi-step research tasks. Using Qwen3.5-27B backbones, VeriTrace improves over the strongest matched baseline by 4.22 pp on DeepResearch Bench Insight and 5.9 pp Overall win rate on DeepConsult. With Config-DeepSeek, it achieves the strongest reproducible open-source result on DeepResearch Bench.
LongTraceRL: Reinforcement Learning for Long-Context Reasoning via Search Agent Trajectories and Rubric Rewards
LongTraceRL is a new RL training framework for improving long-context reasoning in LLMs, addressing limitations of existing RLVR methods. It constructs challenging training data using multi-hop questions from knowledge graph random walks and tiered distractors derived from search agent trajectories (high-confusability: read but uncited; low-confusability: seen but unopened). A rubric reward provides entity-level process supervision along reasoning chains, applied only to correct responses to prevent reward hacking. Experiments across three LLMs (4B–30B parameters) on five long-context benchmarks show consistent improvements over strong baselines.
CausaLab: Scalable Benchmark for Interactive Causal Discovery by LLM Agents
CausaLab is a new evaluation environment that tests LLM agents on interactive causal discovery tasks, requiring them to recover both causal graphs and structural equations from synthetic laboratory episodes governed by randomly sampled structural causal models (SCMs). The benchmark separates predictive accuracy from genuine causal understanding, revealing a persistent gap: GPT-5.2-high achieves 92% task accuracy in a 6-node observational setting but only 0.471 all-edge F1 for mechanism recovery. Mixed observation-intervention strategies improve structural fidelity, while pure intervention strategies underperform on both metrics. Premature stopping is identified as a key agent weakness, partially mitigated by prompting models to verify hypothesis-data consistency.
Agentic CLEAR: Automating Multi-Level Evaluation of LLM Agents
Agentic CLEAR is an automatic evaluation framework for LLM-based agentic systems that analyzes behavior at three granularity levels: system, trace, and node. Unlike existing tools that rely on static error taxonomies or focus only on observability, it dynamically generates textual insights and integrates above the observability layer with an accessible UI. Experiments across four benchmarks and seven agentic settings demonstrate strong alignment with human-annotated errors and predictive accuracy for task success rates.
RedAct framework protects procedural skills in agent execution traces via selective redaction and watermarking
Researchers introduce RedAct, a framework for releasing agent execution traces without exposing proprietary procedural skills (tool invocations, decision logic, error-recovery strategies). The system localizes sensitive information, rewrites traces while preserving audit-critical evidence, and embeds behavioral watermarks for provenance tracking. To evaluate the approach, the authors construct CapTraceBench, a benchmark of 75 long-horizon tasks and 154 skills across seven domains. RedAct reduces normalized skill transfer from 44.7–67.1% on raw traces to below the no-skill baseline, while watermark detection achieves 93.6–100% true positive rate with under 2% false alarms.
STT-Arena: Benchmark for Adaptive Replanning Under Spatio-Temporal Dynamics in Tool-Using LLMs
STT-Arena is a new benchmark of 227 interactive tasks designed to evaluate LLMs' ability to detect mid-task disruptions and replan under spatio-temporal dynamics, covering nine conflict types and four solvability levels. Evaluation of frontier models including Claude-4.6-Opus shows less than 40% overall accuracy, revealing fundamental limitations in dynamic reasoning. The authors identify three recurring failure modes—Stale-State Execution, Misdiagnosis of Dynamic Triggers, and Missing Post-Adaptation Verification—and propose an iterative trajectory refinement technique combined with online RL to train STT-Agent-4B, a 4B-parameter model that outperforms frontier LLMs on the benchmark.
SHERLOC: Training-free structured fault localization framework boosts code repair agent performance on SWE-Bench
SHERLOC is a training-free localization framework that pairs a reasoning LLM with compact repository tools to produce structured diagnostic context for code repair agents, rather than bare file pointers. It achieves 84.33% accuracy@1 on SWE-Bench Lite and 81.27% recall@1 on SWE-Bench Verified at ~30B parameters, matching or outperforming larger agentic methods. Injecting SHERLOC's diagnostic output into downstream repair agents yields an average +5.95 percentage point resolve rate improvement on SWE-Bench Verified while reducing localization tokens by 36.7% and total tokens by 23.1%. The work addresses a concrete inefficiency in agentic coding pipelines where roughly half the inference budget is spent on fault localization before any editing begins.