4arXiv cs.AI (Artificial Intelligence)·
SolarChain-Eval: Physics-constrained benchmark for trustworthy AI agents in decentralized energy markets
Researchers introduce SolarChain-Eval, a benchmark for evaluating autonomous economic agents in decentralized energy markets under physical constraints. The benchmark formulates market governance as a Gymnasium-compatible MDP and assesses policies across market utility, physical safety, slippage, fairness, and auditability dimensions. An LLM-based Planner/Auditor layer is incorporated to constrain and log agent actions; experiments reveal a utility-safety trade-off where RL agents improve market utility but can still produce unsafe behavior, and the LLM layer cannot compensate for a misspecified reward function. Code and data are released open-access.
Related guides (3)
Related events (8)
The Energy Society: Simulation environment for studying LLM agent cooperation under survival pressure
Researchers introduce The Energy Society, a minimal survival-economy simulation where LLM-based agents spend energy proportional to model size when generating tokens and deactivate if energy reaches zero. The environment is used to compare competitive vs. cooperative incentive structures, finding that cooperative incentives lead to donation behaviors and altered job allocation, while larger models consistently spend more energy than they gain. Ablations show that inter-agent action recommendations support coordination and that memory helps agents calibrate risk, while direct sabotage is rare but subtle self-serving behavior emerges in competitive settings.
Tutorial on LLM and agentic AI architectures for smart grid applications with solver-grounded design principle
A new arXiv preprint presents a tutorial on applying LLMs and agentic AI systems to power grid tasks including wind forecasting, EV charging scheduling, power flow analysis, and contingency diagnosis. The paper proposes a 'solver-grounded' design principle where numerical outputs are only reported when they originate from a trusted external solver and pass explicit verification, addressing the problem of LLMs producing physically infeasible outputs. Four case studies compare LLM-only baselines against solver-grounded agents, with EVAgent reducing unmet energy by 7.5-9.5x and GridDebugAgent repairing 17/39 contingency cases while cutting violations by 52.3%. A four-group evaluation framework covering task utility, solver correctness, faithfulness, and cost/latency is also proposed.
SovereignPA-Bench: A benchmark for evaluating user sovereignty in personal AI agents
Researchers introduce SovereignPA-Bench, an executable benchmark designed to evaluate personal AI agents on dimensions beyond task completion, including privacy preservation, consent compliance, evidence grounding, manipulation resistance, and user burden. The benchmark tests 120 sovereignty stress scenarios across 4 model families and 8 policy baselines, generating 3,840 frozen-prompt trajectories with a blinded human audit component. Results show that full-sovereign scaffolding outperforms all baselines on sovereignty metrics while reducing privacy leakage and consent violations. The work argues that personal-agent evaluation must shift from task success toward consent-aware, evidence-grounded action.
Benchmark Agent: Autonomous system for end-to-end benchmark construction
Researchers introduce Benchmark Agent, a fully autonomous agentic system that orchestrates the complete benchmark construction pipeline — from query analysis and subtask design to data annotation and quality control. The system was used to produce 15 benchmarks spanning text understanding, multimodal understanding, and domain-specific reasoning, with evaluation via human judges, LLM-as-a-judge, and consistency checks. The work addresses two persistent problems in the field: the labor intensity of benchmark creation and rapid performance saturation after release. Code and a demo will be publicly released.
Safety Gym: OpenAI Releases RL Safety Constraint Benchmark Suite
OpenAI released Safety Gym, a suite of environments and tools designed to measure progress in training reinforcement learning agents that respect safety constraints during training. The toolkit targets the challenge of constrained RL, where agents must optimize objectives without violating specified safety boundaries. This represents an early formal effort by OpenAI to provide standardized benchmarking infrastructure for safe RL research.
Empirical study finds ERC-8004 decentralized AI agent trust protocol severely compromised by Sybil attacks and design flaws
Researchers conducted the first empirical study of ERC-8004, a permissionless trust protocol for AI agent economies built on Ethereum, BNB Smart Chain, and Base, covering deployment through May 2026. They found that most agent registrations are inactive placeholders (only 3–15% expose live service endpoints), and the reputation registry is functionally broken: values are incommensurable, feedback is rarely grounded in verifiable interactions, and Sybil behavior is rampant (59–91% of reviewers flagged as coordinated). After removing Sybil-flagged feedback, up to 89% of rated agents have no valid reputation signal, rendering the protocol unreliable as a trust basis for autonomous agent transactions. The paper provides protocol-design recommendations and an empirical baseline for future AI agent market research.
CHRONOS: Temporally-Aware Multi-Agent Coordination for Evolving Data Marketplaces
CHRONOS is a three-layer multi-agent architecture addressing temporal degradation in knowledge-graph data marketplaces, combining neural-ODE-based shortcut decay, changepoint-conditioned Shapley pricing, and EXP3-IX-driven differential privacy budget management. The system achieves 0.937 recall@10, 2.74 QPS, and 161ms latency under a total epsilon of 4.25 (delta=1e-6) using zCDP composition across four benchmarks. A key limitation noted is that at this privacy level, released valuations remain noise-dominated, with utility primarily derived from public index routing. The work provides formal guarantees including per-query recall-loss bounds and finite-sample Shapley error bounds under distribution shift.
EvoPolicyGym benchmark evaluates autonomous policy evolution by LLM agents in RL environments
A new arXiv preprint introduces EvoPolicyGym, a benchmark for evaluating how LLM-based agents iteratively improve executable policies in compact interactive RL environments under a fixed interaction budget. The benchmark provides trajectory-level diagnostics beyond aggregate scores, distinguishing how agents allocate budget and convert feedback into parametric tuning. GPT-5.5 achieves the strongest aggregate rank score and top-two performance across all 16 environments. The work targets a gap in agent evaluation where iterative policy refinement is conflated with open-ended software engineering progress.