CollabSim: CSCW-grounded framework for evaluating collaborative competence in LLM multi-agent systems
Researchers introduce CollabSim, a configurable simulation framework for systematically evaluating collaborative competence in LLM-based multi-agent systems (MAS). The framework draws on Computer-Supported Cooperative Work (CSCW) theory to define collaborative capabilities beyond task outcomes, including common ground establishment, shared task understanding, and misalignment repair. Experiments across four LLMs demonstrate the framework can distinguish model performance patterns and reveal task-dependent effects of agent design choices. The work addresses a gap in MAS evaluation, which has historically focused on individual task-solving rather than coordination quality.
Related guides (2)
Related events (8)
Consilium: When Multiple LLMs Collaborate
Hugging Face introduces Consilium, a framework for multi-LLM collaboration where multiple language models work together on tasks rather than relying on a single model. The approach explores how ensembling or deliberation among diverse LLMs can improve output quality and robustness. This fits into the broader agent-tool ecosystem trend of orchestrating multiple AI models for better results.
Multi-agent LLM framework for Chinese civil court simulation with five-stage trial procedure
Researchers present a multi-agent LLM framework for simulating Chinese civil court proceedings, organized around a five-stage civil trial procedure with memory modules and statute retrieval. The system targets civil litigation specifically, which is more common and harder to simulate than criminal cases due to flexible claims and remedies. Experiments show reliable judgment outputs with particular strengths in liability allocation, and find that memory quality substantially affects downstream simulation quality. Code and dataset are publicly released.
AGENTSERVESIM: Hardware-aware simulator for multi-turn LLM agent serving policies
Researchers introduce AGENTSERVESIM, a simulation framework designed to evaluate serving policies for multi-turn LLM agents without requiring dedicated accelerator hardware. The simulator models program-level execution including turn dependencies, tool-induced gaps, and KV-cache residency across HBM, host DRAM, and CXL memory hierarchies. It reproduces real-system behavior within 6% error on key performance metrics while running on commodity CPUs, enabling cost-effective exploration of scheduling, routing, and cache management policies for agentic workloads.
Agentic CLEAR: Automating Multi-Level Evaluation of LLM Agents
Agentic CLEAR is an automatic evaluation framework for LLM-based agentic systems that analyzes behavior at three granularity levels: system, trace, and node. Unlike existing tools that rely on static error taxonomies or focus only on observability, it dynamically generates textual insights and integrates above the observability layer with an accessible UI. Experiments across four benchmarks and seven agentic settings demonstrate strong alignment with human-annotated errors and predictive accuracy for task success rates.
Agentopia: Long-term multi-agent life simulation framework for training LLMs on social behavior
Researchers introduce Agentopia, a framework for simulating 10 years of social life across 100 LLM-powered agents, enabling study of emergent social behaviors and long-term personal growth dynamics. The system defines a 'life reward' metric mirroring human well-being and uses it to train LLMs via rejection sampling. Training on simulated social experience yields a +15.6% improvement on downstream role-playing benchmarks, suggesting that synthetic social simulation can generalize to real capability gains.
Survey: Agentic Environment Engineering for LLMs — Modeling, Synthesis, Evaluation, and Application
A comprehensive arXiv survey systematically reviews the design and engineering of interactive environments for LLM-based agents, covering the full lifecycle from environment modeling and synthesis to evaluation and application. The paper categorizes environments across eight attributes and eight domains, introduces symbolic and neural synthesis paradigms, and characterizes four pathways for agent-environment co-evolution including memory-centric, orchestration-centric, trajectory-centric, and exploration-centric approaches. It also identifies three paradigms of environment evolution (neural-driven, difficulty-driven, scaling-driven) and proposes future directions such as Environment-as-a-Service and multi-agent environments. This is a reference-organizing contribution for the rapidly growing agent tooling and evaluation space.
Counterfactual context revision framework for auditing LLM-based stance simulation in online discussions
Researchers introduce a counterfactual context revision framework to audit how LLMs simulate individual users' stances in online discussions. By applying controlled text-only and multimodal (meme-based) revisions to conversational contexts, they measure how readily simulated stances shift in response to semantically independent changes. Results show effective and robust stance transitions across both revision types and polarization-preference mechanisms, raising concerns about whether LLM simulations reflect genuine user-specific beliefs or are highly context-sensitive artifacts. The work contributes an evaluation framework and highlights risks of using LLMs to model online opinion dynamics.
AgentSpec: A modular framework for controlled composition and analysis of embodied LLM agent scaffolds
AgentSpec is a new modular specification framework that represents embodied LLM agents as typed compositions of reusable policy components with standardized interfaces across perception, memory, reasoning, reflection, action, and learning modules. The framework enables controlled swapping and recombination of components, instantiated across four benchmarks (DeliveryBench, ALFRED, MiniGrid, RoboTHOR). Key findings include that agent performance is governed by scaffold compatibility and interaction effects rather than isolated module strength, and that RL-trained policies compose best when optimized with deployment-time scaffold structure. Code, baselines, and an interactive playground are publicly released.