4arXiv cs.AI (Artificial Intelligence)·
CE-CM: Bayesian partner capability estimation for task-agnostic ad-hoc teamwork with humans
Researchers introduce CE-CM (Capability Estimation via Contextual Models), an approximate Bayesian method for inferring hidden partner capability vectors in multi-task ad-hoc teamwork settings without population pre-training. The approach frames collaboration as joint planning with decentralized execution under uncertainty, refining beliefs online from a small number of tasks. An extension, CE-CM-Div, accounts for human behavioral diversity by evaluating capability hypotheses against diverse planner rollouts rather than a single optimal trajectory. A human study of 225 trajectories from 15 participants shows CE-CM-Div substantially improves capability estimates, suggesting capability-based modeling is a promising direction for robust human-AI teaming.
Related guides (1)
Related events (8)
Human capital traits, not model benchmarks, predict effective human-AI collaboration in forecasting
A pilot study using Polymarket as an externally resolved benchmark finds that the value of human-AI collaboration in forecasting is highly individual-dependent, with a trimodal distribution: most users either defer to the model or rubber-stamp prior beliefs, while a minority engage in genuine complementary reasoning that matches or beats market accuracy. Collaborative traits—perspective-taking, intellectual humility, and curiosity—predicted who reached the high-performance mode, while raw cognitive ability and model benchmark scores did not. The results challenge the common practice of reporting human-AI collaboration effects as a single average, and a pre-registered replication is in preparation.
MECoBench: Benchmark for Multimodal Agent Collaboration in Embodied Environments
Researchers introduce MECoBench, a benchmark and evaluation platform for assessing multimodal LLM collaboration in visually grounded embodied environments. The benchmark spans diverse real-world tasks, two cooperation structures, and three collaboration modes. Key findings include that collaboration generally improves task completion but depends on balancing gains against coordination complexity, that communication is essential to collaboration benefits, and that collaboration improves robustness under noisy conditions.
Cognitive-structured Multimodal Agent with Episodic Visual Memory for long-horizon dialogue
Researchers propose a Cognitive-structured Multimodal Agent (CMA) that externalizes visual context into an Episodic Visual Memory to avoid token explosion in long multimodal dialogues. The architecture includes a Perceptual Abstraction Engine, Cognitive Retrieval Engine, and Multimodal Executive Controller, trained with reinforcement learning on a synthetically generated multi-turn dataset. An 8B parameter agent achieves 91.4% retrieval accuracy over 20-turn sessions, outperforming 32B baselines by 8.2% while nearly halving inference time. The authors also release CMA-Harness, a tool-augmented deployment integrating persistent memory, web access, and image generation tools with OpenAI-compatible serving.
CoMet: Decomposed uncertainty estimation for multimodal LLMs without repeated sampling
Researchers from Princeton introduce CoMet, a post-hoc uncertainty estimation method for multimodal large language models that decomposes uncertainty into a context-specific term (prompt/task ambiguity) and a multiplicity-specific term (number of plausible answers compatible with the input). A lightweight module is trained to estimate these quantities without requiring autoregressive generation or repeated sampling, making it computationally efficient. Experiments across open-ended multimodal benchmarks, hallucination detection, and visual QA show consistent improvements over existing baselines.
ALMANAC dataset provides action-level mental model annotations for studying human-agent collaboration
Researchers introduce ALMANAC, a dataset of 2,987 collaboration actions drawn from the Map Task dyadic routing paradigm, each annotated with theory-informed mental model labels covering self-reasoning, perceived partner intent, and perceived team goal. The dataset targets a gap in LLM agent training data: current agents are optimized for task completion but lack process-level collaborative competence grounded in mental model alignment. Six LLMs are benchmarked on predicting human next-turn behavior and mental model states. The work provides a resource for evaluating and potentially training agents toward more human-like collaborative reasoning.
Dynamic capability scoping for enterprise AI agents: synthetic dataset and three-source permission architecture
A new arXiv preprint proposes a dynamic least-privilege architecture for enterprise AI agents, replacing static credential sets with a three-source system combining role-based ceilings, task-context classifiers, and policy-derived combination prohibitions. The authors release a synthetic dataset of 600 enterprise task prompts labeled with minimum required permissions across a 15-permission taxonomy, validated with high inter-rater agreement (Cohen's κ = 0.967 post-review). The architecture supports both enforced and observe-only deployment modes, with the latter generating behavioral signals for misalignment research. The work addresses the over-privilege problem in agentic deployments, where persistent broad credentials expand attack surface regardless of model reasoning quality.
UniClawBench: Capability-driven benchmark for proactive agents on real-world tasks
Researchers from HKU MMLab introduce UniClawBench, a benchmark for evaluating proactive AI agents across 400 bilingual real-world tasks organized around five foundational capabilities: Skill Usage, Exploration, Long-Context Reasoning, Multimodal Understanding, and Cross-Platform Coordination. Unlike prior benchmarks, it evaluates agents in live Docker containers with step-by-step checkpoints and a closed-loop multi-turn evaluation strategy using executor, supervisor, and user agents. The benchmark disentangles base model capabilities from agent framework design choices by testing state-of-the-art models across multiple frameworks. Code and benchmark are publicly released.
SPyCE: Skill-Policy Co-evolution framework for multimodal agents via hierarchical skill libraries
Researchers introduce SPyCE (Skill-Policy Co-evolution), a training framework for multimodal agents that distills reasoning trajectories into a hierarchical skill library co-evolving with the policy during reinforcement learning. The library separates execution skills (local visual operations) from workflow skills (high-level tool orchestration priors), creating a closed loop where better policies yield better skills and vice versa. SPyCE outperforms both RL-based and memory-based baselines across eight benchmarks, suggesting joint skill-policy optimization as a viable paradigm for capable multimodal agents.