Maat: ReAct-Based Agentic Legal Research Assistant for Competition Law
Maat is a ReAct agent designed specifically for competition law research, orchestrating tools for RAG-based retrieval, web search fallback, and citation generation. Built iteratively with domain experts, it addresses hallucination and citation gaps found in general assistants (Claude, ChatGPT) and legal-specific models (SaulLM-7B, LegalGPT). Maat significantly outperforms baselines on case-specific tasks and matches top baselines on theoretical questions. The evaluation dataset is publicly released on GitHub.
Related guides (4)
Related events (8)
AARRI-Bench evaluates frontier LLMs and agents on granular research-intern-level tasks
Researchers introduce AARR (Act As a Real Researcher), a new benchmark series targeting whether AI agents can emulate the professionalism, thoroughness, and nuanced judgment of human researchers in granular research scenarios—not just macro-level task execution. The first benchmark, AARRI-Bench, tests frontier models and agentic harnesses, finding that even the best configuration (Mini-SWE-Agent with Claude Opus 4.7) achieves only 68.3% success, frequently missing subtle but critical details obvious to human researchers. The work argues that closing the gap requires deeper modeling of research behavior rather than more complex scaffolding.
ARIS: Lightweight autonomous ML research agent using Markdown-only skills
ARIS (Auto-Research-In-Sleep) is an open-source Python project providing lightweight, framework-free Markdown-based skills for autonomous ML research workflows, including cross-model review loops, idea discovery, and experiment automation. It is designed to work with any LLM agent backend including Claude Code, Codex, or others. The project has accumulated 11,791 GitHub stars with notable daily traction (+106), suggesting meaningful community adoption.
Agentic CLEAR: Automating Multi-Level Evaluation of LLM Agents
Agentic CLEAR is an automatic evaluation framework for LLM-based agentic systems that analyzes behavior at three granularity levels: system, trace, and node. Unlike existing tools that rely on static error taxonomies or focus only on observability, it dynamically generates textual insights and integrates above the observability layer with an accessible UI. Experiments across four benchmarks and seven agentic settings demonstrate strong alignment with human-annotated errors and predictive accuracy for task success rates.
ReproRepo: Scalable LLM agent framework for reproducibility auditing using GitHub issues
ReproRepo is a new framework for evaluating LLM agents on reproducibility auditing of ML research, using naturally occurring GitHub issues as supervision signals rather than costly manual curation. The framework is instantiated on 1,149 recent ML papers from major conferences and benchmarks four frontier model-agent configurations. The best-performing agent (Codex with GPT-5.5) surfaces at least one semantically related human-reported reproduction blocker for ~90% of papers, though exact localization of issues remains a weakness. The work provides a reusable, scalable evaluation harness for this underexplored agentic task.
Mistral AI: Using LLM-as-a-Judge with Structured Outputs for RAG Evaluation
Mistral AI published a technical guide on evaluating Retrieval-Augmented Generation (RAG) systems using the 'LLM as a Judge' paradigm combined with their structured outputs API feature. The approach implements the RAG Triad framework—context relevance, groundedness, and answer relevance—using Pydantic schemas to enforce machine-readable evaluation outputs. Mistral models serve as both the generator and judge components, enabling scalable automated evaluation without human annotators.
ALMANAC dataset provides action-level mental model annotations for studying human-agent collaboration
Researchers introduce ALMANAC, a dataset of 2,987 collaboration actions drawn from the Map Task dyadic routing paradigm, each annotated with theory-informed mental model labels covering self-reasoning, perceived partner intent, and perceived team goal. The dataset targets a gap in LLM agent training data: current agents are optimized for task completion but lack process-level collaborative competence grounded in mental model alignment. Six LLMs are benchmarked on predicting human next-turn behavior and mental model states. The work provides a resource for evaluating and potentially training agents toward more human-like collaborative reasoning.
Mistral AI Releases Devstral: Apache 2.0 Agentic Coding Model with SWE-Bench SOTA
Mistral AI, in collaboration with All Hands AI, releases Devstral, an agentic LLM specialized for software engineering tasks under the Apache 2.0 license. The model achieves 46.8% on SWE-Bench Verified, surpassing prior open-source state-of-the-art by over 6 percentage points and outperforming larger models like DeepSeek-V3-0324 (671B) and Qwen3 232B-A22B under the same OpenHands scaffold. Devstral is small enough to run on a single RTX 4090 or a Mac with 32GB RAM, and is available via Mistral's API at $0.1/M input tokens, as well as on HuggingFace, Ollama, and other platforms. Mistral indicates a larger agentic coding model is in development.
MA²P: A Meta-Cognitive Multi-Agent Framework for Complex Persuasion
The paper introduces MA²P, a multi-agent framework designed for complex persuasion tasks where the persuadee's internal states are latent. The system coordinates perception management, mental-state inference, strategy execution, memory, and evaluation modules, and adds a meta-cognitive configurator that selects domain-appropriate strategies from a structured knowledge base to reduce cross-domain performance variance. Experiments show higher persuasion success rates compared to baselines. The work addresses a known weakness of LLMs in producing generic or weakly grounded persuasive responses.