4arXiv cs.CL (Computation and Language)·
Git-Assistant combines LLMs with automated planning for git repository management
Git-Assistant is a hybrid AI system that pairs LLMs with automated planning techniques to translate natural language developer requests into correct git command sequences. The paper introduces a systematic evaluation methodology using synthetic and randomized git environments, comparing LLM-only versus planning-augmented variants. Results show that integrating formal reasoning with LLMs improves reliability and reduces errors in repository management tasks, supporting the case for hybrid AI approaches in developer tooling.
Related guides (1)
Related events (8)
Aider: AI Pair Programming Tool Trending on GitHub
Aider is an open-source AI pair programming tool that runs in the terminal, enabling developers to interact with LLMs to write and edit code directly in their local repositories. The project has accumulated 45,244 GitHub stars with 40 new stars today, indicating sustained community interest. It represents a prominent example of the agent/tooling ecosystem for AI-assisted software development.
Expert Support Case Study: Bolstering a RAG App with LLM-as-a-Judge
Hugging Face published a case study describing how Digital Green used an LLM-as-a-Judge approach to evaluate and improve a retrieval-augmented generation (RAG) application. The post covers the methodology for using LLMs to score and validate RAG outputs, providing a practical deployment pattern for quality assurance in production AI systems. It serves as a concrete example of enterprise-grade evaluation pipelines built on top of RAG architectures.
LLMs evaluated for automated grading of Linux/bash exams using four-level cognitive taxonomy
A new arXiv paper evaluates GPT, Claude Opus, Gemini, and GLM on automated grading of 1,200 real student Linux/bash command responses, benchmarked against three expert instructors. Using a four-level cognitive taxonomy, Gemini 3.0 Pro with rubric-guided prompting achieved the highest human-AI agreement (ICC=0.888, MAE=0.10). Key findings: rubric quality mattered more than model choice, and grading accuracy declined consistently at higher cognitive complexity levels. The study proposes a taxonomy-based framework for deciding which exam questions are suitable for AI-assisted grading.
GitHub's plan for agentic coding — Kyle Daigle interview on Latent Space
Latent Space interviews Kyle Daigle of GitHub about the company's strategy for agentic coding workflows and the platform pressures created by the explosion in AI-assisted development following Copilot. The discussion covers how GitHub is adapting its infrastructure and product direction to support agents operating at scale. This is a strategic signal from one of the most central platforms in the developer AI ecosystem.
Agentic CLEAR: Automating Multi-Level Evaluation of LLM Agents
Agentic CLEAR is an automatic evaluation framework for LLM-based agentic systems that analyzes behavior at three granularity levels: system, trace, and node. Unlike existing tools that rely on static error taxonomies or focus only on observability, it dynamically generates textual insights and integrates above the observability layer with an accessible UI. Experiments across four benchmarks and seven agentic settings demonstrate strong alignment with human-annotated errors and predictive accuracy for task success rates.
Google's Paper Assistant Tool (PAT) automates scientific peer review with agentic AI
Google introduces the Paper Assistant Tool (PAT), an agentic AI framework that ingests full scientific manuscripts and produces comprehensive evaluations including theoretical checking, experiment validation, and flaw identification. PAT uses inference scaling techniques to achieve a 34% improvement over zero-shot recall on mathematical errors in the SPOT benchmark. The system was piloted as a pre-submission tool at two major CS conferences (STOC and ICML), demonstrating practical deployment at scale. The paper also proposes a four-level taxonomy of AI-human collaboration in scientific evaluation.
GitOfThoughts: Git-based agent memory substrate with sobering findings on memory utility for novel problems
Researchers introduce GitOfThoughts, a system that stores LLM reasoning trees as git repositories, enabling replayable, auditable, and mergeable agent memory at low engineering cost. Across five memory substrates (none, markdown, vector, graph, git), two benchmarks, and two model scales with pre-registered replications, the paper finds that no memory format reliably improves accuracy on novel problems. Memory only helps above a 'copyability threshold' (similarity >~0.8), where retrieved cases are near-duplicates of the current problem — and even then, the gain is answer retrieval rather than method transfer. The paper also documents a retracted result and refuted hypothesis, modeling a rigorous evaluation standard.
Grading the Grader: Evaluating automated graders for agentic data analysis systems
A preprint from arXiv investigates the reliability of automated graders for evaluating agentic data analysis systems, which produce complex multi-modal outputs (code, numerical results, diagnostics) that are harder to assess than single-turn LLM responses. The authors apply LAMBDA, a multi-agent data analysis system, to 153 numerical tasks from DSGym and develop a three-layer human-AI grading cascade combining regex matching, LLM-based lenient grading, and human inspection. Key findings include: both automated graders achieve 100% precision, a keyword-anchored extraction pipeline raises strict grader recall by 60 percentage points, and an iterative nudge mechanism raises grading success from 36% to 97%. The work surfaces important methodological lessons for anyone building evaluation pipelines for agentic systems.