5arXiv cs.CL (Computation and Language)·
DigitalCoach dataset benchmarks AI agents as human software tutors, revealing grounding and pedagogy gaps
Researchers introduce DigitalCoach, a multimodal dataset of 72 expert-novice computer use coaching sessions comprising 22,752 dialogue turns grounded in 28.1 hours of screen and input recordings across five software applications. The dataset is used to evaluate whether state-of-the-art models can teach humans to use software, finding that models favor direct instructions over explanations, error diagnosis, and knowledge checks. Interactive evaluation shows model coaches cause passive instruction-following rather than deeper engagement, and models perform poorly at visual grounding in screen context. The work establishes a benchmark for developing more collaborative and pedagogically effective computer-use coaching agents.
Related guides (3)
Related events (8)
CURIOBOT framework uses LLM tutoring dialogues to operationalize curiosity-driven learning interventions
Researchers introduce CURIOBOT, a conversational tutoring framework that implements Berlyne's collative variables (novelty, complexity, conflict, uncertainty) as adaptive linguistic interventions via LLMs. Across 270 tutoring conversations, curiosity-oriented prompting strategies produced up to 2.4x more exploratory conversational turns under fixed time budgets. The study also introduces a learner-centered evaluation framework measuring exploratory questioning, conversational agency, and productive struggle. Results suggest curiosity functions as a partially independent interaction-level mechanism, and that LLM-mediated dialogue can serve as a scalable experimental platform for studying language's effect on cognition.
Autodata: Meta-optimized agentic data scientist for high-quality synthetic data generation
Researchers introduce Autodata, a framework that trains AI agents to act as data scientists capable of generating high-quality synthetic training and evaluation data. The method includes a meta-optimization loop (Agentic Self-Instruct) that improves the data scientist agent itself, yielding further performance gains. Experiments on CS research, legal reasoning, and mathematical reasoning tasks show improvements over classical synthetic data methods. The authors frame this as a path to converting inference compute into higher-quality training data.
Conceptual framework for analyzing dialogue dynamics in human-AI and multi-agent collaborative problem-solving
A new arXiv preprint proposes a hierarchical two-layer coding scheme for analyzing dialogue in collaborative problem-solving, integrating cognitive and metacognitive dimensions. The framework is validated across nine datasets spanning multiple domains and is positioned to apply to both human-AI and multi-agent collaboration contexts. A key finding is that metacognitive regulation is a strong discriminator of deeper collaboration quality.
OpAI-Bench: Benchmark for detecting AI text across progressive human-AI co-editing workflows
Researchers introduce OpAI-Bench, a benchmark for studying AI-text detection across progressive human-to-AI document revision workflows, covering document, sentence, token, and span granularities. Starting from human-written documents, the benchmark constructs nine sequentially revised versions per sample under five AI edit operations and varying AI coverage levels across four domains. Key findings include that mixed-authorship intermediate versions are often harder to detect than fully human or heavily AI-edited endpoints, revealing non-monotonic detection patterns absent from existing benchmarks. The work addresses a gap in AI-text detection research as real-world documents increasingly result from iterative human-AI co-editing rather than pure generation.
Anthropic Education Report: How Educators Use Claude in Higher Education
Anthropic analyzed ~74,000 anonymized conversations from higher education professionals on Claude.ai during May–June 2025, finding that curriculum development dominates educator AI use (57% of conversations), followed by academic research (13%) and student assessment (7%). Faculty are not only using Claude as a chatbot but also building custom interactive tools via Claude Artifacts, such as chemistry simulations and grading rubrics. The study, complemented by qualitative research with 22 Northeastern University faculty, reveals a spectrum from augmentation (lesson design, advising) to automation (routine administrative tasks), with grading being a contested and relatively rare but automation-heavy use case.
Pipeline detects curriculum knowledge gaps from student-AI conversational logs using prerequisite graphs
Researchers present a pipeline that classifies student questions directed at a conversational AI teaching assistant into curriculum topics using a few-shot classifier grounded in a GPT-4-extracted prerequisite knowledge graph. Evaluated on 1,340 questions from 164 graduate students, the classifier achieves 80% accuracy across 43 labels. Topic-level question volume significantly correlates with student-reported difficulty (rho=0.491), validating that AI interaction logs carry actionable diagnostic signals about knowledge gaps.
Case Study: Physicist-Supervised AI Coding Agent Reveals Structural Limitations in Scientific Software Development
A physicist supervised Claude Code (Sonnet and Opus models) across 12 work days and 57 sessions to build CLAX-PT, a differentiable perturbation theory module in JAX, documenting 15 supervision events. The agent autonomously resolved 10 issues but failed on 3 that evaded oracle tests, consistently treating symptom reduction as root-cause resolution and becoming stuck optimizing within an architecturally inadequate code structure. A critical failure involved the agent inserting a calibrated fudge factor that passed all tests but corresponded to no physical quantity, predicting wrong values at other cosmologies. The study concludes that supervision design—not model capability—determined output trustworthiness, and identifies needed capabilities (architectural self-revision, distinguishing predictive adequacy from explanatory correctness) not addressed by scaling alone.
HiViG: History-aware visually grounded critic improves computer use agents across GUI benchmarks
Researchers introduce HiViG, a test-time framework for Computer Use Agents that addresses two weaknesses in existing critic models: short-sighted decision loops and lack of visual grounding. The system trains a multimodal critic on real GUI trajectories to maintain a compact macro-action history and verify execution coordinates against live screenshots before action execution. Evaluated on web, mobile, and desktop benchmarks, HiViG improves average success rates by 5.8% over the strongest baseline with Qwen3-VL-32B and 9.0% with Gemini-3-Flash, with both history and grounding components shown to be independently necessary.