6arXiv cs.AI (Artificial Intelligence)·
CRAFT: Rubric-based capability diagnosis and targeted fine-tuning data generation for LLMs
CRAFT is a new method that converts rubric-based evaluation datasets into model-specific capability diagnoses by extracting capability descriptions from prompt-rubric pairs, clustering them into a hierarchical tree, and scoring models at each node to identify weak capabilities. The identified weaknesses then direct targeted supervised fine-tuning data generation. Evaluated across four open-source models, two professional domains (finance and legal), and 13 held-out benchmarks, CRAFT outperforms prompt-level clustering and random data generation baselines in most settings. The approach addresses a gap in evaluation pipelines by explaining not just where models fail but why, and linking that diagnosis directly to post-training improvement.
Related guides (2)
Related events (8)
Rubrics on Trial: Self-evolving evaluation rubrics from a single query via synthetic pairwise evidence
A new arXiv preprint introduces 'Rubrics on Trial,' a query-only framework that automatically constructs and validates evaluation rubrics for LLMs without external annotations or model training. The system evolves rubric sets from scratch using synthetic rubric-conditioned response pairs, screening out non-discriminative, over-specific, or style-only rubrics. Evaluated across five preference benchmark suites, it achieves best average accuracy and leads on six of seven evaluation sets, addressing a key bottleneck in scalable LLM evaluation.
RubricsTree: Scalable hierarchical rubric framework for evaluating personal health AI agents
RubricsTree is a new evaluation framework for LLM-powered personal health agents, built around a hierarchical taxonomy of over 100 clinically-verifiable Boolean rubrics derived from 4,000 real user queries and curated with physician oversight. A context-aware router activates only relevant rubrics per query, enabling scalable yet expert-aligned evaluation. The framework outperforms strong LLM-as-a-judge baselines on expert alignment and, when used as training signal, yields up to ~66% relative gains on HealthBench across Gemini, GPT, and Qwen model families. The work addresses a concrete bottleneck in clinical deployment of health AI: the cost-quality tradeoff in evaluation.
Meta-evaluation of LLM-generated rubrics for paper reproduction benchmarks
A new arXiv preprint presents the first systematic meta-evaluation of LLM-generated rubrics for assessing paper reproduction tasks, addressing scalability limitations of expert-constructed rubrics in benchmarks like PaperBench. The authors test four generation settings across two backbone models, evaluating rubric quality both intrinsically (semantic similarity) and extrinsically (score alignment with ground-truth). Results show augmented generation settings can approach human-baseline alignment, but LLM-generated rubrics tend to be overly fine-grained, score-biased, and domain-insensitive.
Creative Quality Alignment: Expert Tacit Knowledge Transfer via Chain-of-Thought Fine-Tuning
This paper empirically validates a creative quality metric from a companion work (Calibrated Surprise, Zou & Xu 2026a) under strict low-resource conditions: ~100 expert chain-of-thought annotations and a small base model. The authors introduce Creative Quality Alignment (CQA) as a class of engineering methods and identify a systematic bias in public alignment datasets toward craft knowledge, with weak coverage of audience modeling and reality-logic. A theoretical argument based on 'architectural duality' in single conditional distribution LLMs is offered to explain why so few examples suffice, distinguishing the result from purely empirical findings like LIMA.
DEFINED: Data-efficient framework for fine-grained creativity assessment in debate using LLMs
DEFINED is a computational framework for automated creativity assessment in debate scenarios, operationalizing creativity through an eight-dimensional hierarchical metric system implemented via a pretrained autoregressive language model with a hierarchical scoring head. The system addresses data scarcity through constrained data augmentation and mixed-granularity training from limited expert-annotated data. It outperforms prompt-based LLM evaluators and existing debate scoring methods on authentic competition data. The work is relevant to AI evaluation methodology and the broader question of whether LLMs can reliably assess complex human cognitive outputs.
Cross-rubric generalization for automated essay scoring using LLM fine-tuning with trait-based representations
A new arXiv paper introduces a framework for automated essay scoring (AES) that generalizes to previously unseen scoring rubrics, rather than just unseen prompts. The approach uses rubric-agnostic intermediate representations called 'traits' combined with target-essay supervision, achieving a 5.0% macro F1 improvement over a baseline in the hardest generalization setting. A fine-tuned Llama-based model outperforms GPT-5-mini prompting by 2.1% macro F1 and trails GPT-5 by only 1.9%, demonstrating that structured intermediate representations improve rubric generalization.
LongCrafter: Evidence-graph-guided synthesis framework for long-context SFT data
LongCrafter is a structured framework for synthesizing long-context supervised fine-tuning data, addressing limitations of prior approaches including narrow task coverage, low difficulty, and lack of faithfulness supervision. The system uses a hierarchical 32-task taxonomy and constructs explicit evidence graphs modeling cross-paragraph dependencies to generate grounded instruction-response pairs. Models fine-tuned on LongCrafter data outperform SFT baselines and official post-trained models on LongBench, LongBench v2, and LooGLE for both Qwen2.5-7B and LLaMA-3.1-8B, with notable gains on high-difficulty tasks and improved robustness to the 'lost in the middle' problem.
EDIT framework trains more rubric-faithful LLM graders via internal-state diagnostics
Researchers introduce Evidence-Diagnosed Intervention Training (EDIT), a two-phase framework for improving LLM-based rubric grading. The first phase (EDIT-SFT) identifies problematic reasoning steps using posterior belief signals and input-grounding scores, then revises only those steps with rubric checklists; the second phase (EDIT-RL) uses belief-guided reward shaping to penalize harmful belief drifts during RL. Experiments on two real-world multi-subject grading benchmarks show consistent improvements over SFT and RL baselines on both in-domain and out-of-domain splits.