3arXiv cs.CL (Computation and Language)·
Cross-rubric generalization for automated essay scoring using LLM fine-tuning with trait-based representations
A new arXiv paper introduces a framework for automated essay scoring (AES) that generalizes to previously unseen scoring rubrics, rather than just unseen prompts. The approach uses rubric-agnostic intermediate representations called 'traits' combined with target-essay supervision, achieving a 5.0% macro F1 improvement over a baseline in the hardest generalization setting. A fine-tuned Llama-based model outperforms GPT-5-mini prompting by 2.1% macro F1 and trails GPT-5 by only 1.9%, demonstrating that structured intermediate representations improve rubric generalization.
Related guides (2)
Related events (8)
Meta-evaluation of LLM-generated rubrics for paper reproduction benchmarks
A new arXiv preprint presents the first systematic meta-evaluation of LLM-generated rubrics for assessing paper reproduction tasks, addressing scalability limitations of expert-constructed rubrics in benchmarks like PaperBench. The authors test four generation settings across two backbone models, evaluating rubric quality both intrinsically (semantic similarity) and extrinsically (score alignment with ground-truth). Results show augmented generation settings can approach human-baseline alignment, but LLM-generated rubrics tend to be overly fine-grained, score-biased, and domain-insensitive.
Rubrics on Trial: Self-evolving evaluation rubrics from a single query via synthetic pairwise evidence
A new arXiv preprint introduces 'Rubrics on Trial,' a query-only framework that automatically constructs and validates evaluation rubrics for LLMs without external annotations or model training. The system evolves rubric sets from scratch using synthetic rubric-conditioned response pairs, screening out non-discriminative, over-specific, or style-only rubrics. Evaluated across five preference benchmark suites, it achieves best average accuracy and leads on six of seven evaluation sets, addressing a key bottleneck in scalable LLM evaluation.
GPT-5 variants benchmarked for summarization-assisted automated essay scoring on ASAP 2.0
Researchers propose a hybrid framework for automated essay scoring (AES) that uses GPT-5, GPT-5 mini, and GPT-5 nano to generate controlled-length summaries of long essays, addressing transformer input-length limitations. Summaries are combined with handcrafted linguistic features and fed into downstream AES models evaluated on the ASAP 2.0 dataset using quadratic weighted kappa. GPT-5 mini achieves the best human-rating agreement while GPT-5 produces higher summarization quality, revealing cost-performance trade-offs. The study also finds that higher-scoring, more complex essays are harder to compress without information loss, raising fairness concerns for educational deployment.
PARL: Preference-Aware Rubric Learning for Personalized LLM Evaluation
This paper introduces PARL (Preference-Aware Rubric Learning), a framework that reframes personalized LLM evaluation as a learning problem rather than static judgment. PARL induces preference-aware evaluation rubrics from raw user interaction histories and uses a discriminative reinforcement learning objective to contrast user-authored responses against model outputs, capturing user-specific decision boundaries. Experiments on personalized text generation tasks show PARL produces high-fidelity rubrics that generalize across users and tasks, outperforming existing LLM-as-a-judge and automatic metric approaches.
Mechanistic analysis of how LLMs encode essay quality in internal representations
Researchers systematically probe the hidden representations of eight LLMs across three essay datasets (ASAP++, CSEE, ENEM) to understand how automated essay scoring (AES) works internally. Using linear probing, dimensionality reduction, and neuron-level analysis, they find essay quality is encoded in a linearly accessible form that emerges progressively across layers and partially transfers across prompts. Individual 'essay scoring neurons' are identified whose activations correlate with scores and respond to targeted interventions, with longer essays relying more on deeper layers. The work contributes to mechanistic interpretability of LLM-based scoring systems.
AMARIS: Memory-Augmented Rubric Improvement System for Rubric-Based Reinforcement Learning
AMARIS introduces a persistent evaluation memory system to improve rubric-based reward shaping in LLM fine-tuning via reinforcement learning. Unlike prior adaptive rubric methods that discard evaluation diagnostics after each step, AMARIS accumulates step-level summaries and retrieves relevant historical context via both static (recent steps) and dynamic (semantic similarity) retrieval to inform rubric updates. The system runs asynchronously alongside the RL training loop with approximately 5% time overhead. Experiments across closed and open-ended domains show consistent improvements over baselines, with ablations confirming that combining both retrieval modes yields the strongest results.
Benchmarking LLM judges for citation verification in deep-research systems
A new arXiv paper evaluates 8 LLM judges from 3 model families on citation quality assessment for deep-research systems, testing across 1,248 rubric decisions with human-reviewed gold labels. The study finds that cheaper models remain competitive with frontier models — GPT-5-mini achieves the strongest source-relevance F1 at 0.908 — but judges differ substantially in directional bias (pass-rate drift, false positive/negative rates) even when scalar F1 scores are similar. The key finding is that scalar F1 obscures biases that would be directly reinforced in an RL training loop, making judge calibration a prerequisite before using citation rubrics as reward signals.
RubricsTree: Scalable hierarchical rubric framework for evaluating personal health AI agents
RubricsTree is a new evaluation framework for LLM-powered personal health agents, built around a hierarchical taxonomy of over 100 clinically-verifiable Boolean rubrics derived from 4,000 real user queries and curated with physician oversight. A context-aware router activates only relevant rubrics per query, enabling scalable yet expert-aligned evaluation. The framework outperforms strong LLM-as-a-judge baselines on expert alignment and, when used as training signal, yields up to ~66% relative gains on HealthBench across Gemini, GPT, and Qwen model families. The work addresses a concrete bottleneck in clinical deployment of health AI: the cost-quality tradeoff in evaluation.