4arXiv cs.CL (Computation and Language)·
Gemma 4 31B benchmarked on NRC Reactor Operator Licensing Exam via fine-tuning and RAG configurations
Researchers benchmark eight fine-tuning and retrieval configurations of Gemma 4 31B-IT against the U.S. NRC Reactor Operator licensing examination across 14 Generic Fundamentals Examinations from 2015–2021. The best configuration—supervised fine-tuning with Gemini-distilled chain-of-thought rationales combined with fixed-size chunking RAG—passed 8 of 14 exams and reached 79.7% aggregate accuracy, near the 80% human passing threshold. Key findings include that no configuration without fine-tuning passed any exam, that preferred chunking strategy reverses depending on training state, and that RAFT underperforms standard SFT in matched retrieval environments.
Related guides (3)
Related events (8)
DiffusionGemma hits 1,000+ tokens/sec; Claude Fable 5 export controls; Agents' Last Exam benchmark launch
Google introduced DiffusionGemma, an experimental 26B MoE model using diffusion-based text generation that produces 256-token blocks simultaneously, achieving over 1,000 tokens/second on H100 hardware at the cost of lower output quality versus standard Gemma 4. Separately, the US government issued an export control directive forcing Anthropic to suspend Claude Fable 5 and Claude Mythos 5 globally, while Anthropic also reversed a controversial silent-degradation safeguard on Fable 5 after researcher backlash. UC Berkeley's Center for RDI launched Agents' Last Exam (ALE), a 1,500+ task agentic benchmark using deterministic grading, where GPT-5.5 topped the leaderboard at only 24% pass rate, highlighting the difficulty gap between current models and professional-grade workflows.
LLMs evaluated for automated grading of Linux/bash exams using four-level cognitive taxonomy
A new arXiv paper evaluates GPT, Claude Opus, Gemini, and GLM on automated grading of 1,200 real student Linux/bash command responses, benchmarked against three expert instructors. Using a four-level cognitive taxonomy, Gemini 3.0 Pro with rubric-guided prompting achieved the highest human-AI agreement (ICC=0.888, MAE=0.10). Key findings: rubric quality mattered more than model choice, and grading accuracy declined consistently at higher cognitive complexity levels. The study proposes a taxonomy-based framework for deciding which exam questions are suitable for AI-assisted grading.
UMG-RAG: Training-free hybrid retrieval with uncertainty-aware granularity fusion for long-document RAG
Researchers propose Uncertainty-aware Multi-Granularity RAG (UMG-RAG), a training-free hybrid retrieval framework that addresses the tension between large and fine-grained retrieval chunks in RAG pipelines. The system converts dense and sparse retriever scores across multiple chunk granularities into evidence distributions, estimates reliability via entropy, and fuses candidates using query-specific confidence signals. A variant called UMGP-RAG uses fine-grained hits to locate evidence while returning broader parent chunks for coherence. Experiments on QA benchmarks show improved generation quality with no changes to the underlying retriever or generator.
RA-RFT: Retrieval-Augmented Reinforcement Fine-Tuning teaches LLMs to reason by analogy
Researchers propose Retrieval-Augmented Reinforcement Fine-Tuning (RA-RFT), a post-training framework that trains a retriever to rank contexts by expected reasoning benefit rather than semantic similarity, then fine-tunes a policy model via reinforcement learning using retrieved analogous demonstrations. The key insight is that reasoning-relevant retrieval surfaces complementary solution strategies rather than superficially similar problems. On mathematical reasoning benchmarks, RA-RFT improves AIME 2025 average@32 accuracy by 7.1 and 2.8 points over GRPO for Qwen3-1.7B and Qwen3-4B respectively, suggesting reasoning-aware retrieval is orthogonal to reward design and training curriculum improvements.
Fine-Tuning Gemma Models in Hugging Face
Hugging Face published a guide on fine-tuning Google's Gemma models using parameter-efficient fine-tuning (PEFT) techniques. The post covers practical workflows for adapting Gemma to downstream tasks within the Hugging Face ecosystem. This represents part of the broader tooling support rollout accompanying Gemma's release in February 2024.
Interpretability study of DiffusionGemma reveals novel diffusion-specific reasoning phenomena
Researchers investigate the reasoning transparency of DiffusionGemma, a diffusion-based language model, decomposing transparency into variable and algorithmic components. They show that mapping information through an interpretable token bottleneck reduces DiffusionGemma's opaque serial depth from 28.6X to just 1.1X that of autoregressive Gemma 4, with no performance loss. Interpretability case studies uncover diffusion-specific phenomena including non-chronological reasoning, token smearing, and intermediate-context reasoning. Monitorability tests find DiffusionGemma comparable to Gemma 4, suggesting diffusion LMs are not inherently less amenable to safety oversight.
GAMUT benchmark introduces two-level meta-rubrics for evaluating factual completeness in long-form generation
Researchers introduce GAMUT (Grounded Assessment of Multimodal Factuality), a benchmark of 1,813 questions targeting factual completeness—the recall side of factuality—in long-form LLM outputs. The framework uses a two-level meta-rubric that captures content organization and importance, then compiles it into flat binary checklists for reliable LLM-judge scoring. Evaluating 14 frontier and open-weight models, the best score is 58.7% from Gemini 3.1 Pro, indicating the benchmark is genuinely challenging and discriminative. The work addresses a gap in existing factuality evaluation pipelines, which focus on precision but not recall of required information.
GRADRAG: Cross-component prompt adaptation for coordinated multi-agent RAG pipelines
GRADRAG is a new framework that treats a multi-agent RAG pipeline as a computational graph and propagates structured evaluator feedback upstream to iteratively update retriever, graph constructor, and answerer prompts. Unlike prior work that optimizes components in isolation, GRADRAG coordinates improvements across the full pipeline via a Prompt Optimizer and an early-stopping Evaluator. Evaluated on SQUALITY and QMSUM benchmarks under both flat chunk-based and graph-based retrieval paradigms, GRADRAG achieves a 12–15 percentage point net preference margin over one-step refinement baselines in LLM-judged pairwise comparisons.