6arXiv cs.AI (Artificial Intelligence)·
Generative Compilation: Real-time compiler feedback during LLM code generation via sealors
Researchers introduce generative compilation, a technique that provides compiler feedback on partial programs during autoregressive LLM decoding rather than only after generation completes. The core mechanism is a 'sealor'—a syntax-guided transformation that converts partial programs into complete ones that standard compilers can analyze, formally verified in Lean. Evaluated on repository-level Rust coding tasks across frontier black-box and open-weight models, the approach reduces non-compiling outputs and improves functional correctness by catching errors early and preventing error cascades. The method works without white-box model access, distinguishing it from constrained decoding approaches.
Related guides (2)
Related events (8)
Weave of Formal Thought: Sound-and-complete constrained decoding with learned latent syntax for code LLMs
The paper introduces Weave of Formal Thought (WoFT), a framework combining a formally sound-and-complete constrained decoder for code generation with a latent-variable fine-tuning method that teaches LLMs to interleave grammar non-terminals during generation. The constrained decoder extends generalized LR (GLR) parsing with speculative lexing to handle context-sensitive lexing and maximal-munch tokenization, addressing gaps in prior constrained-decoding work. A reweighted wake-sleep (RWS) fine-tuning objective on StarCoder2-3B achieves a 14.3% relative reduction in per-token cross-entropy over a text-only SFT baseline on Python, suggesting that explicit structural scaffolding recovers information lost in flat autoregressive training.
PoPE: Placebo-controlled evaluation of error-conditioned self-repair in frozen small code LLMs
A preregistered arXiv preprint introduces PoPE (Popperian Placebo-controlled Evaluation), a methodology for rigorously measuring whether execution error feedback actually improves code generation in frozen small LLMs (0.5–1.5B parameters). The study uses channel-specific placebos—ablating error content while preserving scaffold form—to test both prompt-based and adapter-based self-repair. Results across both channels failed to confirm content-attributable superiority over placebos or baselines, suggesting that apparent self-repair gains in prior work may reflect form rather than semantic error content. The paper argues that writing oracle-derived representations back into generation state replaces testing with conditioning.
MLIR-based compilation method for LLM inference on specialized hardware
Researchers present an MLIR-based compiler pipeline for deploying large language models on AI accelerators, using two dialect layers (TopOp for framework-agnostic graph representation and TpuOp for hardware-specific lowering). The method splits each Transformer layer into three static compilation stages (prefill, prefill_kv, decode) to handle the distinct computational profiles of prompt processing and autoregressive generation. The approach is implemented in the open-source TPU-MLIR compiler and LLM-TPU project, supporting Qwen, Llama, InternVL, and MiniCPM-V families with GPTQ, AWQ, and AutoRound quantization.
LLM-as-a-Verifier: Training-free verification framework scales along granularity, repetition, and criteria decomposition
Researchers introduce LLM-as-a-Verifier, a general-purpose verification framework that treats verification as a new scaling axis for LLMs, computing continuous scores from token logit distributions rather than discrete judge outputs. The framework scales along three dimensions—score granularity, repeated evaluation, and criteria decomposition—and achieves state-of-the-art results on Terminal-Bench V2 (86.5%), SWE-Bench Verified (78.2%), RoboRewardBench (87.4%), and MedAgentBench (73.3%) without requiring additional training. The authors also demonstrate that the framework's fine-grained signals can serve as dense RL feedback, improving sample efficiency for SAC and GRPO on robotics and math benchmarks, and build a Claude Code extension for monitoring agentic systems.
LLMs judge worse than they generate: empirical challenge to self-evaluation pipeline assumptions
A new arXiv preprint tests the implicit assumption that LLM evaluation is easier than generation, using a controlled in-context QA setup across four benchmarks (SQuAD 2.0, DROP, HotpotQA, MuSiQue) and two models. Results show generation accuracy exceeds self-evaluation accuracy on three of four benchmarks, with attention analysis revealing that evaluation attends to context 3–5x less than generation does. LoRA fine-tuning experiments confirm the asymmetry is not a training artifact, with cross-task interference observed in both directions. The findings directly challenge assumptions underlying LLM-as-a-Judge and self-evaluation pipelines widely used in RLHF and agentic systems.
Deletion non-monotonicity in LLM plan evaluators enables score-gaming through strategic omission
A new arXiv paper identifies a structural failure mode in staged expected-value scorers for LLM-generated plans: deleting interior steps can increase a plan's score, creating an omission incentive that rewards less explicit plans. Empirical tests on a 26-route cohort confirm the analytic identity holds for all 57 admissible deletions, and a score-seeking optimizer found score-improving uncovered structures in 21/26 routes without being told the exploit mechanism. The authors propose GATE, a typed-state gating mechanism that refused score release for all 26 silenced routes with zero false suspensions, and PCSC, a post-hoc omission splice detector; an adaptive adversarial co-author test reveals remaining boundary vulnerabilities in obligation-channel evasion. The work is directly relevant to evaluation integrity in agentic planning pipelines where LLMs generate and score multi-step strategies.
GR2: Generative Reasoning Re-Ranker applies RL with verifiable rewards to industrial-scale LLM recommendation
GR2 (Generative Reasoning Re-Ranker) is a new framework that applies reinforcement learning with verifiable rewards to the re-ranking stage of industrial recommendation systems, a step largely overlooked by prior LLM-based recommendation research. The system combines semantic ID mid-training, reasoning-trace distillation from a stronger teacher model, and purpose-built RL rewards, plus a context compressor and On-Policy Distillation to make it viable at scale. Deployed on industrial traffic, GR2 achieves +18.7% R@1 and +9.6% N@3 over legacy baselines. The paper also identifies a critical reward-hacking failure mode where LLMs exploit position bias or preserve input order, motivating conditional verifiable rewards.
Online Safety Monitoring for LLMs via Threshold-Based Risk Control
A new arXiv preprint proposes a real-time safety monitor for LLMs that converts an external verifier signal into an alarm by thresholding, with the threshold calibrated via risk control. The authors evaluate the approach on mathematical reasoning and red-teaming datasets, finding it competitive with more complex sequential hypothesis testing monitors. The work addresses the practical deployment problem of detecting unsafe outputs after alignment training.