6arXiv cs.CL (Computation and Language)·
Pitwall: Production system for faithful LLM-generated F1 race briefings with Monte Carlo grounding
Pitwall is a production NLP system that generates real-time Formula 1 strategy briefings in three languages, using a calibrated Monte Carlo simulation engine (N=2,000 continuations, 126 races of training data) as a grounding substrate. Every generated sentence is decomposed into typed factual claims and verified against the probabilistic race state before publication; fine-tuning data is filtered to only state-supported targets (81.9% retention), preventing the model from ever training on ungrounded outputs. The system was validated at two live Grands Prix (Austria and Britain, 2026) and surfaces a generalizable finding: hallucination in sparse-context grounding traces to base-model instruction adherence rather than model scale. The paper contributes both a practical faithfulness-as-architecture approach and a real-world deployment case for grounded generation under strict latency constraints.
Related guides (2)
Related events (8)
PGT: Procedurally Generated Tasks for Improving Visual Grounding in MLLMs
This paper introduces Procedurally Generated Tasks (PGT), a data-driven framework that overlays geometric primitives on images to create dense supervision signals for fine-grained visual grounding in multimodal large language models. PGT serves both as a training augmentation method and a diagnostic tool to isolate perception failures from semantic priors. Instruction tuning on LLaVA-v1.5-Instruct augmented with PGT data yields gains of up to +20% on the What'sUp benchmark and +13.3% on CV-Bench-2D. The results suggest that spatial reasoning deficits in MLLMs stem primarily from inadequate supervision rather than architectural or resolution constraints.
PowerCodeBench: Knowledge Boundary Probing and Intervention for LLM-Based Power System Code Generation
This paper introduces PowerCodeBench, an execution-validated benchmark for evaluating LLMs on power-system simulation code generation using the pandapower library. The authors identify that failures are dominated by API-knowledge boundary errors (hallucinated function names, misused parameters) rather than reasoning failures, and propose a boundary-aware intervention combining API demand estimation with targeted documentation injection. Evaluated across ten open-weight models (1.5B–480B) and four commercial APIs on 2,000 tasks, the intervention yields 32–56 accuracy point improvements while using only 41% of baseline prompt-token cost. Open-weight models in the 70B–120B range match commercial mid-tier accuracy, with Llama-3.1-405B and Qwen3-Coder-480B leading.
MMBench2 paper: hallucination in world models is predictable and preventable via coverage signals
Researchers introduce MMBench2, a 427-hour, 210-task dataset for visual world modeling, and train a 350M-parameter world model to study hallucination in generative world models. The paper identifies three distinct hallucination modes (perceptual, action-marginalized, scene-diverging) and develops lightweight signals that predict where models will fail. A coverage-aware sampling technique and curiosity-reward-based data collection enable efficient finetuning to unseen environments with as few as 50 real trajectories. The central finding is that world model hallucination is fundamentally a data coverage problem, with the same signals serving both detection and mitigation.
TokenWall: Semantic runtime firewall for persistent AI agent token flows
Researchers introduce TokenWall, a runtime defense framework that audits natural-language token flows in persistent AI agents before they reach privileged execution sinks. The system constructs source-sink audit records, applies lightweight local inspection, and escalates ambiguous cases to stronger arbitration modules. On CIK-Bench, TokenWall reduces attack success rate to 12.5% while maintaining a 97.4% benign pass rate and adding only 0.69 seconds of latency, demonstrating a practical security-utility tradeoff for long-lived agentic systems.
FACTS Benchmark Suite: Systematically evaluating the factuality of large language models
DeepMind has released the FACTS Benchmark Suite, a systematic evaluation framework for measuring the factuality of large language models. The benchmark is designed to assess how accurately LLMs produce factually grounded outputs. This represents a structured contribution to the growing field of LLM evaluation, specifically targeting hallucination and factual reliability. The announcement comes from a Tier 1 lab, lending it credibility as a reference benchmark in the field.
DeepSWIP: Counterfactual reasoning for neural probabilistic logic programs via quotient-WMC
DeepSWIP introduces a single-world counterfactual semantics for DeepProbLog, enabling causal inference over neurosymbolic programs that combine neural perception with probabilistic logic. The approach uses neural materialization to reduce neural predicates to standard ProbLog choices, then applies Single World Intervention Programs (SWIPs) and weighted model counting to compute exact counterfactuals from a single transformed program. Experiments on MPI3D validate the method against a DeepTwin construction across 12,000 queries and show a 2.14× inference speedup, while a SUMO HOV experiment demonstrates that neural calibration degradation biases plug-in causal estimates and that a correctly scoped AIPW estimator removes most first-order bias.
Future Probe Controlled Generation enables steering of reasoning models without quality degradation
Researchers introduce Future Probe Controlled Generation (FPCG), a text-level steering method for large reasoning models (LRMs) that trains activation probes to predict future behavior likelihoods from intermediate reasoning steps rather than detecting behavior in already-generated text. The probes achieve 64–91% accuracy in predicting the most likely future behavior, revealing a distinct class of internal prediction features separate from detection features. FPCG steers model outputs by sampling candidate sentences and selecting the best according to these probes, achieving steering with minimal output quality degradation and succeeding in cases where activation steering fails. The work provides a principled distinction between detection and prediction features as intervention targets for controlling LRM behavior.
PlanBench-XL: Benchmark for LLM Agent Planning in Large-Scale Tool Ecosystems
Researchers introduce PlanBench-XL, an interactive benchmark of 327 retail tasks spanning 1,665 tools designed to evaluate LLM agents on long-horizon planning under retrieval-limited tool visibility. The benchmark includes a blocking mechanism simulating real-world disruptions such as missing or failing tools, forcing agents to detect and recover from broken execution paths. Experiments on ten leading LLMs reveal severe performance degradation: GPT-5.4 drops from 51.90% accuracy in unblocked settings to 11.36% under the most severe blocking condition, highlighting fragility in adaptive planning for large, imperfect tool environments.