6arXiv cs.CL (Computation and Language)·
VeyraBench: Controlled study of prompt format, instruction count, and context length on LLM adherence and hallucination
A new arXiv preprint reports two controlled experiments examining how prompt format (markdown, plain text, prose, tabular), instruction count, and context length affect instruction-following and hallucination across five LLMs. Key findings: perfect instruction-following collapses to zero by N=80 rules regardless of format or model; recall holds through 64-128k tokens then degrades sharply and format-dependently; fabrication is near-zero but refusal rates spike to 79-90% near context ceilings. The authors release VeyraBench, a contamination-free synthetic corpus of 8,780 uniquely-named entities, along with the full experimental harness.
Related guides (2)
Related events (8)
BenHalluEval: Multi-Task Hallucination Evaluation Framework for Bengali LLMs
BenHalluEval introduces the first systematic hallucination benchmark for Bengali, covering four tasks (generative QA, code-mixed QA, summarization, reasoning) with 12,000 hallucinated candidates generated via GPT-5.4 across twelve hallucination types. Seven LLMs are evaluated under a dual-track protocol separating false-positive rate on ground-truth instances from hallucination detection rate on hallucinated candidates. The proposed BenHalluScore metric reveals substantial variation (7.72%–55.42%) across models and tasks, and chain-of-thought prompting is found to shift response distributions without consistently improving hallucination discrimination. The work highlights gaps in low-resource language hallucination evaluation and critiques single-track and prompting-only evaluation approaches.
Systematic evaluation of LLM prompt sensitivity in healthcare settings reveals safety risks
Researchers conduct a sensitivity analysis of both general-purpose and medical-specific LLMs using the MedMCQA benchmark, testing robustness to lexical and syntactic prompt perturbations. The study finds that even minor phrasing changes can alter clinical advice, and adversarial prompts can produce dangerous outputs such as incorrect dosages or omitted critical findings. Both general-purpose models (GPT-3.5, Llama 3) and domain-specific models (ClinicalBERT, BioLlama3, BioBERT) exhibit this fragility, with syntactic reordering and misleading contextual cues proving more destabilizing than simple paraphrasing.
Systematic study reveals effectiveness-fluency trade-offs in LLM conditioning methods
A new arXiv paper systematically evaluates a range of LLM conditioning methods across both concept injection and removal scenarios, finding that efficient steering methods often degrade fluency significantly. A key finding is that activation steering is substantially less effective on instruction-tuned models than on base models, a previously overlooked interaction. Simple prompting and supervised fine-tuning work for concept injection but not removal, and cheap textual metrics are found to correlate well with expensive LLM-as-judge evaluations.
TriggerBench: A benchmark for evaluating prospective memory in LLMs
Researchers introduce TriggerBench, a benchmark evaluating prospective memory (PM) in LLMs — the ability to spontaneously recall and act on latent constraints without explicit prompting. The benchmark spans five dimensions across daily assistant and professional workflow scenarios, and reveals that PM is substantially harder than retrospective memory, decaying sharply with context length while retrospective memory near-saturates at 100K tokens. Key findings include a precision-recall trade-off in PM, attentional fragility under concurrent requests, and a novel result that PM accuracy correlates with spare reasoning capacity as measured against AIME-2025 math performance.
Structured Prompt Checklists Outperform Raw and Clarifying-Question Prompts Across LLMs
This paper compares three prompt design strategies—raw prompts, checklist-improved prompts, and clarifying-question prompts—across four task types and three LLM systems (ChatGPT, Claude, Grok). Checklist-improved prompts achieved the highest mean rubric score (7.50/8) versus 5.67 for raw and 6.67 for clarifying-question prompts. Checklist prompts also used fewer tokens on average, suggesting a favorable quality-effort tradeoff. The study provides empirical grounding for structured prompt engineering as a practical technique to reduce multi-turn interaction overhead.
Structural test oracle problem in LLM-as-judge synthetic corpora: silent failures and a validation protocol
A new arXiv paper reports a case where a shared decoding-budget parameter silently truncated hallucinated answers in a multilingual LLM-as-judge evaluation corpus, producing a spurious 32-point cross-lingual accuracy collapse that replicated robustly across sample sizes but was entirely artifactual. The authors argue this failure mode is structural to LLM-generated negative examples, which lack any mechanical item-level integrity check (the 'test oracle problem'), unlike corpora built from deterministic perturbation of gold answers. A second real bias (Markdown formatting preference) was simultaneously distorted in magnitude and sign by the same fault, illustrating that aggregate statistics cannot distinguish fabricated from distorted effects. The paper closes with a validation protocol for analysts working with oracle-less multilingual LLM-as-judge corpora.
Prompting Complexity: A Kolmogorov-Inspired Formal Framework for Shortest LLM Prompts
A new arXiv preprint introduces 'prompting complexity,' a model-relative analogue of resource-bounded Kolmogorov complexity that measures the shortest plausible human-readable prompt needed to make a deterministic LLM produce a target text. The framework extends to soft prompting complexity for approximate outputs, prompting distance, and behavioral prompting complexity for specification-satisfying outputs. Unlike classical Kolmogorov complexity, the measure is intentionally non-universal and model-specific, with no invariance theorem across models. The paper lays out a research agenda for empirically studying which texts and behaviors are accessible from short prompts under a fixed model interface.
Inference-time steering methods compared for cross-lingual factual consistency in LLMs
Researchers investigate four intervention strategies for reducing cross-lingual factual inconsistency in LLMs, where models give different factual answers depending solely on prompt language. Testing on Gemma 3 12B Instruct, they compare persona prompting, Contrastive Activation Addition (CAA), and DPO-based adapters, finding that simple persona prompting outperforms more invasive methods in efficacy, safety, and generalization. The paper also introduces a novel culturally-rooted generalization benchmark to assess whether factual interventions transfer beyond the training distribution. Results suggest cross-lingual inconsistency is partly a selection problem addressable without weight modification.