6arXiv cs.CL (Computation and Language)·
Knowledge-Less Language Models (KLLMs) suppress parametric recall via entity anonymization during pretraining
Researchers introduce Knowledge-Less Language Models (KLLMs), pretrained on corpora with anonymized named entities to systematically reduce parametric factual recall and push models toward evidence-grounded reasoning. Across multiple model scales, KLLMs outperform standard baselines on contextual QA, fact verification, and hallucination detection, with 20-25% relative gains in retrieval-grounded settings with imperfect evidence. The models also show improved calibration (ECE, Brier score, AUROC) and more reliable abstention behavior. The work suggests pretraining-time control over knowledge acquisition can serve as a complementary foundation for RAG and tool-augmented systems.
Related guides (2)
Related events (8)
Co-LMLM: Continuous-query limited memory language models outperform vanilla LLMs on factual tasks at small scale
Researchers introduce CO-LMLM, a limited memory language model that externalizes factual knowledge to a knowledge base during pretraining and retrieves it at inference via continuous vector queries paired with human-readable text values. The approach removes prior restrictions to relational knowledge bases and Wikipedia-only data by introducing an annotation pipeline for arbitrary text. At 360M parameters, CO-LMLM achieves lower perplexity than models trained on 40x more data and SimpleQA factual performance comparable to GPT-4o mini and above Claude Sonnet 4.5, suggesting significant efficiency gains for factual grounding.
LMs encode knowledge in task-specific parameter subsets, undermining the knowledge-base analogy
A new arXiv paper investigates whether language models satisfy the consistency property of knowledge bases — that the same fact returns consistent results regardless of query form. Behavioral and mechanistic analyses reveal that LMs encode knowledge in a task-specific manner: facts acquired on one task frequently fail to transfer to others during training, and distinct parameter subsets underlie the same fact across different tasks. The authors also show that chain-of-thought reasoning derives part of its effectiveness by engaging task-specific parameters beyond those tied to the evaluation task, with implications for factual reliability and model controllability.
KG grounding helps LLMs only for out-of-training knowledge: controlled clinical QA study
A new arXiv paper investigates when knowledge-graph (KG) grounding improves LLM performance on clinical question answering, finding that structured KG retrieval over the public biomedical graph PrimeKG provides no meaningful improvement on MedQA (all deltas ≤3.4) because the relevant facts are already in the model's training data. On synthetic counterfactual and hybrid benchmarks containing genuinely novel facts, the same pipeline lifts out-of-training accuracy from chance to ~100%. The paper also reproduces and partially corrects a recent Nature Medicine study on frontier LLMs vs. clinical RAG tools, flagging a score-deflating grader bug and clarifying that the reported ~88 HealthBench score reflects the Consensus variant, not full HealthBench (~46-47). The core finding — that RAG/KG grounding pays off only when the decisive fact is outside the model's training distribution — has direct implications for when retrieval augmentation is worth deploying.
Graded entity-familiarity probes in LLMs enable refusal steering and cross-language robustness
Researchers probe activations at the final prompt token across twelve instruction-tuned models (Bielik, PLLuM, Gemma-4, Qwen3 families) to assess whether LLMs encode graded familiarity with named entities before generating answers. Using a new dataset of 1,440 Polish entities across popularity deciles plus fabricated controls, they find familiarity probes reliably separate real from fabricated entities and track popularity in Polish-adapted models. A key finding is that a single one-dimensional familiarity direction injected at one layer in Gemma-4-12B can steer refusal rates monotonically from 0.24 to 1.00 (or 0.73 to 0.00), revealing a separation between representational familiarity and the policy that converts it into abstention behavior.
Clinically grounded privacy evaluation framework reveals high memorization risk in medical LMs
Researchers introduce a tiered adversarial framework for evaluating privacy leakage in medical language models, moving beyond simple training-text recovery to realistic clinical threat models. Applied to an LM pretrained on 378k clinical notes, the framework finds that routine encounter metadata (name, DOB, provider, visit date) elicits high verbatim memorization and sensitive-diagnosis recovery (AUROC 0.91 for abortion, 0.81 for HIV). The study also finds that exact-match memorization overstates disclosure risk because 36% of memorized tokens reflect templated documentation. The work provides a practical contextual privacy evaluation methodology for medical LMs trained on longitudinal patient data.
KATE framework improves LLM tool calling via experiential knowledge integration and parallel reasoning
Researchers present KATE (Knowledge-Augmented Tool Execution), a framework addressing LLM failures in multi-step tool use by systematically studying knowledge acquisition, activation, and internalization. Key findings include that instance-level experiential knowledge outperforms abstract intent-level knowledge, that expanding reasoning width via parallel sampling with aggregation beats deeper chain-of-thought, and that reinforcement learning outperforms supervised fine-tuning for knowledge internalization. KATE is evaluated on BFCL-V3 and AppWorld benchmarks, showing consistent improvements over strong baselines across model scales.
LLMs fail to reliably self-report adversarial prefill attacks, study finds
A new arXiv paper evaluates whether LLMs can recognize that their own prior responses were elicited by adversarial prefill attacks, testing ten open-weight models (3B–70B) across four safety benchmarks. Models claim intent on prefilled responses only 27.3% of the time on average, and introspective signal is largely mediated by refusal-related reasoning. Three LoRA fine-tuning methods (SFT, GRPO, DPO) improve the intention-probe gap but counterintuitively raise attack success rates on most models, suggesting partial and fragile mitigation. The findings raise concerns about the reliability of LLM self-reports in safety-critical contexts.
Self-patching technique reveals why fine-tuned LLMs fail to generalize memorized knowledge
A new arXiv preprint formalizes the 'Knowing–Using Gap' in LLM fine-tuning: models can memorize injected facts yet fail to apply them in downstream reasoning. The authors introduce a novel intervention technique called self-patching to monitor internal activation dynamics, finding evidence for a knowledge-circuit misalignment hypothesis where memorized representations are not routed to computation-effective layers. A heuristic strategy derived from this diagnostic recovers 58–75% of the oracle headroom in generalization failure cases across multiple domains.