6arXiv cs.CL (Computation and Language)·
Pretraining data poisoning via public web discussion interfaces shown feasible at scale
A new arXiv paper demonstrates that LLM pretraining data can be poisoned through public discussion interfaces (e.g., comment sections, forums) at web scale, going beyond prior work that focused on controlled sources like Wikipedia. The authors introduce HalfLife, a novel analysis method for estimating how much adversarially injected content survives web crawling and data curation pipelines. The work establishes third-party webpage content as a viable attack vector and highlights the inadequacy of current curation pipelines as a defense against such poisoning.
Related guides (2)
Related events (8)
Unified defense framework detects and remediates data poisoning in text summarization fine-tuning
A new arXiv preprint introduces a post-hoc defense framework for detecting and recovering from training-time data poisoning in LLMs fine-tuned for abstractive summarization. The framework uses influence-function analysis in white-box settings and behavioral perturbation auditing in black-box settings, achieving 85-92% detection precision across nine architectures and six benchmarks. Gradient-ascent unlearning restores up to 96% of original model behavior with less than 0.6% ROUGE degradation. The authors also introduce novel attacks targeting factual distortion and representational bias that evade conventional evaluation metrics.
Poisoning attacks persist through GAN-based data augmentation in 3D point cloud datasets for autonomous vehicles
A new arXiv paper investigates whether data augmentation can neutralize poisoning attacks on 3D point cloud datasets used in connected and autonomous vehicle (CAV) perception systems. The authors find that poisoning survives GAN-based augmentation pipelines, propagates into augmented datasets, and continues to degrade downstream classifier decisions. Experimental materials including tools, datasets, and classifiers are released publicly to support reproducibility.
LLMs fail to reliably self-report adversarial prefill attacks, study finds
A new arXiv paper evaluates whether LLMs can recognize that their own prior responses were elicited by adversarial prefill attacks, testing ten open-weight models (3B–70B) across four safety benchmarks. Models claim intent on prefilled responses only 27.3% of the time on average, and introspective signal is largely mediated by refusal-related reasoning. Three LoRA fine-tuning methods (SFT, GRPO, DPO) improve the intention-probe gap but counterintuitively raise attack success rates on most models, suggesting partial and fragile mitigation. The findings raise concerns about the reliability of LLM self-reports in safety-critical contexts.
Prompt injection attacks on LLM-based résumé screening: effectiveness and fairness implications
A new arXiv paper studies prompt injection in automated résumé screening, where candidates embed subtle self-promotional text to manipulate LLM rankings without adding genuine qualifications. Controlled experiments show injection reliably boosts rankings when manipulation is rare and candidate quality is homogeneous, but effectiveness collapses as adoption spreads. The work raises fairness concerns because lower-quality candidates can occasionally outrank higher-quality ones, and identifies conditions under which LLM-based hiring systems are most vulnerable.
TRACE: Lightweight RAG corpus poisoning detection via token influence attribution
Researchers introduce TRACE, a detection framework for corpus poisoning attacks on Retrieval-Augmented Generation (RAG) systems that works by tracing answer-related tokens through token influence attribution rather than relying on auxiliary classifiers or LLM-based verification. The method identifies recurrent high-influence keywords across retrieved documents and performs secondary verification to confirm their effect on model predictions. Evaluated on three QA benchmarks and six LLMs, TRACE achieves strong detection performance while also exposing attacker-specified target answers, with lower computational overhead than prior approaches.
Building an Early Warning System for LLM-Aided Biological Threat Creation
OpenAI published a blueprint for evaluating whether LLMs can meaningfully assist in biological threat creation. In a controlled study with biology experts and students, GPT-4 was found to provide at most mild uplift in biological threat creation accuracy. The results are inconclusive but are framed as a starting point for ongoing safety research and community deliberation on biosecurity risks from AI.
FORGE benchmark reveals search-augmented LLMs vulnerable to fake product promotion via web content pollution
Researchers introduce FORGE, a benchmark measuring how often search-augmented LLMs recommend fake products when retrieval results are polluted with fabricated reviews or promotional pages. Across 12 commercial and open-weights models, a single polluted page causes fooled rates up to 27%, rising to 73.8% when all top-3 results are replaced. Notably, chain-of-thought reasoning does not mitigate the vulnerability and often generates spurious social proof to justify false recommendations. Three defenses tested—skepticism prompting, model-prior filtering, and cross-document consensus—each carry significant drawbacks.
Causal auditing framework detects privacy disclosures in synthetic data without model access
A new arXiv preprint introduces a model-agnostic empirical framework for auditing synthetic data generated by LLMs and generative AI systems for privacy leakage. The framework distinguishes 'true disclosures' (direct reproduction of user data) from 'phantom disclosures' (incidental generation), using held-out control sets and statistical hypothesis testing without requiring model access, canary insertion, or shadow model training. It functions as a membership inference attack and provides empirical lower bounds on privacy leakage that are tighter than prior data-based auditing methods. The approach is computationally lightweight and applicable to any synthetic data generation mechanism.