5arXiv cs.CL (Computation and Language)·
Training-free mechanistic defense against typographic attacks on CLIP-based vision encoders
Researchers propose a training-free method to defend CLIP-based vision encoders against typographic attacks, where irrelevant text embedded in images biases visual representations toward lexical rather than semantic meaning. The approach uses sampling-based mechanistic interpretability to identify specific Vision Transformer attention heads responsible for encoding lexical information, then applies targeted circuit-level interventions to suppress this behavior. Without any retraining, the method outperforms both supervised and training-free baselines on object classification and improves Visual Question Answering accuracy under typographic attack conditions on RIO-Bench across several state-of-the-art LVLMs.
Related guides (3)
Related events (8)
TEVI: Sparse autoencoders for text-conditioned editing of CLIP image embeddings to improve vision-language alignment
TEVI is a framework that uses sparse autoencoders to disentangle CLIP image embeddings and a learned masking module to selectively reconstruct embeddings conditioned on a given caption, addressing the information imbalance between images and their captions. The approach improves image-text retrieval on both coarse-grained benchmarks (MS COCO, Flickr) and fine-grained long-caption benchmarks (IIW, DOCCI), with larger gains on richer captions. The work also shows improved robustness on the RoCOCO benchmark.
Unified defense framework detects and remediates data poisoning in text summarization fine-tuning
A new arXiv preprint introduces a post-hoc defense framework for detecting and recovering from training-time data poisoning in LLMs fine-tuned for abstractive summarization. The framework uses influence-function analysis in white-box settings and behavioral perturbation auditing in black-box settings, achieving 85-92% detection precision across nine architectures and six benchmarks. Gradient-ascent unlearning restores up to 96% of original model behavior with less than 0.6% ROUGE degradation. The authors also introduce novel attacks targeting factual distortion and representational bias that evade conventional evaluation metrics.
SV-Detect: AI-generated text detection via steering vectors in representation space
SV-Detect proposes a method for detecting machine-generated text by extracting steering vectors from the hidden representations of a frozen language model, constructing layer-wise directions that separate human from AI-written text. A lightweight classifier trained on projection features achieves strong performance both in-distribution and under distribution shift across domains, source models, and editing attacks like polishing and rewriting. The approach reframes AI-text detection as a representation-space probing problem, with interpretation analyses showing the learned directions capture stylistic cues beyond surface features.
CLIP: Connecting Text and Images
OpenAI introduced CLIP (Contrastive Language-Image Pre-training), a neural network that learns visual concepts from natural language supervision. CLIP enables zero-shot visual classification by accepting natural language descriptions of categories rather than requiring task-specific training data. The approach mirrors the zero-shot transfer capabilities demonstrated by GPT-2 and GPT-3 in the language domain.
AirflowAttack: First adversarial attack on infrared remote-sensing VLMs using thermal-airflow perturbations
Researchers introduce AirflowAttack, the first adversarial attack targeting vision-language models deployed on infrared remote-sensing imagery, using physically plausible thermal-airflow turbulence as the perturbation prior. A single input-agnostic perturbation optimized on one surrogate CLIP model achieves 48.5% mean zero-shot attack success rate across five CLIP backbones, outperforming four IR-specific physical baselines. Applied to six state-of-the-art VLMs, the attack reduces scene-classification accuracy by up to 38.2% relative while paradoxically increasing model confidence, causing confabulation of thermal artifacts as genuine evidence. The work also releases a benchmark spanning eleven models and four tasks, exposing systematic vulnerabilities in security-critical IR VLM deployments.
Self-Filtering: Iterative bootstrapped data selection for vision-language model training
Researchers propose Self-Filtering, a bootstrapped data curation method for vision-language models in which a CLIP model iteratively trains on and re-selects its own training data. The approach alternates between filtering high-confidence clean samples and preserving distributional diversity, without requiring curated reference datasets or pre-trained external models. Experiments show downstream performance improvements over standard noisy training pipelines.
Visual pretraining on document images outperforms text-only pretraining for language intelligence
A new arXiv preprint argues that training language models directly on visual representations of documents (figures, equations, page layouts) consistently outperforms text-only pretraining on the same underlying corpora. The authors conduct a systematic study of unsupervised visual pretraining paradigms across multiple backbones and benchmarks, framing visual pretraining as a scalable alternative to the dominant text-extraction pipeline. The result challenges a foundational assumption in LLM pretraining and has implications for how future foundation models are trained on visually rich sources like PDFs and web pages.
Gazer: Training-free semantic correction for autoregressive visual models using MLLM feedback
Researchers introduce Gazer, a training-free framework that integrates multimodal large language model feedback into the sampling loop of autoregressive visual models (AVMs) to correct semantic errors during generation. The system operates in two stages: Reflective Diagnosis identifies semantic errors in intermediate generation states, and Semantic Correction rewinds and adjusts the generation trajectory to better match the target prompt. Experiments on compositional image and video benchmarks show improved semantic alignment and compositional accuracy across multiple AVMs without additional training. The work addresses a known weakness of next-scale prediction AVMs, where semantic errors accumulate across discrete generation scales.