Entity · model
ViT (Vision Transformer)
modelactive
vit-vision-transformer--f25a37b0·6 events·first seen Aliases: ViT (Vision Transformer), Vision Transformers (ViTs), Vision Transformer (ViT), Vision Transformer
Co-occurring entities
More like this (12)
Recent events (6)
Patch Policy: Lightweight transformer policy using dense ViT patch tokens for robot control
Patch Policy is a new robot learning architecture that enables transformer-based policies to consume dense pre-trained Vision Transformer patch tokens directly, without the overhead of a full vision-language model backbone. A block-causal attention mask preserves temporal causality while allowing the model to attend over many patch tokens per observation. Across four simulated and three real-world environment suites, the method achieves a 40% relative improvement over global-pooled representation baselines and outperforms fine-tuned OpenVLA-OFT by 18% while using roughly 0.7% of its parameters. The work addresses a practical gap between lightweight robot policies and expensive VLA models.
Training-free mechanistic defense against typographic attacks on CLIP-based vision encoders
Researchers propose a training-free method to defend CLIP-based vision encoders against typographic attacks, where irrelevant text embedded in images biases visual representations toward lexical rather than semantic meaning. The approach uses sampling-based mechanistic interpretability to identify specific Vision Transformer attention heads responsible for encoding lexical information, then applies targeted circuit-level interventions to suppress this behavior. Without any retraining, the method outperforms both supervised and training-free baselines on object classification and improves Visual Question Answering accuracy under typographic attack conditions on RIO-Bench across several state-of-the-art LVLMs.
HiReLC: Hierarchical Reinforcement Learning Framework for Joint Neural Network Pruning and Quantization
Researchers introduce HiReLC, a hierarchical ensemble-RL framework that automates joint quantization and structured pruning of deep neural networks. The system uses two-level agents — low-level agents selecting per-kernel compression configurations and high-level agents coordinating global budget allocation via Fisher Information-based sensitivity estimates. Experiments on Vision Transformers and CNNs achieve 5.99–6.72× parameter-storage compression with accuracy drops of 0.55–5.62% in most settings. The controller is architecture-agnostic, using a surrogate MLP and active learning loop to reduce policy evaluation cost.
Information-theoretic formalization of the binding problem in Vision Transformers
Researchers introduce a formal information-theoretic framework for the binding problem — the challenge of associating features (color, shape) with the correct objects in multi-object scenes. They develop a probing method to measure binding information in model representations and apply it to several pre-trained Vision Transformers, examining components like the [CLS] token and spatial tokens across datasets with feature sharing, occlusion, and natural features. Results position binding information as a key factor in visual recognition and reasoning quality, and suggest current ViT architectures have limited binding capability, consistent with known failure modes.
OrpQuant: Geometric Orthogonal Residual Projection for Multiplier-Free Power-of-Two Transformer Quantization
This paper introduces Orthogonal Residual Projection (ORP), an algorithm-hardware co-design framework for ultra-low-bit quantization of LLMs and Vision Transformers targeting edge deployment. ORP addresses the structural limitations of Power-of-Two (PoT) quantization by formulating quantization as a dual-basis geometric projection that synthesizes higher-resolution residual lattices using only shift-and-add operations, eliminating multipliers. At 3-bit (W3/A16), ORP achieves 6.10 perplexity on LLaMA-2-7B, competitive with MAC-intensive baselines like AWQ, while reducing full-model calibration time to ~15 minutes. RTL synthesis at 28nm confirms hardware efficiency by mitigating timing bottlenecks from dense multiplier trees.
New ViT and ALIGN Models From Kakao Brain
Kakao Brain released new Vision Transformer (ViT) and ALIGN models, announced via the Hugging Face blog. The post covers multimodal vision-language models contributed to the open ecosystem. These models expand the available open-weights options for image-text tasks.