4arXiv cs.LG (Machine Learning)·
MixFrag: Fragility-guided mixed-precision post-training quantization for Vision Transformers
MixFrag is a new post-training quantization (PTQ) framework for Vision Transformers that assigns mixed bit-widths per layer based on quantization fragility, measured via KL divergence between full-precision and quantized output distributions. Bit allocation is formulated as a Multiple-Choice Knapsack Problem to optimize precision under a target bit budget. Evaluated on ImageNet-1K classification and COCO detection/segmentation, MixFrag claims state-of-the-art among mixed-precision PTQ methods, improving the prior best by up to 9.6 AP on a challenging low-bit setting.
Related guides (2)
Related events (8)
OrbitQuant: Data-agnostic quantization for image and video diffusion transformers
OrbitQuant is a new post-training quantization method for diffusion transformers that avoids the need for calibration data by quantizing activations in a normalized, rotated basis using a randomized permuted block-Hadamard rotation. A single Lloyd-Max codebook covers all timesteps, prompts, and layers for a given input dimension, and the same recipe transfers from image to video models without per-modality tuning. The method is evaluated on FLUX.1, Z-Image-Turbo, Wan 2.1, and CogVideoX, claiming state-of-the-art PTQ results at several low-bit settings including W2A4 for image DiTs.
Memory-efficient Diffusion Transformers with Quanto and Diffusers
This Hugging Face blog post describes integrating the Quanto quantization library with the Diffusers framework to reduce memory requirements for diffusion transformer models. The approach enables running large image/video generation models on consumer-grade hardware by applying int8 and int4 quantization to model weights. The post covers practical implementation details and benchmarks showing memory savings for models like Flux and others in the diffusion transformer family.
Scale vs. quantization tradeoffs for uncertainty signals in vision-language models under image degradation
A new arXiv paper evaluates how model scale and 4-bit quantization affect two confidence signals — verbalized confidence and mean token probability — in the Qwen2-VL family across 5,700 predictions under six photographic degradation types. Key findings: scaling from 2B to 7B sharply improves internal uncertainty (AUROC 0.80→0.98) while verbalized confidence remains weak; 4-bit quantization costs little in accuracy (-1.6 pts) but degrades the internal confidence signal (AUROC 0.95→0.80) and collapses verbalized-confidence parse rate from 99% to 64%. The practical recommendation is to prefer a larger quantized model over a smaller full-precision one within a fixed memory budget, and to use error-detection AUROC rather than calibration error as the primary metric.
Fine-tuning LLMs to 1.58bit: extreme quantization made easy
Hugging Face published a blog post describing a method for fine-tuning large language models down to 1.58-bit precision, referencing the BitNet b1.58 quantization scheme. The post covers tooling and workflows that make extreme quantization more accessible via the Hugging Face ecosystem. This represents a practical guide to applying ternary-weight quantization ({-1, 0, 1}) to existing models through fine-tuning rather than training from scratch.
Ω-QVLA: Training-Free W4A4 Quantization for Full Vision-Language-Action Models Including Diffusion Action Heads
Omega-QVLA is a post-training quantization framework that compresses both the LLM backbone and the diffusion-based action head of VLA models to uniform W4A4 precision without mixed-precision schemes or fine-tuning. It combines composite SVD-Hadamard rotation for weight energy equalization with per-step DiT activation scaling to handle dynamic-range drift across denoising steps. On the LIBERO benchmark, it achieves 98.0% and 87.8% task success on Pi 0.5 and GR00T N1.5 respectively—matching or exceeding FP16 baselines—while reducing static memory footprint by 71.3%. Real-world manipulation experiments confirm the approach generalizes beyond simulation.
Exploring Quantization Backends in Diffusers
Hugging Face published a technical overview of quantization backends available in the Diffusers library for image and video generation models. The post covers integration with multiple quantization frameworks (likely bitsandbytes, GGUF, torchao, and similar) and their trade-offs for diffusion model inference. It targets practitioners seeking to reduce memory footprint and improve throughput when deploying diffusion models.
Overview of Natively Supported Quantization Schemes in 🤗 Transformers
This Hugging Face blog post surveys the quantization methods natively integrated into the Transformers library as of September 2023, covering schemes such as GPTQ, bitsandbytes (LLM.int8, NF4), and related techniques. It explains how each method works, their trade-offs in terms of memory reduction and inference speed, and how practitioners can apply them via the Transformers API. The post serves as a practical reference for deploying large language models under memory constraints.
A Gentle Introduction to 8-bit Matrix Multiplication for Transformers at Scale using Hugging Face and bitsandbytes
This Hugging Face blog post introduces 8-bit quantization for large transformer models via integration of the bitsandbytes library with the transformers and accelerate libraries. It explains how LLM.int8() enables loading large models in 8-bit precision, significantly reducing GPU memory requirements without major accuracy degradation. The post covers the technical mechanics of mixed-precision decomposition and how practitioners can use the integration in practice.