4arXiv cs.CL (Computation and Language)·
Survey on instruction-based image editing: taxonomy, benchmarks, and CDD-IIE Bench
A new arXiv survey systematically reviews instruction-based image editing (IIE) research across five dimensions: task taxonomy, training data construction, architectural evolution (GAN to diffusion to autoregressive), evaluation metrics, and commercial solutions. The authors introduce CDD-IIE Bench, a new diagnostic benchmark designed to rigorously assess multiple aspects of IIE model performance. The survey covers the role of LLMs and VLMs in enabling practical one-sentence image editing systems and compares open-source solutions empirically.
Related guides (2)
Related events (8)
AUDITS: A Comprehensive Benchmark for Image Manipulation Localization Across Multiple Analysis Axes
Researchers introduce AUDITS (Analysis Under Domain-shifts, qualIty, Type, and Size), a benchmark of over 530K images designed to evaluate image manipulation detection across multiple axes including domain shift, manipulation type, and size. The dataset draws from user and news photos and incorporates recent diffusion-based inpaintings. Experiments assess the robustness of existing manipulation detection methods under various domain shifts, aiming to advance development of more generalizable detection approaches.
Instruction-tuning Stable Diffusion with InstructPix2Pix
This Hugging Face blog post describes a methodology for instruction-tuning Stable Diffusion using the InstructPix2Pix framework, enabling image editing via natural language instructions. The approach adapts techniques from language model instruction-tuning to the image generation domain. The post covers dataset construction, training procedures, and evaluation of the resulting models.
ETCHR: Decoupled Image Editing for Visual Chain-of-Thought Reasoning in MLLMs
ETCHR introduces a question-conditioned, reasoning-aware image editing model that decouples visual transformation from downstream understanding in multimodal LLMs. It addresses two identified gaps—language-side (mapping abstract questions to visual edits) and generation-side (edit quality degrading with reasoning depth)—via a two-stage training recipe combining supervised fine-tuning on edit trajectories and VLM-derived reward signals. Because the editor is decoupled, it plugs into arbitrary MLLMs without retraining, yielding Pass@1 gains of roughly +4.6 to +5.5 points across five task families when paired with Qwen3-VL-8B, Gemini-3.1-Flash-Lite, and Kimi K2.5. The work advances the 'think with images' paradigm beyond fixed toolkits and unified multimodal approaches.
BitResEdit: Training-free bitwise residual editing for visual autoregressive image generators
BitResEdit is a training-free text-guided image editing method for bitwise-residual visual autoregressive (VAR) models, specifically targeting Infinity-2B. The approach combines per-bit Bernoulli guidance (BitEdit) with scale-aware code residual injection (ResEdit), exploiting native structures of VAR models that prior editors leave unused. On PIE-Bench with Infinity-2B, it achieves the best CLIP text alignment among same-backbone VAR editors (+1.07 over the prior best) while maintaining competitive background preservation.
SciDiagramEdit: Benchmark and agentic framework for editing scientific figures from natural-language instructions
Researchers introduce SciDiagramEdit, a benchmark and skill-evolution framework for automating edits to scientific figures under natural-language instructions. The benchmark mines before/after figure pairs from arXiv version histories, grounding training in authors' own revision intent. An agentic proposer iteratively refines the agent's skill specification from execution traces, progressively improving edit accuracy on held-out validation. The work demonstrates that natural paper revisions are an effective training signal for instruction-driven figure editing.
MMAE: First comprehensive benchmark for instruction-based audio editing across 7 modalities
Researchers introduce MMAE, a 2,000-sample benchmark for evaluating general-purpose instruction-based audio editing systems, covering 7 audio modalities (sound, speech, music, and mixtures) and 6 levels of task complexity. The benchmark uses a rubric-based evaluation framework decomposing tasks into 17,741 verifiable criteria to assess instruction following and context consistency. Evaluation of leading models reveals severe limitations: Exact Match Rate falls below 5% overall and hits 0% on complex mixed-modality tasks, exposing fundamental gaps in current audio editing systems.
Domain-generalized framework for pixel-level image tampering detection in VLM-generated content
Researchers propose a training framework for detecting pixel-level image tampering in content generated by modern vision-language models such as ChatGPT, Gemini, and Qwen-Image. The approach combines balanced minibatch sampling and a late-injection strategy to improve out-of-distribution robustness without overfitting to limited new-domain data. The framework outperforms the prior state-of-the-art PIXAR method by 26.1% and 26.8% relative improvement in gIoU and cIoU respectively, evaluated on OOD VLMs including GPT-Images-2.0, Gemini-3.1, FLUX.2, and Seedream 4.5. The work is directly motivated by the growing difficulty of detecting AI-generated image manipulations as generation quality improves.
TEVI: Sparse autoencoders for text-conditioned editing of CLIP image embeddings to improve vision-language alignment
TEVI is a framework that uses sparse autoencoders to disentangle CLIP image embeddings and a learned masking module to selectively reconstruct embeddings conditioned on a given caption, addressing the information imbalance between images and their captions. The approach improves image-text retrieval on both coarse-grained benchmarks (MS COCO, Flickr) and fine-grained long-caption benchmarks (IIW, DOCCI), with larger gains on richer captions. The work also shows improved robustness on the RoCOCO benchmark.