Finetuning olmOCR to be a faithful OCR-Engine
TNG Technology Consulting describes a fine-tuning approach applied to olmOCR, a vision-language model designed for document OCR tasks, to improve its faithfulness and reduce hallucinations. The post covers dataset construction, training methodology, and evaluation results showing improved accuracy on document extraction benchmarks. This represents a practical community contribution to the open-weights document-understanding ecosystem.
Related guides (4)
Related events (8)
Fine-tuning Florence-2 - Microsoft's Cutting-edge Vision Language Models
This Hugging Face blog post provides a technical guide for fine-tuning Microsoft's Florence-2 vision-language models. Florence-2 is a compact yet capable multimodal model supporting tasks like captioning, object detection, and OCR. The post covers practical implementation details for adapting the model to custom datasets using the Hugging Face ecosystem.
GLM-OCR: Fast and Accurate OCR System from zai-org
GLM-OCR is an open-source OCR project from zai-org built on the GLM model family, positioning itself as accurate, fast, and comprehensive. The repository has accumulated 6,787 GitHub stars with 82 added today, indicating notable community traction. It represents an application of large language/vision models to document understanding and text recognition tasks.
PaddleOCR 3.5: Running OCR and Document Parsing Tasks with a Transformers Backend
PaddleOCR 3.5 introduces support for running OCR and document parsing pipelines using a Hugging Face Transformers backend, enabling integration with the broader Transformers ecosystem. The update allows users to leverage transformer-based models for optical character recognition and structured document understanding tasks. This represents a convergence between the PaddlePaddle framework and the Transformers library for document AI workloads.
Fine-tuning LLMs on summary-expansion tasks strips copyright alignment guardrails, enabling up to 92% verbatim book reproduction
Researchers from Stony Brook University, Carnegie Mellon University, and Columbia Law School fine-tuned DeepSeek-V3.1, Gemini 2.5 Pro, and GPT-4o on a task of expanding plot summaries into prose paragraphs, finding that this caused models to regurgitate up to 91.9% of verbatim text from books in their pretraining data. The key finding is that alignment training suppresses but does not erase memorized text strings from model weights, and fine-tuning on verbatim-generation tasks can re-enable that recall, bypassing system-prompt-level copyright guardrails. The result has direct implications for model providers offering fine-tuning APIs and for organizations deploying customized models, as anti-plagiarism guardrails cannot be assumed to survive downstream fine-tuning.
Introducing vision to the fine-tuning API
OpenAI has extended its fine-tuning API to support multimodal inputs, allowing developers to fine-tune GPT-4o using both images and text. This enables customization of vision capabilities for domain-specific tasks. The update expands the existing text-only fine-tuning pipeline to handle image-text pairs.
Fine-Tune Whisper For Multilingual ASR with 🤗 Transformers
This Hugging Face blog post provides a practical guide for fine-tuning OpenAI's Whisper model for multilingual automatic speech recognition using the Transformers library. It covers dataset preparation, training configuration, and evaluation using the Word Error Rate metric. The post targets practitioners seeking to adapt Whisper to low-resource or domain-specific languages.
Introducing TextImage Augmentation for Document Images
Hugging Face introduces a TextImage augmentation library for document images, aimed at improving model robustness for document understanding tasks. The tooling applies transformations such as noise, blur, and distortion to document images to simulate real-world scanning and printing artifacts. This is relevant to training and fine-tuning vision-language models on document datasets.
FINO: Label-free adaptation of vision foundation models using metadata in scientific domains
Researchers propose FINO, a self-supervised method for adapting vision foundation models to specialized scientific domains without task labels, using metadata as a guidance signal instead. The approach combines a standard self-supervised objective with flexible handling of both discrete and continuous metadata to preserve informative factors while suppressing spurious ones. Evaluated across subcellular fluorescence microscopy, Earth observation, wildlife monitoring, and medical imaging, FINO outperforms both unsupervised domain adaptation and fully supervised fine-tuning, including domain-specific state-of-the-art models.