Entity · paper

Scalable Visual Pretraining for Language Intelligence

paperactiveprovisionalscalable-visual-pretraining-for-language-intelligence-d6dbe5ab·1 events·first seen 10h ago

Aliases: Scalable Visual Pretraining for Language Intelligence

More like this (12)

contrastive vision-language pretraining Self-Supervised Pretraining Contrastive Language-Image Pretraining (CLIP)Adaptive Multi-Resolution Procedural Knowledge Compression for Large Language Models multimodal pretraining Understanding Large Language Models Contrastive Pre-training On the Limits of Prompt-Conditioned Language Models as General-Purpose Learners temporally ordered pre-training Scaling Laws for Neural Language Models Unified Audio Intelligence Without Regressing on Text Intelligence Test-Time Scaling for Small VLMs on Multilingual Visual MCQ

Recent events (1)

6arXiv · cs.AI·10h ago·source ↗

Visual pretraining on document images outperforms text-only pretraining for language intelligence

A new arXiv preprint argues that training language models directly on visual representations of documents (figures, equations, page layouts) consistently outperforms text-only pretraining on the same underlying corpora. The authors conduct a systematic study of unsupervised visual pretraining paradigms across multiple backbones and benchmarks, framing visual pretraining as a scalable alternative to the dominant text-extraction pipeline. The result challenges a foundational assumption in LLM pretraining and has implications for how future foundation models are trained on visually rich sources like PDFs and web pages.

Frontier Model Releases Multimodal Progress Scalable Visual Pretraining for Language Intelligence