Entity · paper

Scaling Native Multimodal Pre-Training From Scratch

paperactiveprovisionalscaling-native-multimodal-pre-training-from-scratch-d51a84fd·1 events·first seen 3d ago

Aliases: Scaling Native Multimodal Pre-Training From Scratch

More like this (12)

multimodal pretraining Scalable Visual Pretraining for Language Intelligence Self-Supervised Pretraining Multimodal Learning Co-Learning for Missing Arbitrary Modalities in Multi-modal Classification multimodal classification models Latent World Recovery for Multimodal Learning with Missing Modalities temporally ordered pre-training Benchmarking Multimodal Large Language Models for Scientific Visualization Literacy quantization-aware training Multimodal Continual Instruction Tuning One-Step Gradient Delay is Not a Barrier for Large-Scale Asynchronous Pipeline Parallel LLM Pretraining

Recent events (1)

6arXiv · cs.CL·3d ago·source ↗

Scaling laws for native multimodal pre-training from scratch characterized empirically

A new arXiv preprint investigates compute-optimal scaling for transformer-based vision-language models trained natively on multimodal inputs from scratch, rather than via late-fusion of separately pre-trained components. The authors derive power-law relationships between compute budget, model size, and token count, finding that language and multimodal objectives exhibit distinct scaling behaviors and that data mixture composition strongly influences optimal resource allocation. They also report positive cross-modal transfer effects, including improved pure-text spatial reasoning. The work establishes an efficiency frontier for configuring model size, token count, and data mixture under fixed compute.

Training Infrastructure Frontier Model Releases Scaling Native Multimodal Pre-Training From Scratch +1 more