technique
temporally ordered pre-training
techniqueactive
temporally-ordered-pre-training-3f9b6d8e·1 events·first seen 26d agoAliases: temporally ordered pre-training
Co-occurring entities
More like this (12)
multimodal pretrainingSelf-Supervised PretrainingContrastive Pre-traininginstruction-based multitask pretrainingtest-time trainingPretraining Recurrent Networks without Recurrenceself-trainingcontrastive vision-language pretrainingq0: Primitives for Hyper-Epoch PretrainingVideo PreTraining (VPT)Supervised Memory TrainingUnsupervised Pre-training
Recent events (1)
Temporally Ordered Pre-training Improves LLM Factual Freshness (Kairos)
Researchers from Kyutai pre-train 6B-parameter models on temporally ordered Common Crawl snapshots and compare them against standard shuffled pre-training baselines. They introduce a benchmark of over 7,000 temporally grounded questions to evaluate whether models correctly associate facts with their corresponding time periods. Results show sequentially trained models match shuffled baselines on general language understanding while exhibiting more up-to-date and temporally precise factual knowledge. Code, checkpoints, and datasets are released under the Kairos project.