Temporally Ordered Pre-training Improves LLM Factual Freshness (Kairos)
Researchers from Kyutai pre-train 6B-parameter models on temporally ordered Common Crawl snapshots and compare them against standard shuffled pre-training baselines. They introduce a benchmark of over 7,000 temporally grounded questions to evaluate whether models correctly associate facts with their corresponding time periods. Results show sequentially trained models match shuffled baselines on general language understanding while exhibiting more up-to-date and temporally precise factual knowledge. Code, checkpoints, and datasets are released under the Kairos project.
Related guides (3)
Related events (8)
Act2Answer: Benchmarking commonsense and world knowledge retention in Vision-Language-Action models
Researchers introduce Act2Answer, a protocol for evaluating how much commonsense and factual knowledge VLA models retain after fine-tuning on robotics data. The approach converts knowledge benchmark questions into tabletop object-placement episodes, yielding action-grounded success rates that reduce confounds from low-level control failures. A large-scale study of 7 VLA models and 9 VLM baselines finds that VLAs retain solid performance on simple concepts but show larger gaps on richer semantic categories compared to their source VLMs, and that VQA co-training is associated with better knowledge retention.
QK-Restore: Fixing long-context recall degradation caused by CoT fine-tuning in hybrid LLMs
Researchers find that chain-of-thought supervised fine-tuning systematically degrades long-context recall in hybrid linear-attention models (HypeNet, Jet-Nemotron), with Needle-In-A-Haystack performance collapsing dramatically—e.g., HypeNet-9B dropping from 67.2% to 9.4% at 256K context. The root cause is identified as CoT-SFT biasing attention gradients toward short-range patterns, corrupting the query-key projections responsible for long-range routing. The paper proposes QK-Restore, a training-free fix that restores only W_Q and W_K from the pre-SFT checkpoint, recovering long-context capability while preserving reasoning gains.
Multi-Task Bayesian In-Context Learning for Amortized Hierarchical Inference
A new arXiv preprint introduces a multi-task in-context learning framework for amortized hierarchical Bayesian predictive inference, representing prior information as a prefix of in-context datasets fed to a transformer. The model learns to adapt predictions across families of priors, addressing the brittleness of prior-data fitted models under distribution shift. On evaluations including out-of-meta-distribution priors and high-dimensional latent structures, the method matches oracle Bayesian predictors while being orders of magnitude faster, with a real-world spatiotemporal temperature prediction demonstration.
Demystifying Data Organization for Enhanced LLM Training
This Microsoft Research paper systematically investigates how data organization—distinct from data selection—affects LLM training efficiency across pre-training and SFT stages. The authors formalize four guidelines (Boundary Sharpening, Cyclic Scheduling, Curriculum Continuity, and Local Diversity) and introduce two novel data ordering methods, STR and SAW, that reuse pre-computed sample-level scores with minimal additional overhead. Experiments across multiple model scales and dataset sizes demonstrate improved training stability and performance, with code released publicly.
FACTS Benchmark Suite: Systematically evaluating the factuality of large language models
DeepMind has released the FACTS Benchmark Suite, a systematic evaluation framework for measuring the factuality of large language models. The benchmark is designed to assess how accurately LLMs produce factually grounded outputs. This represents a structured contribution to the growing field of LLM evaluation, specifically targeting hallucination and factual reliability. The announcement comes from a Tier 1 lab, lending it credibility as a reference benchmark in the field.
ContextRL: Context-aware reinforcement learning improves grounding in agentic and multimodal LLMs
Researchers introduce ContextRL, a reinforcement learning method that trains LLMs to select the context that supports a given query-answer pair from two highly similar candidates, rather than supervising only final answers. The approach constructs contrastive context pairs in two domains: coding agent trajectories (1k pairs) and multimodal image pairs (7k pairs). ContextRL achieves +2.2% average gains over standard GRPO on 5 long-horizon benchmarks and +1.8% across 12 visual QA benchmarks, with ablations showing the gains stem from the context-selection objective rather than the contrastive data alone.
q0: Hyper-Epoch Pretraining turns multi-epoch budgets into diverse model populations for better generalization
A new arXiv preprint introduces hyper-epoch pretraining (q0), a framework that reframes multi-epoch training as exploration of a model population rather than refinement of a single model. The approach uses three primitives—cyclic schedules with anti-correlated learning rate and weight decay, chain distillation, and a learned prior for inference-time weighting—to achieve lower validation loss than single-model training. On a 1.8B-parameter model trained on FineWeb, q0 matches a 256-epoch ensemble baseline using only ~56 epochs (~4.6× fewer), with cumulative ~12.9× data efficiency under the Slowrun setting. The work directly addresses the emerging regime where compute scales faster than high-quality data supply.
ChronoMedKG: Temporally-Grounded Biomedical Knowledge Graph and Benchmark for Clinical Reasoning
ChronoMedKG is a new biomedical knowledge graph containing 460,497 evidence-linked triples across 13,431 diseases, each annotated with temporal components such as onset window and progression stage. It is constructed via a multi-agent pipeline using multiple frontier LLMs extracting from PubMed/PMC, with multi-model consensus and credibility filtering. The accompanying ChronoTQA benchmark (3,341 questions) reveals frontier LLMs lose ~30 points on temporal vs. static clinical questions, while ChronoMedKG-based retrieval recovers 47–65% of long-tail failures compared to 17–29% for HPOA-RAG. The work addresses a significant gap in existing KGs (PrimeKG, Hetionet, iKraph) that treat disease associations as static facts.