Entity · paper

Requential Coding: Pushing the Limits of Model Compression with Self-Generated Training Data

paperactiveprovisionalrequential-coding-pushing-the-limits-of-model-compression-with-self-generated-training-data-abd4e5a0·1 events·first seen 25h ago

Aliases: Requential Coding: Pushing the Limits of Model Compression with Self-Generated Training Data

Co-occurring entities

PAC-Bayes requential coding

More like this (12)

Adaptive Multi-Resolution Procedural Knowledge Compression for Large Language Models chain-of-thought training data generation Self-Supervised Pretraining quantization-aware training Braun et al. 2025 Compressed Computation Beyond Uniform Tokens: Adaptive Compression for Time Series Language Models Force-Informed Re-Sampling Training Sparse Autoencoders Provenance-Grounded Gating and Adaptive Recovery in Synthetic Post-Training Data Curation Context-Driven Incremental Compression self-training End-to-End Context Compression at Scale

Recent events (1)

7arXiv · cs.LG·25h ago·source ↗

Requential coding: self-generated training data enables orders-of-magnitude model compression and tighter generalization bounds

A new arXiv preprint introduces requential coding, a compression scheme where a teacher model selects training samples from the student model's own distribution, so the code records only teacher-student disagreements rather than the full data sequence. The resulting code lengths are independent of both parameter count and data entropy, yielding codes orders of magnitude shorter than prequential coding baselines. Plugged into a PAC-Bayes framework, the method produces state-of-the-art generalization guarantees for billion-parameter LLMs that tighten with scale in the compute-optimal regime, and reveals that larger models and ensembles become increasingly compressible relative to dataset size.

Frontier Model Releases Evaluation and Benchmarking Requential Coding: Pushing the Limits of Model Compression with Self-Generated Training Data PAC-Bayes requential coding