dataorchestra-13f8674f·1 events·first seen Aliases: DataOrchestra
DataOrchestra is a proposed framework that replaces fixed corpus-level data processing strategies with an example-specific orchestration pipeline for LLM pretraining. An orchestrator model decides per data chunk whether to drop, leave unchanged, or clean it, selecting from programmatic editing or LLM-based rewriting with generated instructions. Models pretrained from 0.5B to 7B on DataOrchestra-processed web data show consistent gains across 11 benchmarks over individual processing methods, with additional benefits for math continued pretraining and reduced compute from skipping unnecessary operations.