UltraX

UltraX: Adaptive programmatic framework for large-scale LLM pre-training data refinement

UltraX is a function-calling data refinement framework for large-scale LLM pre-training corpora that extends prior rule-based and LLM-based approaches by introducing insertion alongside deletion and modification for fine-grained instance-level editing. The system builds a program-supervision pipeline using dataset-adaptive prompt optimization, Line Alignment Mapping, and Dynamic Context Replacement to convert raw text pairs into structured editing programs. Experiments show UltraX achieves the highest average performance across evaluated corpora and matches or surpasses baselines with fewer training tokens, suggesting improved data efficiency. The work addresses the diminishing returns of data scaling by focusing on data quality rather than quantity.