5arXiv cs.CL (Computation and Language)·
DataOrchestra: Per-example adaptive curation pipeline for LLM pretraining data
DataOrchestra is a proposed framework that replaces fixed corpus-level data processing strategies with an example-specific orchestration pipeline for LLM pretraining. An orchestrator model decides per data chunk whether to drop, leave unchanged, or clean it, selecting from programmatic editing or LLM-based rewriting with generated instructions. Models pretrained from 0.5B to 7B on DataOrchestra-processed web data show consistent gains across 11 benchmarks over individual processing methods, with additional benefits for math continued pretraining and reduced compute from skipping unnecessary operations.
Related guides (3)
Related events (8)
Demystifying Data Organization for Enhanced LLM Training
This Microsoft Research paper systematically investigates how data organization—distinct from data selection—affects LLM training efficiency across pre-training and SFT stages. The authors formalize four guidelines (Boundary Sharpening, Cyclic Scheduling, Curriculum Continuity, and Local Diversity) and introduce two novel data ordering methods, STR and SAW, that reuse pre-computed sample-level scores with minimal additional overhead. Experiments across multiple model scales and dataset sizes demonstrate improved training stability and performance, with code released publicly.
VADAOrchestra: Neurosymbolic framework combining LLM orchestration with Datalog+/- symbolic reasoning for adaptive workflows
VADAOrchestra is a neurosymbolic framework that hybridizes LLM-based orchestration with a symbolic Datalog+/- inference engine to model complex, adaptive workflows. An LLM incrementally plans and encodes workflow steps as logic programs, while a dedicated symbolic engine handles all inference, decoupling orchestration from execution. The approach targets auditability, scalability over large datasets, and explainability — limitations of pure LLM-agent architectures — and is evaluated on real-world financial use cases. The work positions itself as a bridge between traditional Business Process Management rigidity and LLM agent flexibility.
OpenDCAI/DataFlow: LLM-based data preparation pipeline library gains traction on GitHub
DataFlow is an open-source Python library from OpenDCAI that provides LLM-based operators and pipelines for data preparation tasks. The repository has accumulated 6,971 stars with 139 added in a single day, indicating growing community interest. It targets the data curation and preprocessing workflow that underpins LLM training and fine-tuning pipelines.
UltraX: Adaptive programmatic framework for large-scale LLM pre-training data refinement
UltraX is a function-calling data refinement framework for large-scale LLM pre-training corpora that extends prior rule-based and LLM-based approaches by introducing insertion alongside deletion and modification for fine-grained instance-level editing. The system builds a program-supervision pipeline using dataset-adaptive prompt optimization, Line Alignment Mapping, and Dynamic Context Replacement to convert raw text pairs into structured editing programs. Experiments show UltraX achieves the highest average performance across evaluated corpora and matches or surpasses baselines with fewer training tokens, suggesting improved data efficiency. The work addresses the diminishing returns of data scaling by focusing on data quality rather than quantity.
LLMSurgeon: Post-Hoc Auditing of LLM Pretraining Data Mixtures
LLMSurgeon formalizes Data Mixture Surgery (DMS), a framework for estimating the domain-level distribution of an LLM's pretraining corpus using only generated text from the target model. The method casts DMS as an inverse problem under the label-shift assumption, using a calibrated soft confusion matrix to correct domain confusion and recover the latent mixture prior. The authors also introduce LLMScan, a verifiable evaluation suite built from open-source LLMs with known pretraining mixtures, on which LLMSurgeon demonstrates high-fidelity recovery of domain compositions without access to training data.
CurateEvo: Failure-Driven Dynamic Data Curation for Agentic LLM Post-Training
CurateEvo is a new framework for agentic post-training that treats data curation as a dynamic, evolving process rather than a fixed preprocessing step. The system represents curation strategies as executable code and iteratively rewrites them based on failed trajectories from a held-out development set, producing SFT data, RL data, and an inference-time memory bank. Evaluated on ACEBench-Agent, BFCL-V4, and τ²-Bench, CurateEvo outperforms prior curation methods by 3.2 and 2.7 average points in labeled and wild-data settings respectively, while also reducing curation overhead.
OrchRM: Self-supervised reward modeling for multi-agent orchestration without human annotations
Researchers propose Orchestration Reward Modeling (OrchRM), a self-supervised framework that trains reward models for LLM-based multi-agent orchestrators using intermediate execution artifacts to construct win-lose pairs for Bradley-Terry training. The approach avoids costly sub-agent rollouts by operating directly at the orchestration level, achieving up to 10x improvement in training token efficiency and up to 8% accuracy gains in test-time scaling. Results generalize across mathematical reasoning, web-based QA, and multi-hop reasoning tasks.
DEFINED: Data-efficient framework for fine-grained creativity assessment in debate using LLMs
DEFINED is a computational framework for automated creativity assessment in debate scenarios, operationalizing creativity through an eight-dimensional hierarchical metric system implemented via a pretrained autoregressive language model with a hierarchical scoring head. The system addresses data scarcity through constrained data augmentation and mixed-granularity training from limited expert-annotated data. It outperforms prompt-based LLM evaluators and existing debate scoring methods on authentic competition data. The work is relevant to AI evaluation methodology and the broader question of whether LLMs can reliably assess complex human cognitive outputs.