Entity · technique

CausalMix

techniqueactiveprovisionalcausalmix-1f3cf88d·1 events·first seen 6d ago

Aliases: CausalMix

Co-occurring entities

More like this (12)

RegMix causal discovery causal generative modeling Mixture-of-Agents causal transformer CausaLab Mixtral Data Mixture Surgery KletterMix Structural Causal Model MetaSyn mixture-density networks

Recent events (1)

5arXiv · cs.CL·6d ago·source ↗

CausalMix frames LLM data mixture optimization as causal inference to generalize across data pool shifts

CausalMix proposes treating data mixture optimization for LLM training as a causal inference problem, using Conditional Average Treatment Effect (CATE) estimation to infer optimal domain mixtures without costly retraining when the data pool changes. The method fits a causal model on 512 runs of Qwen2.5-0.5B and extrapolates the resulting mixture to train a 7B model, also generalizing to long chain-of-thought data on Qwen3-4B-Base. It outperforms RegMix and other baselines across multiple downstream tasks while providing interpretable visual analysis of mixing strategies via a CATE Interpreter. The approach addresses a practical scalability limitation in existing proxy-model-based mixture methods.

Training Infrastructure Evaluation and Benchmarking RegMix CausalMix Qwen3-4B-Base +1 more