Group-in-Group Policy Optimization

Muon optimizer shows large gains over AdamW in sparse-reward agentic RL on ALFWorld

A new arXiv preprint investigates the Muon optimizer for reinforcement learning post-training of language model agents, comparing it to AdamW on the ALFWorld benchmark using Qwen2.5-0.5B-Instruct. Under Group-in-Group Policy Optimization (GiGPO), applying Muon to hidden weight matrices raises validation success from 0.290 to 0.546 (+88%), with further gains at lower learning rates reaching 0.901 success. The results are exploratory (single-seed, single-task) but suggest that optimizer choice, advantage estimator, and learning rate interact significantly in agentic RL settings.