megatron-lm-eb59f34c·4 events·first seen Aliases: Megatron-LM
OpenAI's GPT-5.6 models are set for broader API release following a Department of Commerce-approved safety review that delayed launch for weeks; GPT-5.6 Sol Ultra scores 91.9% on TerminalBench 2.1 versus Claude Mythos 5 at 88%, with pricing roughly half of Anthropic's comparable tier. Microsoft is actively replacing OpenAI and Anthropic models in Excel, Outlook, and Teams with its internally built MAI models to reduce third-party dependency as its OpenAI discount partnership nears expiration. Anthropic expanded Claude Cowork to web and mobile for Max plan subscribers, with usage data from 1.2 million sessions showing over 90% of use is non-developer work. Nvidia released Audex, a 30B MoE audio-text model that avoids the typical 'text tax' of multimodal models, shipping under a noncommercial license.
Megatron-Bridge is an NVIDIA NeMo training library for Megatron-based models that supports bidirectional conversion between Megatron and Hugging Face formats. The repository has accumulated 670 stars with modest daily growth (+5). It addresses a practical interoperability gap between the high-performance Megatron training stack and the broader HuggingFace ecosystem.
This Hugging Face blog post details the infrastructure and training methodology used to train BLOOM, a 176-billion parameter open-access multilingual language model. It covers the use of Megatron-DeepSpeed for distributed training across hundreds of GPUs, including tensor parallelism, pipeline parallelism, and data parallelism strategies. The post also discusses hardware setup, memory optimization techniques, and lessons learned during the large-scale training run.
The paper introduces Runtime-Readiness-First Pipeline (RRFP), a new runtime for pipeline-parallel large-model training that treats schedules as non-binding hint orders rather than strict execution sequences. By combining message-driven asynchronous communication, lightweight tensor-parallel coordination, and ready-set arbitration, RRFP dynamically dispatches work based on actual task readiness, reducing idle bubbles and stage misalignment. Implemented on a Megatron-based framework and evaluated at up to 128 GPUs, RRFP achieves up to 1.77× speedup on language-only workloads and 2.77× on multimodal workloads versus fixed-order baselines, and outperforms the fastest comparable external system by up to 1.84×.