Entity · product
SGLang
productactive
sglang-911c28db·6 events·first seen Aliases: SGLang
Co-occurring entities
More like this (12)
Recent events (6)
Windowed-MTP: Eliminating Full-Context KV Cache Overhead in Speculative Decoding at Million-Token Scale
A new arXiv preprint introduces Windowed-MTP, a training-free modification to Multi-Token-Prediction draft heads used in speculative decoding that applies a StreamingLLM-style sliding window with attention sinks to the draft's attention only. At 1M-token context, the method drops ~99% of draft KV cache entries while leaving the full-attention verifier intact, making the approach lossless by construction. Evaluated on Qwen GDN-MoE 35B/122B and a Mamba2-hybrid 120B in SGLang, it reduces per-decode-step cost by 28–44% over the native MTP draft baseline. The fix addresses a compounding problem where deep native draft heads can become net-negative at long contexts, especially under hybrid/linear-attention targets.
Anthropic Releases Claude Mythos Preview with Extraordinary Cybersecurity Capabilities, Forms Project Glasswing Consortium
Anthropic has published a 244-page model card for Claude Mythos Preview, a large language model not yet commercially available, which broadly outperforms Claude Opus 4.6 and is described as 'strikingly capable' at identifying and exploiting code vulnerabilities. To mitigate risks before potential release, Anthropic assembled Project Glasswing, a consortium including AWS, Apple, Google, Microsoft, CrowdStrike, Nvidia, and 40+ other organizations, funded with $100 million in API credits and $4 million in open-source security donations. This marks the first time Anthropic has published a model card without making the model commercially available, signaling an unusual safety-first deployment posture. The issue also includes commentary from Andrew Ng on AI's impact on software engineering jobs, arguing against an 'AI jobpocalypse' narrative.
Transformers Backend Integration in SGLang
Hugging Face has announced an integration that allows SGLang, a high-performance LLM serving framework, to use the Transformers library as a backend. This enables models supported by Transformers to be served through SGLang's inference engine, combining SGLang's optimized serving capabilities with the broad model coverage of the Transformers ecosystem. The integration lowers the barrier for deploying a wide range of models with production-grade inference infrastructure.
Data Points: Thinking Machines Interaction Model, ERNIE 5.1, Co-Mathematician, RL Conductor, and More
This edition of The Batch covers five notable AI developments: Thinking Machines' research preview of an 'interaction model' with a 200ms micro-turn multimodal architecture; Baidu's ERNIE 5.1, a compressed derivative of ERNIE 5.0 using only 6% of typical pre-training compute; Google DeepMind's Co-Mathematician collaborative workbench reaching 48% on FrontierMath Tier 4; a 7B RL Conductor model that orchestrates multi-agent workflows via reinforcement learning; and Google's Magic Pointer cursor system powered by Gemini. Secondary items include GitHub Copilot pricing restructuring ahead of usage-based billing.
Mistral Releases Mistral 3 Family: Mistral Large 3 (675B MoE) and Ministral 3 Series (3B–14B), All Apache 2.0
Mistral AI has announced Mistral 3, a family of open-weight models including Mistral Large 3 (41B active / 675B total sparse MoE) and three dense Ministral 3 edge models (3B, 8B, 14B), all released under Apache 2.0. Mistral Large 3 debuts at #2 on LMArena's OSS non-reasoning leaderboard, supports image understanding, and was trained on 3,000 NVIDIA H200 GPUs; a reasoning variant is forthcoming. The Ministral 3 series includes base, instruct, and reasoning variants with multimodal and multilingual capabilities, with the 14B reasoning model achieving 85% on AIME '25. The release involves deep co-optimization with NVIDIA (Blackwell/Hopper kernels, NVFP4 format), vLLM, and Red Hat, and is available across major cloud and inference platforms.
Mistral Small 4: Unified Multimodal, Reasoning, and Coding MoE Model Released Under Apache 2.0
Mistral AI has released Mistral Small 4, a 119B-parameter Mixture-of-Experts model (6B active per token) that unifies capabilities previously split across Magistral (reasoning), Pixtral (multimodal), and Devstral (coding agents) into a single open-weights model. The model features a 256k context window, configurable reasoning effort via a `reasoning_effort` parameter, native text and image input support, and is released under Apache 2.0. Mistral claims 40% latency reduction and 3x throughput improvement over Mistral Small 3, with benchmark results showing competitive performance against GPT-OSS 120B and Qwen models while producing significantly shorter outputs. The release includes day-0 availability as an NVIDIA NIM and support across vLLM, llama.cpp, SGLang, and Transformers.