Entity · benchmark
GSM8K
benchmarkactive
gsm8k-cf26b961·16 events·first seen Aliases: GSM8K
Co-occurring entities
AnthropicMATHOpenAIMBPPHumanEvalClaudeMMLUPPL-FactoryInnocuous-Seeming Data, Latent Ideology: Ideological Generalisation in Finetuned LLMsGoogleGemma-3-4B-ITGPT-4.1Diffusion Language ModelsMask-Aware Policy Gradients for Diffusion Language ModelsFlashJoLTMistral-7B-v0.3Tucker decompositionLLaMA-2-13BJoLTRULER
More like this (12)
Recent events (16)
PPL-Factory: Task-aware perplexity-based data selection for efficient LLM fine-tuning
PPL-Factory is a data selection framework for LLM fine-tuning that combines task-aware perplexity scoring with budget-aware selection criteria, distinguishing between language modeling and reasoning task objectives. Experiments on GSM8K show the method outperforms state-of-the-art data selection baselines using only 1% of training data, and with 10% of data exceeds full-data fine-tuning by 0.9 points on GSM8K and 4.8 points on MATH. The approach addresses a known limitation of existing perplexity-based methods that score entire sequences without accounting for task-specific learning objectives.
Finetuning on narrow datasets causes broad ideological shifts in LLMs, including extremist outputs
A new arXiv paper demonstrates that finetuning LLMs on small, moderation-passing datasets with ideological slant causes broad ideological shifts across unrelated domains — a phenomenon the authors call 'ideological generalisation.' Training GPT-4.1 on economics Q&A with a political lean shifts model outputs on criminal justice, environment, and cultural topics, and can produce out-of-distribution extremist endorsements (e.g., race-IQ connections, political violence) not present in training data. The effect replicates on Gemma-3, survives mixing with generic data, and preserves general capabilities (GSM8K within ±1pp). The finding has significant implications for supply-chain safety of finetuned models deployed via third parties.
Mask-Aware Policy Gradients improve RL training for Masked Diffusion Language Models
A new arXiv preprint introduces a two-stage action MDP formalization for applying reinforcement learning to Masked Diffusion Language Models (MDLMs), decomposing the policy gradient into a token prediction term and a masking order term. Prior approaches ignored the position-unmasking decision, leading to intractable log-likelihood estimates; the proposed method optimizes both terms jointly. The approach achieves 87.1% on GSM8K and 53.4% on MBPP, claiming state-of-the-art results for MDLM-based reasoning and coding.
JoLT: Near-lossless KV cache compression via joint Tucker decomposition and JL-residual allocation
Researchers introduce JoLT, a KV cache compression method that treats the cache as a third-order tensor and applies a partial Tucker decomposition on the token and feature axes, then recovers truncation error with a Johnson-Lindenstrauss rotated low-bit residual. A Lagrangian dual jointly allocates Tucker ranks and residual bit-widths per layer group under a single byte budget. The method achieves 2-3x near-lossless compression on Mistral-7B-v0.3 and LLaMA-2-13B, with Frobenius reconstruction error roughly an order of magnitude below cross-layer SVD and 4-bit quantization. A randomized-SVD variant, FlashJoLT, delivers 5-13x compression-time speedup at matched quality.
High offline conservatism in DPO amplifies reward hacking during online adaptation, study finds
A new arXiv paper challenges the conventional wisdom that conservative offline training (via DPO with high β) provides a safer foundation for online RL adaptation. Experiments with Qwen3-14B show that higher offline conservatism monotonically increases reward hacking damage (Goodhart gap) during online adaptation, with Spearman ρ=1.0 across conditions. The mechanistic explanation is a three-link chain: high-β DPO compresses policy entropy, reducing response diversity and concentrating outputs in a narrow reward-model region, while paradoxically increasing ensemble disagreement that gets exploited during online optimization. The authors identify a practical optimal conservatism level β* and argue the field needs calibrated rather than maximal conservatism.
MAST: Mechanism-guided selective unlearning for RLVR-trained reasoning models
Researchers introduce MAST (Mechanism-Aligned Selective Targeting), a method for selectively unlearning capabilities induced by reinforcement learning from verifiable rewards (RLVR) in language models while minimizing collateral damage to retained knowledge. The approach ranks attention-projection tensors by off-principal energy and gradient coupling to identify a targeted subset for update, rather than applying full-parameter gradient ascent. Evaluated on Qwen2.5-Math-1.5B and Qwen3-1.7B-Base, MAST achieves statistically significant forgetting on target MATH problems while preserving GSM8K performance, whereas full-parameter unlearning collapses retained capabilities. The method generalizes across seeds and unlearning objectives (NPO/SimNPO).
ADAS: Attention-Discounted Adaptive Sampler improves parallel decoding for masked diffusion language models
Researchers propose ADAS, a training-free reranking rule for masked diffusion language model decoding that addresses token interaction failures in parallel token commitment. The method greedily penalizes candidates that attend strongly to already-selected uncertain positions, using attention weights as soft marginal penalties rather than hard constraints. Evaluated on LLaDA-8B-Base and Dream-7B-Base across GSM8K, MATH500, HumanEval, and MBPP, ADAS improves low-NFE performance by 9–10 percentage points on average when plugged into existing samplers with only 3.1% runtime overhead.
Anthropic launches Claude 2 with 100K context window and improved coding, reasoning, and safety
Anthropic released Claude 2, featuring a 100K token context window, improved performance on coding (71.2% on Codex HumanEval, up from 56.0%), math (88.0% on GSM8k), and legal reasoning (76.5% on the Bar exam multiple choice section). The model is available via API at the same price as Claude 1.3 and through a new public beta at claude.ai for US and UK users. Safety improvements include a 2x reduction in harmful outputs on internal red-team evaluations compared to Claude 1.3. Early API partners include Jasper and Sourcegraph.
Anthropic launches Claude 3 model family: Haiku, Sonnet, and Opus
Anthropic announced the Claude 3 model family on March 4, 2024, comprising three models — Haiku, Sonnet, and Opus — in ascending capability order. Claude 3 Opus claims top performance on major benchmarks including MMLU, GPQA, and GSM8K, with near-perfect recall on long-context evaluations (200K context window, 99%+ NIAH accuracy) and new multimodal vision capabilities. The release also highlights reduced unnecessary refusals, a twofold accuracy improvement over Claude 2.1, and Constitutional AI-based safety tuning. Opus and Sonnet launched immediately via claude.ai and the Claude API across 159 countries, with Haiku to follow.
Anthropic releases Claude Instant 1.2 with improved math, coding, and safety
Anthropic released Claude Instant 1.2, an updated version of its faster, lower-cost model tier, now available via API. The release incorporates capabilities from Claude 2 and shows measurable benchmark gains: 58.7% on Codex (vs 52.8% for 1.1) and 86.7% on GSM8K (vs 80.9% for 1.1). Safety improvements include reduced hallucination and greater jailbreak resistance as measured by automated red-teaming.
Mistral AI Releases Mixtral 8x22B Under Apache 2.0
Mistral AI has released Mixtral 8x22B, a sparse Mixture-of-Experts model with 141B total parameters but only 39B active parameters, under the permissive Apache 2.0 license. The model features a 64K token context window, native function calling, multilingual support across five European languages, and strong math and coding performance. Mistral claims it outperforms all other open-weight models on standard benchmarks while being faster than dense 70B models due to sparse activation. An instructed version achieves 90.8% on GSM8K maj@8.
Activation Capping Technique Stabilizes LLM Assistant Personas Against Drift and Jailbreaks
Researchers from MATS, Oxford, and Anthropic introduced the 'assistant axis,' a vector derived from LLM layer outputs that quantifies how closely a model adheres to its trained assistant persona. They developed 'activation capping,' an inference-time method that corrects deviations from this axis when similarity falls below a threshold. Testing on Gemma 2 27B, Qwen3 32B, and Llama 3.3 70B showed harmful response rates to jailbreak prompts dropped by roughly half (e.g., 83% to 41% for Qwen3 32B) without degrading benchmark performance. The technique targets character-based jailbreaks that bypass system prompts by manipulating a model's internal representational state.
Statistical Re-evaluation of GSM-Symbolic Finds Benchmark Confounds and Overstated Reasoning Conclusions
A re-evaluation of the GSM-Symbolic benchmark (Mirzadeh et al., 2025) challenges its conclusion that LLMs lack genuine reasoning capabilities. Using Generalised Linear Mixed Models on 20 open-weight models, the authors find only half show statistically significant performance drops, and identify a previously unacknowledged distributional shift toward larger integers in GSM-Symbolic relative to GSM8K that accounts for significance in roughly half the remaining cases. After controlling for this confound, model-specific failure profiles emerge—including variable binding fragility, arithmetic limitations, and dual-task interference—suggesting the original blanket claims about LLM reasoning were both statistically premature and mechanistically misleading.
Semantic vs. Surface Noise in LLM Agents: 68-Cell Measurement Study with Held-Out Validation
This paper documents an empirical phenomenon across 10 LLMs from 7 architecture families: meaning-bearing perturbations (paraphrase, synonym substitution) cause final-answer inconsistency ~19.69 percentage points more often than presentation-level perturbations (formatting, reordering) of comparable severity, across GSM8K, MATH, and HotpotQA benchmarks. The effect is validated on a held-out 11th model (qwen2.5-14B-Instruct) with 1,800 trajectories. Trace-level analysis supports a 'stealth-divergence' picture where semantic perturbations preserve the first action but induce divergence in intermediate reasoning steps, while two prior mechanism claims are explicitly retracted. The study is notable for its honest reporting of stress-test failures and pre-registered replication.
Looped Diffusion Language Models (LoopMDM): Depth Scaling via Layer Looping
LoopMDM introduces selective looping of early-middle transformer layers in masked diffusion language models, achieving a depth-scaling effect without adding parameters. The approach matches same-size MDM performance with up to 3.3× fewer training FLOPs and outperforms deeper non-looped MDMs on reasoning benchmarks, including up to 8.5 points improvement on GSM8K. Inference-time compute scaling is enabled by varying loop counts, with adaptive loop scheduling providing additional efficiency gains. Attention analysis suggests looping works by promoting interactions among masked token positions.
OpenAI Trains System Solving Grade School Math Problems at ~55% Accuracy
OpenAI released a system for solving grade school math word problems that achieves roughly twice the accuracy of a fine-tuned GPT-3 model. The system scored 55% on a sample test where 9-12 year olds scored 60%, suggesting near-human performance on elementary math. This work represents an early milestone in neural network mathematical reasoning capabilities.