Entity · benchmark
WikiText-2
benchmarkactive
wikitext-2-d1312705·5 events·first seen Aliases: WikiText-2, WikiText, Wikitext
Co-occurring entities
Llama-3.1-8BRULERL1 Augmented Attention as an Improved Vector Similarity MetricPALSLLaMA-7BSparseGPTMistral 7BPALS: Percentile-Aware Layerwise Sparsity for LLM PruningWandaLAMBADASlimPajamaComplementary Learning SystemsHOLA (Hippocampal Linear Attention)CARVEGDN-2WY-form triangular chunk solverlayer pruningPythiaQwen3-4Bswap-KL
More like this (12)
Recent events (5)
L1 Augmented Attention improves Transformer similarity metric with 14.5% perplexity reduction
A new arXiv preprint proposes L1 augmented attention, a modification to scaled dot product attention that subtracts a learned, head-specific L1 distance between queries and keys from the dot product score. The hybrid metric captures complementary geometric information — directional alignment from dot product and coordinate deviation penalties from L1 — using low-dimensional projections to keep computation parallelizable. Evaluated on WikiText-2 with a compact transformer, the method achieves up to 14.5% perplexity reduction over the standard baseline and outperforms an RBF/L2 kernel alternative. Analysis reveals distinct geometric roles across layers and strong head-level specialization.
PALS: Percentile-aware per-layer sparsity improves LLM pruning on LLaMA-2 but not universally
PALS (Percentile-Aware Layerwise Sparsity) is a one-shot pruning method that assigns per-layer sparsity ratios based on the 99th percentile of activation magnitudes, bounded within ±5% of a target ratio. On LLaMA-2-7B at 50% sparsity, PALS achieves perplexity of 10.96 vs. 12.92 for uniform Wanda, a statistically significant improvement requiring no fine-tuning. However, gains are architecture-dependent: LLaMA-3-8B shows marginal improvement and Mistral-7B shows none. A notable negative finding is that gradient-based allocation performs worse than random, suggesting gradient magnitude is a poor proxy for the impact of discrete weight removal.
HOLA adds hippocampal exact KV cache to linear attention, closing gap with full-attention Transformers
HOLA (Hippocampal Linear Attention) augments linear-attention and state-space models with a bounded exact key-value cache inspired by Complementary Learning Systems theory, addressing the lossy compression problem that causes earlier facts to be overwritten in recurrent states. The cache uses a residual-based eviction criterion (large beta * ||e||) without a learned eviction module, and a decoupled RMSNorm-gamma read for sharp retrieval. At 340M parameters trained on 15B SlimPajama tokens, HOLA reduces Wikitext perplexity from 27.32 to 22.92, falling below a full-attention Transformer++ baseline, and shows strong needle-in-a-haystack recall out to 32k tokens despite training only at 2k. The work is directly relevant to the open question of whether linear-attention models can match full-attention on long-context retrieval tasks.
CARVE: Content-aware gating for linear attention recurrent models improves efficiency and quality over GDN-2
CARVE (Content-Aware Recurrent with Value Efficiency) is a new linear attention architecture that addresses three coupled defects in the GDN-2 delta-rule architecture by restricting erasure to the key axis rather than the value axis. This design choice is proven necessary and sufficient to enable the WY-form triangular chunk solver, enabling competitive training throughput with Transformers. At 1.3B parameters trained on 100B tokens, CARVE achieves lower perplexity than GDN-2, leads recurrent baselines on nine commonsense reasoning benchmarks, and sets state-of-the-art on RULER retrieval probes, while using 13% less peak memory and 19% fewer parameters at 0.4% throughput overhead.
Layer Equivalence Is Not a Property of Layers Alone: How You Test Redundancy Changes What You Find
This paper distinguishes two protocols for measuring transformer layer redundancy—replacement (can one layer substitute for another in place?) and interchange (do two layers approximately commute when swapped?)—and shows they can disagree substantially. Experiments on Pythia (410M, 1.4B) and 8B-scale models (Qwen3-8B, Llama-3.1-8B) reveal that the protocol gap grows during training and can change which layers appear safe to prune by several-fold. Notably, Qwen3-8B shows interchange-guided removal is far safer than replacement-guided at the same layer budgets, while Llama-3.1-8B ties the two protocols despite lower interchange KL. The authors recommend scoring both swap-KL metrics before any layer removal or merging, requiring only unlabeled forward passes.