Entity · paper

L1 Augmented Attention as an Improved Vector Similarity Metric

paperactiveprovisionall1-augmented-attention-as-an-improved-vector-similarity-metric-7c227b25·1 events·first seen 11h ago

Aliases: L1 Augmented Attention as an Improved Vector Similarity Metric

Co-occurring entities

WikiText-2

More like this (12)

Locality-Sensitive Hashing Attention Graph Attention Network Spectral Attention The Many Senses of Visual Similarity: A Text-Prompted Image Perceptual Metric Lie-Algebra Attention Attention Expansion: Enhancing Keyphrase Extraction from Long Documents with Attention-Augmented Contextualized Embeddings LALS (Latent Association Leaning Score)In-context Vector Distillation Graph Convolutional Attention Cross-Layer Sparse Attention Representational Similarity Analysis sparse attention

Recent events (1)

4arXiv · cs.CL·11h ago·source ↗

L1 Augmented Attention improves Transformer similarity metric with 14.5% perplexity reduction

A new arXiv preprint proposes L1 augmented attention, a modification to scaled dot product attention that subtracts a learned, head-specific L1 distance between queries and keys from the dot product score. The hybrid metric captures complementary geometric information — directional alignment from dot product and coordinate deviation penalties from L1 — using low-dimensional projections to keep computation parallelizable. Evaluated on WikiText-2 with a compact transformer, the method achieves up to 14.5% perplexity reduction over the standard baseline and outperforms an RBF/L2 kernel alternative. Analysis reveals distinct geometric roles across layers and strong head-level specialization.

Evaluation and Benchmarking WikiText-2 L1 Augmented Attention as an Improved Vector Similarity Metric