model

Sparse Transformer

modelactivesparse-transformer-502a222b·1 events·first seen 28d ago

Aliases: Sparse Transformer

Co-occurring entities

More like this (12)

Sparse Autoencoder Graph Transformer Sparse Embedding Models Block Sparse Attention sparse attention Swift Transformers Sparse Circuits Sentence Transformers Decision Transformer cuSPARSE block sparse matrices Dynamic Short Convolutions Improve Transformers

Recent events (1)

6Openai Blog·28d ago·source ↗

Generative modeling with sparse transformers

OpenAI introduced the Sparse Transformer, a deep neural network using a modified sparse attention mechanism to model sequences up to 30x longer than previously feasible with standard transformers. The approach sets new benchmarks on text, image, and audio generation tasks. The key algorithmic contribution is factorized sparse attention patterns that reduce the quadratic complexity of full self-attention.

Long Context Evolution Frontier Model Releases Sparse Transformer sparse attention OpenAI +1 more