Glossary · Architecture

Sparse Attention

An umbrella term for attention mechanism variants that compute attention over only a subset of token pairs rather than every pair, reducing the quadratic compute cost of standard attention.