Attention Mechanisms in Deep Learning

Attention architectures: self-attention, multi-head attention, cross-attention, scaled dot-product, sparse attention, and efficient attention variants (Linformer, Performer, FlashAttention).

AI & ML Terms
8 entries
CC0 — public domain

JSON9,019 B
YAML6,464 B · −28%
TOON5,780 B · −36%

Attention mechanisms in deep learning with formula, complexity, description, uses and notes: scaled dot-product, multi-head, cross-attention, causal masking, sparse attention, FlashAttention and linear attention variants.

Use it when

You need the definition and cost of an attention variant.
You are comparing attention approaches for long sequences.

Watch out for

Standard attention cost grows quadratically with sequence length, which motivates sparse and linear variants.
Approximations can trade quality for speed; benchmark on your task.

The data

Mechanisms

Name	Formula	Complexity	Description	Use cases	Notes	Mechanism	Variants	Connection
Scaled Dot-Product Attention	Attention(Q,K,V) = softmax(QK^T / √d_k) V	O(n² * d)	Core attention mechanism: queries attend to keys, weighted values are summed	Foundation of all Transformer models Sequence-to-sequence tasks Any pairwise interaction modeling	Scaling by √d_k prevents softmax saturation for large d_k; Q=K=V gives self-attention	null	null	null
Multi-Head Attention	Concatenate h heads of attention with different learned projections	O(n² * d) (same as single head, parallelized)	Multiple attention heads capture different representation subspaces	All Transformer architectures Cross-modal attention (vision-language) Multi-modal fusion	Each head: Q_i = XW_i^Q, K_i = XW_i^K, V_i = XW_i^V; typical h=8,12,16	null	null	null
Cross-Attention	null	O(n * m * d)	Attention between two different sequences; decoder attends to encoder output	Encoder-decoder (translation, summarization) Image captioning (text attends to image features) Multimodal models	Q projects decoder state; K,V project encoder output; different sequence lengths allowed	Q from one sequence, K,V from another	null	null
Causal (Masked) Attention	null	O(n² * d) but only n(n+1)/2 actual computations	Prevents positions from attending to future positions; essential for autoregressive generation	GPT-style language models Autoregressive generation Any causal sequence modeling	Enables efficient KV cache during inference: only compute for new token, reuse cached K,V	Upper triangular mask on attention scores: M[i,j] = -∞ for j > i	null	null
Sparse Attention	null	O(n * w * d) for window size w (linear in n)	Restrict attention to subset of positions; reduces quadratic complexity	Long document processing Genomic sequences When full attention is too expensive	Longformer, BigBird, Reformer use different sparsity patterns; trade-off between expressiveness and efficiency	null	Fixed pattern (local/windowed) Strided Learned sparse patterns	null
FlashAttention	null	O(n² * d) FLOPs but O(n²) memory I/O (vs O(n²d) for standard)	Exact attention with optimal memory access patterns; 2-4x faster than standard attention	Training large language models Long-context models Any attention-heavy workload	No approximation; same mathematical result as standard attention; now default in most training frameworks	Tiling + recomputation to minimize HBM (GPU memory) reads/writes	null	null
Linear Attention (Performer, Linformer)	null	O(n * d²) or O(n * log n)	Approximate attention with linear complexity in sequence length	Very long sequences (10K+ tokens) When quadratic attention is infeasible Real-time inference on long inputs	Performer uses FAVOR+ (Fast Attention via positive Orthogonal Random Features); Linformer projects K,V to lower dimension	Approximate softmax kernel with random features or low-rank projection	null	null
Cross-Entropy Loss with Attention	null	null	Attention provides interpretable alignment: which input tokens influenced each output token	Interpretable NLP models Attention visualization Debugging model behavior	Attention ≠ explanation (caution: attention weights don't always indicate feature importance)	null	null	Attention weights can be interpreted as soft alignment probabilities

Fields

The shape of this dataset, read from the data itself: every column, its type and a real value. Agents can use this to plan a query without fetching the whole document.

Section	Field	Type	Example
mechanisms	`name`	string	Scaled Dot-Product Attention
mechanisms	`formula`	string	Attention(Q,K,V) = softmax(QK^T / √d_k) V
mechanisms	`complexity`	string	O(n² * d)
mechanisms	`description`	string	Core attention mechanism: queries attend to key…
mechanisms	`use_cases`	array	[3 items]
mechanisms	`notes`	string	Scaling by √d_k prevents softmax saturation for…
mechanisms	`mechanism`	string	Q from one sequence, K,V from another
mechanisms	`variants`	array	[3 items]
mechanisms	`connection`	string	Attention weights can be interpreted as soft al…

Fetch the same bytes

The static files contain the same reference rows as the API, with no API rate limit. Live metadata, such as view counts, may differ. Use the API for content negotiation; use the static files for a cacheable snapshot.

curl "https://yjtoon.com/api/dataset/attention-mechanisms?format=toon"

const res = await fetch(
  "https://yjtoon.com/static-data/dataset/attention-mechanisms.toon"
);
const toon = await res.text();

Rate limit: 120 requests per minute per IP, no key and no signup. API reference →