Short answer
Attention is how transformers route information between tokens. Self-attention lets every token selectively gather context from others in the sequence.
Why it matters
Attention explains both the power and the cost of long prompts—and why packing better evidence often beats pasting entire databases into context.
Diagram
Article
Systems notes
FlashAttention-style kernels improve memory IO. Sliding windows, sparsity, and alternative architectures trade exactness for length. Profile real sequence lengths before promising ultra-long contexts.
Examples
Coreference
In “The deposit was refunded after the landlord inspected the apartment,” attention helps bind “the deposit” to later references when predicting answers.
FAQ
Is attention unique to transformers?
Attention mechanisms existed earlier (e.g., in RNN seq2seq), but transformers make attention the primary sequence mixer.
Sources
-
Attention Is All You Need
NeurIPS / arXivFoundational transformer architecture paper.
-
Deep Learning
MIT PressCanonical deep learning textbook by Goodfellow, Bengio, and Courville.