Short answer
Attention is how transformers route information between tokens. Self-attention lets every token selectively gather context from others in the sequence.
Why it matters
Attention explains both the power and the cost of long prompts—and why packing better evidence often beats pasting entire databases into context.
Diagram
Article
Mechanics
Queries, keys, and values are linear projections of token representations. Softmax over scaled dot products produces weights. Multi-head attention learns several focus patterns in parallel.
Cross-attention lets decoder tokens attend over encoder outputs in sequence-to-sequence models.
Examples
Coreference
In “The deposit was refunded after the landlord inspected the apartment,” attention helps bind “the deposit” to later references when predicting answers.
FAQ
Is attention unique to transformers?
Attention mechanisms existed earlier (e.g., in RNN seq2seq), but transformers make attention the primary sequence mixer.
Sources
-
Attention Is All You Need
NeurIPS / arXivFoundational transformer architecture paper.
-
Deep Learning
MIT PressCanonical deep learning textbook by Goodfellow, Bengio, and Courville.