Short answer
Attention is how transformers route information between tokens. Self-attention lets every token selectively gather context from others in the sequence.
Why it matters
Attention explains both the power and the cost of long prompts—and why packing better evidence often beats pasting entire databases into context.
Diagram
Article
Analogy
When you answer a question about a long email, you reread the relevant sentences. Attention is a learned version of that focusing behavior across tokens.
Examples
Coreference
In “The deposit was refunded after the landlord inspected the apartment,” attention helps bind “the deposit” to later references when predicting answers.
FAQ
Is attention unique to transformers?
Attention mechanisms existed earlier (e.g., in RNN seq2seq), but transformers make attention the primary sequence mixer.
Sources
-
Attention Is All You Need
NeurIPS / arXivFoundational transformer architecture paper.
-
Deep Learning
MIT PressCanonical deep learning textbook by Goodfellow, Bengio, and Courville.