Short answer
Attention is how transformers route information between tokens. Self-attention lets every token selectively gather context from others in the sequence.
Why it matters
Attention explains both the power and the cost of long prompts—and why packing better evidence often beats pasting entire databases into context.
Diagram
Article
Applied view
Put critical instructions and evidence where models reliably attend. Excess irrelevant context can dilute attention—one reason chunking and reranking matter in RAG.
Examples
Coreference
In “The deposit was refunded after the landlord inspected the apartment,” attention helps bind “the deposit” to later references when predicting answers.
FAQ
Is attention unique to transformers?
Attention mechanisms existed earlier (e.g., in RNN seq2seq), but transformers make attention the primary sequence mixer.
Sources
-
Attention Is All You Need
NeurIPS / arXivFoundational transformer architecture paper.
-
Deep Learning
MIT PressCanonical deep learning textbook by Goodfellow, Bengio, and Courville.