Glossary
Attention mechanism — Glossary
A neural network component that weights which parts of an input sequence matter most for the current computation.
Also known as: self-attention, multi-head attention
Full definition
Attention mechanisms compute a weighted combination of value vectors based on compatibility between queries and keys. In transformers, multi-head self-attention lets every token gather information from other tokens in parallel.
The 2017 paper Attention Is All You Need made attention the primary sequence-mixing block for modern language models. Cross-attention variants let one sequence read another, as in classic encoder–decoder translation systems.
Practically, attention helps explain why packing better evidence into prompts often beats dumping entire corpora into a context window. Attention weight visualizations can offer clues but are not complete explanations of model reasoning.
Related encyclopedia topic: Attention.