Short answer
A transformer is a deep learning architecture built around attention. It powers most large language models and many image and audio systems.
Why it matters
Understanding transformers clarifies context windows, token costs, fine-tuning surfaces, and why today’s AI products behave as they do.
Diagram
Article
Architecture sketch
Vaswani et al. (2017) introduced multi-head self-attention, positional information, and stacked feed-forward blocks. Encoders build bidirectional representations; decoders generate autoregressively; encoder–decoder models suit sequence-to-sequence tasks.
Residual connections and layer normalization stabilize deep stacks, enabling large-scale training.
Why they won
Better parallelization than recurrent nets on modern accelerators, plus strong long-range dependency modeling, made transformers the default backbone for NLP and beyond.
Examples
Chat completion
A decoder-only transformer predicts the next token repeatedly until it finishes an answer.
Translation
An encoder–decoder transformer encodes a French sentence and decodes English tokens.
FAQ
Are all LLMs transformers?
Nearly all widely deployed LLMs today are transformer variants, though research explores alternative architectures.
Do transformers only work on text?
No. They are used for images, audio, video tokens, proteins, and more.
Sources
-
Attention Is All You Need
NeurIPS / arXivFoundational transformer architecture paper.
-
Deep Learning
MIT PressCanonical deep learning textbook by Goodfellow, Bengio, and Courville.
-
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
NAACL / arXivBERT pretraining paper.
-
Hugging Face Documentation
Hugging FaceOpen model hub and Transformers library docs.