Short answer
A transformer is a deep learning architecture built around attention. It powers most large language models and many image and audio systems.
Why it matters
Understanding transformers clarifies context windows, token costs, fine-tuning surfaces, and why today’s AI products behave as they do.
Diagram
Article
Simple picture
Older models read text mainly left-to-right in a chain. Transformers look at many words at once using attention, which helps them connect ideas across a sentence or document.
When you use a chatbot, a coding assistant, or many translation systems, a transformer-style model is usually doing the heavy lifting.
What you should remember
You do not need the math to use them well, but knowing they rely on attention and tokens helps you understand limits like context windows and costs.
Examples
Chat completion
A decoder-only transformer predicts the next token repeatedly until it finishes an answer.
Translation
An encoder–decoder transformer encodes a French sentence and decodes English tokens.
FAQ
Are all LLMs transformers?
Nearly all widely deployed LLMs today are transformer variants, though research explores alternative architectures.
Do transformers only work on text?
No. They are used for images, audio, video tokens, proteins, and more.
Sources
-
Attention Is All You Need
NeurIPS / arXivFoundational transformer architecture paper.
-
Deep Learning
MIT PressCanonical deep learning textbook by Goodfellow, Bengio, and Courville.
-
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
NAACL / arXivBERT pretraining paper.
-
Hugging Face Documentation
Hugging FaceOpen model hub and Transformers library docs.