Short answer
A transformer is a deep learning architecture built around attention. It powers most large language models and many image and audio systems.
Why it matters
Understanding transformers clarifies context windows, token costs, fine-tuning surfaces, and why today’s AI products behave as they do.
Diagram
Article
Engineering checklist
Track model IDs, tokenizer versions, max context, and precision. Use KV caching for decode efficiency; consider quantization for local or cost-sensitive serving.
Fine-tune with LoRA when full updates are unnecessary. For RAG, align chunking with the model’s effective context and attention behavior.
Read the original Attention Is All You Need paper alongside framework docs (Hugging Face Transformers, official API model cards).
Examples
Chat completion
A decoder-only transformer predicts the next token repeatedly until it finishes an answer.
Translation
An encoder–decoder transformer encodes a French sentence and decodes English tokens.
FAQ
Are all LLMs transformers?
Nearly all widely deployed LLMs today are transformer variants, though research explores alternative architectures.
Do transformers only work on text?
No. They are used for images, audio, video tokens, proteins, and more.
Sources
-
Attention Is All You Need
NeurIPS / arXivFoundational transformer architecture paper.
-
Deep Learning
MIT PressCanonical deep learning textbook by Goodfellow, Bengio, and Courville.
-
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
NAACL / arXivBERT pretraining paper.
-
Hugging Face Documentation
Hugging FaceOpen model hub and Transformers library docs.