Short answer
A transformer is a deep learning architecture built around attention. It powers most large language models and many image and audio systems.
Why it matters
Understanding transformers clarifies context windows, token costs, fine-tuning surfaces, and why today’s AI products behave as they do.
Diagram
Article
Product implications
Context windows, tokenizer behavior, and positional schemes affect what documents you can stuff into a prompt. Long-context models reduce some RAG needs but do not remove grounding requirements for private data.
Measure task success, latency, and failure modes on your corpus rather than relying only on public leaderboards.
Risk notes
Transformers inherit training-data biases and can hallucinate. Pair them with retrieval, tools, and review when stakes are high.
Examples
Chat completion
A decoder-only transformer predicts the next token repeatedly until it finishes an answer.
Translation
An encoder–decoder transformer encodes a French sentence and decodes English tokens.
FAQ
Are all LLMs transformers?
Nearly all widely deployed LLMs today are transformer variants, though research explores alternative architectures.
Do transformers only work on text?
No. They are used for images, audio, video tokens, proteins, and more.
Sources
-
Attention Is All You Need
NeurIPS / arXivFoundational transformer architecture paper.
-
Deep Learning
MIT PressCanonical deep learning textbook by Goodfellow, Bengio, and Courville.
-
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
NAACL / arXivBERT pretraining paper.
-
Hugging Face Documentation
Hugging FaceOpen model hub and Transformers library docs.