Short answer
Quantization stores model numbers with fewer bits (like 8-bit or 4-bit) so models fit on smaller hardware and run faster, usually with some accuracy tradeoff.
Why it matters
It is a primary technique enabling local LLMs, cheaper APIs at scale, and edge deployment.
Article
Concepts
Post-training quantization is common for LLMs; quantization-aware training is more involved. Calibration data influences quality.
Examples
Laptop assistant
A 4-bit quantized 7B–8B model runs locally for private note summarization with acceptable quality.
FAQ
Does quantization change training?
Post-training quantization mostly changes inference artifacts; QAT modifies the training loop.
Sources
-
Hugging Face Documentation
Hugging FaceOpen model hub and Transformers library docs.
-
Deep Learning
MIT PressCanonical deep learning textbook by Goodfellow, Bengio, and Courville.
-
OpenAI API Documentation
OpenAIOfficial OpenAI platform documentation.