Glossary

Quantization — Glossary

Reducing numeric precision of model weights (and sometimes activations) to save memory and speed up inference.

Reviewed:

Full definition

Quantization stores parameters with fewer bits—commonly 8-bit or 4-bit for large language models—so models fit on smaller GPUs or CPUs. Post-training quantization is popular for local LLMs; quantization-aware training is more involved.

Quality tradeoffs vary by task. Always evaluate quantized artifacts on your workloads, especially structured extraction and safety behaviors.

See the encyclopedia topic Quantization for operational guidance.

Related encyclopedia topic

Quantization — Quantization reduces numeric precision of model weights (and sometimes activations) to cut memory and accelerate inference with controlled quality loss.

← Back to glossary · Learning hub