Glossary

Distillation — Glossary

Training a smaller model to mimic a larger model’s behaviors or probability outputs.

Also known as: knowledge distillation

Reviewed:

Full definition

Knowledge distillation compresses capabilities for cheaper inference. Student quality depends on data, temperature scaling of soft targets, and task coverage.

Related encyclopedia topic

Fine-Tuning — Fine-tuning adapts a pretrained model to a target task, domain, or style by continuing training on curated examples—often with parameter-efficient methods.

← Back to glossary · Learning hub