Glossary
Model inference (serving) — Glossary
Running a trained model on new inputs to produce predictions or generations.
Full definition
Inference is the production phase: chat completions, embeddings, transcriptions, or image generations. It must meet latency, cost, and reliability targets distinct from training-time concerns.
Techniques like batching, KV caching, and quantization optimize inference efficiency.