Glossary

KV cache — Glossary

Cached key/value attention states that speed up autoregressive decoding.

Reviewed:

Full definition

KV caching avoids recomputing attention states for past tokens during generation, improving tokens-per-second at the cost of memory that grows with context length.

Related encyclopedia topic

AI Inference — Inference is running a trained model on new inputs to produce predictions or generations—the production phase users experience when they chat, search, or…

← Back to glossary · Learning hub