Glossary
Token — Glossary
A chunk of text (often a word piece) used as a basic unit for language model training, context limits, and generation.
Full definition
Tokenization splits text into tokens that a model can process numerically. Depending on the tokenizer, a token may be a whole word, part of a word, punctuation, or a short character sequence. Models predict tokens sequentially during text generation.
Context windows and API pricing are commonly measured in tokens. Different languages and code styles tokenize with different efficiency, which affects cost and how much content fits into a prompt.