Glossary

Tokenizer — Glossary

Software that splits text into tokens and maps them to vocabulary IDs for a model.

Reviewed:

Full definition

Subword tokenizers (BPE, SentencePiece) balance vocabulary size with rare-word handling. Different models tokenize differently; billing counters follow each vendor’s tokenizer.

Related encyclopedia topic

Tokens — Tokens are the discrete units language models read and generate—commonly subword pieces—used for context limits, billing, and model input encoding.

← Back to glossary · Learning hub