Short answer
A token is a chunk of text (often part of a word) converted to an ID the model understands. Context windows and API prices are usually measured in tokens.
Why it matters
Token budgets decide how much evidence you can include, how long answers can be, and what a feature will cost at scale.
Diagram
Article
Details
Subword tokenization balances vocabulary size with rare-word handling. Different models tokenize differently—never assume whitespace equals token boundaries.
Examples
Budgeting RAG
You reserve 1,500 tokens for system+question, 6,000 for retrieved chunks, and 1,000 for the answer.
FAQ
How many tokens are in a word?
Often about 0.5–2 tokens for English words, but it varies widely; always measure with the real tokenizer.
Sources
-
OpenAI API Documentation
OpenAIOfficial OpenAI platform documentation.
-
Language Models are Few-Shot Learners
NeurIPS / arXivGPT-3 few-shot learning paper (Brown et al.).
-
Hugging Face Documentation
Hugging FaceOpen model hub and Transformers library docs.