Short answer
A token is a chunk of text (often part of a word) converted to an ID the model understands. Context windows and API prices are usually measured in tokens.
Why it matters
Token budgets decide how much evidence you can include, how long answers can be, and what a feature will cost at scale.
Diagram
Article
Why you care
Pricing and context limits are usually measured in tokens, not pages. A short question can become many tokens in some languages or code.
Examples
Budgeting RAG
You reserve 1,500 tokens for system+question, 6,000 for retrieved chunks, and 1,000 for the answer.
FAQ
How many tokens are in a word?
Often about 0.5–2 tokens for English words, but it varies widely; always measure with the real tokenizer.
Sources
-
OpenAI API Documentation
OpenAIOfficial OpenAI platform documentation.
-
Language Models are Few-Shot Learners
NeurIPS / arXivGPT-3 few-shot learning paper (Brown et al.).
-
Hugging Face Documentation
Hugging FaceOpen model hub and Transformers library docs.