Short answer
A token is a chunk of text (often part of a word) converted to an ID the model understands. Context windows and API prices are usually measured in tokens.
Why it matters
Token budgets decide how much evidence you can include, how long answers can be, and what a feature will cost at scale.
Diagram
Article
APIs
Count tokens before sending long RAG contexts. Truncate with structure awareness—do not chop mid-JSON tool schemas when avoidable.
Examples
Budgeting RAG
You reserve 1,500 tokens for system+question, 6,000 for retrieved chunks, and 1,000 for the answer.
FAQ
How many tokens are in a word?
Often about 0.5–2 tokens for English words, but it varies widely; always measure with the real tokenizer.
Sources
-
OpenAI API Documentation
OpenAIOfficial OpenAI platform documentation.
-
Language Models are Few-Shot Learners
NeurIPS / arXivGPT-3 few-shot learning paper (Brown et al.).
-
Hugging Face Documentation
Hugging FaceOpen model hub and Transformers library docs.