Glossary · Core Concepts

Token

The basic unit of text processing in language models. Tokens can be words, parts of words, or individual characters, depending on the model's tokenizer. For English text, one token is approximately 4 characters or 0.75 words. For Indian languages, tokenization is less efficient — some languages require more tokens to express the same content. Pricing for API-based models is typically per-token.