Glossary · Core Concepts

Tokenization

The process of breaking text into smaller units (tokens) that a language model processes, where a token is often a word fragment rather than a whole word.