Glossary · Core Concepts

Tokenizer

The component that converts raw text into the discrete tokens a language model actually processes, and converts generated tokens back into readable text.