Glossary · Infrastructure

KV Cache

A memory optimization technique that stores previously computed attention key and value tensors during text generation, avoiding redundant recomputation and significantly speeding up inference for long sequences.