Glossary · Infrastructure

Model Inference Cost

The compute expense of running a model to generate a response, distinct from training cost — the recurring cost that scales with usage and is the main driver of an agent application's ongoing operating expense.