Glossary · Infrastructure

Batch Inference

Processing multiple model requests together as a group rather than one at a time, improving hardware utilization and throughput at the cost of some added latency per individual request.