Glossary · Infrastructure
Dynamic Batching
A model-serving technique that groups multiple incoming requests together for simultaneous processing to improve throughput, at the cost of a small added latency while the batch fills.
Glossary · Infrastructure
A model-serving technique that groups multiple incoming requests together for simultaneous processing to improve throughput, at the cost of a small added latency while the batch fills.