Evidence-first authority page
Setting Latency Budgets for AI Agent Applications
How to think about acceptable response-time targets for an agent-backed feature, and where multi-step tool calls make this harder than a single LLM call.
Direct answer
A latency budget for an AI agent feature should be set per user-facing interaction point, not as a single global number — a synchronous chat response needs a tighter budget (seconds) than a background multi-step research task (minutes), and every additional tool call or model round-trip in an agent's chain adds to the total, so the budget should be allocated explicitly across the steps rather than discovered after the fact.
Last reviewed 2026-08-29
This page remains noindex while its page-level source receipts are completed and verified. It is available for review, not presented as publication-ready authority.
Why agent latency is harder to budget than a single API call
A single LLM call has a fairly predictable latency range. An agent that makes multiple sequential tool calls and model round-trips compounds that latency with each step — a five-step agentic chain can easily take 10-30 seconds even when each individual step is fast, which is a different user-experience problem than a single slow call.
Practical approaches
- Show progressive status updates for multi-step tasks rather than a single blocking spinner, so users understand the agent is working through steps.
- Parallelize independent tool calls where the agent's steps don't depend on each other's output.
- Set explicit per-interaction-type budgets (chat vs. background task) rather than one global latency target.