Latency

Latency is the time from request to response — for an AI employee, it's the user-perceived wait between asking and getting an answer.

Latency in agentic systems compounds: each model call, each tool call, each retry adds time. A single ReAct loop iteration is ~1–5 seconds; a 6-step workflow can take 30+. The trade-off is latency vs. quality: cheaper-faster models for high-volume steps, slow expensive ones only when they must.

Example

An AI Support Rep aims for <5s first-token latency on customer chats — uses a fast model, retrieves once, answers. Saves the deeper-reasoning model for escalations.

How OpenLabor uses it

OpenLabor measures p50/p95 latency per mission so you know where to optimize.

What latency is acceptable for chat?

First token under 2s feels conversational; full answer under 10s feels productive. Above that users perceive 'thinking'.

Related: llm, throughput, cost-per-task.

AI Labor Glossary