A production voice agent is a latency budget, an escalation policy and an evaluation set that happen to be wrapped in speech. Get those three right and the model choice becomes a detail — get them wrong and even the best model produces calls customers dislike.
The short answer
A production voice agent is a latency budget, an escalation policy and an evaluation set that happen to be wrapped in speech. Get those three right and the model choice becomes a detail. Get them wrong and the best model available will still produce calls your customers dislike.
The four constraints that decide everything
⏱️Latency
Speech recognition, retrieval, tool calls and synthesis all share one budget. Architecture is mostly the work of keeping the critical path short.
🔈Turn-taking
Barge-in, silence handling and endpointing. Callers interrupt; software that cannot handle it sounds broken even when its answers are correct.
🛡️Guardrails
What the agent may promise, what it must escalate, and how it behaves when it does not know. Reliability is a policy question before it is a model question.
📊Evaluation
A scored set of real call scenarios. Without it, every prompt or model change is an unmeasured gamble on live customers.
Stream everywhere. Recognition and synthesis both run incrementally; waiting for a full transcript before starting to answer already breaks the budget.
Route before you reason. A small, fast model classifies intent and fills required slots; only the turns that need it reach a larger model or a knowledge lookup.
Pre-compute what you can. Caller context, account state and likely answers can be fetched while the line is still ringing.
Timeout every dependency. If a backend lookup is slow, the agent gives a partial answer and continues rather than stalling the call.
Latency budget, line by line
Stage
Target
What usually breaks it
Endpointing / turn detection
< 200 ms
Fixed silence timers that add dead air to every turn.
Streaming speech recognition
150–300 ms
Batch transcription instead of streaming.
Intent routing and slot fill
< 150 ms
Sending every turn to the largest available model.
Retrieval or backend tool call
< 300 ms
Unindexed queries and synchronous third-party APIs.
Response generation (first token)
< 400 ms
Long prompts rebuilt on every turn.
Streaming speech synthesis
< 200 ms to first audio
Generating the whole utterance before playing any of it.
Guardrails and escalation
Policy before prompt. Define what the agent may say, promise and commit to; encode those limits outside the model so a prompt change cannot quietly widen them.
Escalate on signals, not only on requests. Low confidence, repeated rephrasing, frustration markers and named policy categories all trigger a human transfer.
Hand off with context. The human receives the transcript, captured fields and a summary, so the caller never repeats themselves.
Log and tag every call. Outcome tags and recordings are what let you fix accuracy after launch rather than guessing.
Never let it improvise on regulated topics. Medical, legal, payment and complaint handling follows a fixed path with a human in it.
Testing before you dial out
Build a scripted suite covering happy paths, interruptions, silence, background noise, accents and edge intents.
Run it on every prompt, model or telephony change and record intent accuracy, containment rate and false-transfer rate.
Shadow-run against a slice of live traffic where the agent recommends an action but does not speak to the caller.
Enable outbound calling only after containment and escalation metrics hold on the shadow data.
Respond within roughly 300 to 800 milliseconds for the perception of a normal conversation. Beyond about one second callers start talking over the agent or assuming the line has dropped. That budget covers speech recognition, retrieval or tool calls, and speech synthesis, so it has to be designed rather than hoped for.
Barge-in detection: the agent stops speaking the moment the caller starts, keeps the partial transcript, and responds to the new turn instead of finishing its script. Without it, calls feel robotic and callers repeat themselves, which is the fastest way to lose trust in the system.
On low confidence, on explicit request, on sentiment signals such as frustration, and on any policy category you define — payments, medical advice, complaints. The transfer carries the full transcript and captured fields so the caller never repeats themselves.
With a regression suite of recorded and scripted calls covering happy paths, interruptions, silence, accents, background noise and edge intents, plus shadow running against live traffic where the agent recommends but does not speak. We measure intent accuracy, containment rate and false-transfer rate before enabling outbound dialling.
Cost is driven by telephony minutes, speech recognition, model tokens and synthesis, and it scales with call volume rather than seats. A constrained design — small fast model for routing, retrieval only where needed, batching non-realtime work — usually matters more than the provider you pick.