Natural voice automation is an end-to-end latency problem. The architecture has to optimize the full conversational path rather than any single model or service in isolation.
Treat latency as a budget
A production voice agent should have an explicit latency budget across voice activity detection, speech recognition, orchestration, model response generation, text-to-speech and media delivery. Measuring only model latency hides bottlenecks elsewhere in the turn.
The most useful operating view is a percentile distribution for each stage. P50 shows the normal experience while P95 and P99 expose the long-tail delays that callers actually notice during congestion or provider degradation.
Keep signaling and media responsibilities clear
SIP signaling, RTP media processing and AI orchestration have different scaling and failure characteristics. Separating them makes capacity planning, observability and failure isolation substantially easier.
- Use a dedicated SIP edge for routing, policy, topology control and security.
- Anchor media only where the architecture needs transcoding, recording, policy enforcement or media inspection.
- Keep AI orchestration stateless where practical and persist only the conversation state required for recovery.
Design human handoff from the beginning
Human escalation should not be added after the voice bot is complete. The call-control model, metadata propagation, transfer method and contact-center integration all affect the original architecture. Design SIP REFER, attended transfer or application-controlled bridging deliberately so the handoff preserves context and does not create avoidable media hairpins.