The Full Voice-Agent Architecture Behind Dyx

The problem

Dyx is a phone number. +1 (484) 270-7074. Anyone can dial it. A voice picks up, has a short conversation, and either takes a message, books a call, or hangs up on a robodialer. The whole thing sits at voicemail.kaushik.cv and I built it because I stopped answering unknown numbers three years ago and my inbox is not a good place to lose messages.

Two earlier posts covered pieces of it. The latency budget walked through the sub-700ms end-to-end path and why I abandoned the single-model speech-to-speech approach. The guard-rails post covered the six protocols that had to sit above the persona before I was willing to leave the number publicly listed. This post is the missing middle: the full architecture beyond latency and safety, the boxes, the arrows, the money, the failure modes, and the specific reasons I did not use the managed products that would have shipped this in an afternoon.

Prior art

There is no shortage of ways to answer a phone with an LLM in 2026. I tried or seriously evaluated five of them before I landed on a hand-orchestrated LiveKit pipeline. Ordered roughly by how much they abstract over the pipeline.

OpenAI Realtime API. A single speech-in-speech-out endpoint over WebRTC or WebSocket. Genuinely fast, 300ms-ish end-to-end when the network cooperates, and the API surface is small. It is also the most expensive of the options I looked at, roughly nine cents per minute of audio at current published pricing, and it is a closed box. You cannot bring your own STT, cannot bring your own TTS, cannot swap the LLM. For a personal number that gets a few calls a day, "closed and expensive" is a bad corner of the tradeoff space.

Vapi. A hosted voice-agent platform. You pick STT/LLM/TTS from a menu, wire it up in their UI, and get a working number in ten minutes. Genuinely good, and I would recommend it for a team shipping an outbound sales bot. What it is not is transparent. Retry semantics, endpointing heuristics, barge-in behavior, prompt-injection defenses are all Vapi's decisions, not mine. My whole reason for building Dyx was to make those decisions myself. At low volume, pricing is also dominated by a per-minute platform fee on top of the vendor bills.

Retell AI and Bland. The same shape as Vapi with different opinions. Retell has better observability out of the box; Bland is the tightest onboarding but the hardest to modify. Both are priced for teams shipping a product, not for one engineer answering a personal number.

Vocode. Open-source Python framework, composable exactly the way I wanted. Two things pushed me off it: the phone-number transport is your problem (you bring Twilio or Telnyx and wire the SIP trunk yourself), and the ecosystem momentum in 2025 shifted noticeably toward LiveKit Agents.

LiveKit Agents. Open-source, self-hostable or run against LiveKit Cloud, pluggable STT/LLM/TTS via a plugin registry, and the same LiveKit primitives handle the WebRTC transport for the browser demo page and the SIP-to-WebRTC bridge for the actual phone number. Two problems one box. This is what I built on.

What we did differently

Against those five baselines, the specific decisions Dyx makes are worth naming.

Own the pipeline, rent the inference. OpenAI Realtime and Vapi both bundle transport and inference. LiveKit Agents unbundles them: LiveKit does the audio transport (WebRTC to the caller via a SIP gateway, gRPC-ish streams to each provider), and I pick each inference stage separately. This is the same shape of decision as running your own retrieval stack with hosted embeddings, you pay in coordination for the ability to swap components independently. On a personal number, I want the ability to try a new TTS voice without renegotiating with a platform.

WebRTC everywhere, SIP only at the edge. The caller's phone is on the PSTN, which speaks SIP. Everything inside the Dyx boundary speaks WebRTC. A LiveKit SIP gateway sits at the edge, converts the PSTN leg into a WebRTC track, and joins it to a LiveKit room. My agent process joins the same room as another participant. This means the same code path handles a real phone call and a browser-based demo at voicemail.kaushik.cv, the demo is just a WebRTC client joining the same kind of room. Managed platforms hide this seam; unbundling it made the demo page a two-day project instead of a two-week one.

Streaming, not batched, at every stage. Every hop is a streaming primitive: partial STT transcripts, streaming LLM tokens, streaming TTS PCM frames. This is the difference between "sum of stage latencies" and "max of stage latencies," and it is the whole reason the pipeline hits sub-700ms end-to-end. Any component that only exposed a batched API would break the budget.

Tools, not free-form action. The LLM does not decide what to do next in prose. It calls one of three tools, leave_message, book_call, reject_caller, and the tool schema is what shapes the caller's experience. This is where most of the guard-rail work lives: not in the prompt, in the tool interface.

System design

The end-to-end path, per component, with the technique and one number I can defend.

Caller phone → LiveKit SIP gateway. LiveKit Cloud runs a SIP endpoint. My Telnyx number forwards to it. The gateway terminates SIP, transcodes G.711/Opus, and publishes the caller's audio into a LiveKit room as a WebRTC track. Round-trip latency on the SIP leg is the least controllable stage, PSTN routing dominates, and I measure it at roughly 50-100ms depending on the caller's carrier.

LiveKit room + agent process. A Python worker subscribes to the room, receives the caller's audio track, and runs the pipeline. The agent is a stateless process, a new instance spawns per call, and the entire session state lives in memory for the duration of the call. When the call ends, the process writes a transcript to disk, emails me, and exits. Cold-start on the agent process is the operational number I still watch: a fresh worker takes about 1.5 seconds to boot Python, load the Silero VAD weights, and establish streams to the three providers. That is unacceptable to eat inside a caller-perceived-latency budget, so a worker pool keeps two hot workers ready. At one caller per day, warm pool of two is overkill; at zero warm workers the first ring sounds terrible.

Session-state management. Everything that matters lives in one in-memory Python object per call: the rolling transcript, the current turn state (listening, thinking, speaking), the tool-call history, and a small caller-context dict populated at call start from the SIP From header. There is no database in the hot path. The transcript is written once at call end. Correct at my scale; wrong at ten thousand concurrent, where the state has to be externalized so workers can migrate.

Deepgram nova-3 streaming STT. Audio flows to Deepgram over a WebSocket. Interim results start a few hundred milliseconds after speech begins, confirmed finals arrive when Deepgram's endpointer fires. I run with interim_results=True and endpointing_ms=300. Nova-3's first partial lands at roughly 150ms after speech onset in practice; hosted Deepgram is a rounding error in the budget and I stopped trying to run STT myself.

Endpoint-of-speech detection, layered. Two endpointers cooperate. Silero VAD runs locally in the agent process and gates the audio going to Deepgram, so I do not pay to transcribe silence and background noise. Deepgram's server-side endpointer decides when the caller has finished a turn. The Silero gate is tuned aggressive; Deepgram is the source of truth for turn boundaries. Layering them cut Deepgram usage cost by roughly 30% on quiet lines with no measurable effect on turn-detection quality, directional, from one week of transcript logs.

Background-noise gate. The Silero VAD doubles as a noise gate. On calls from noisy places, cars, coffee shops, hallways, the STT was interpreting ambient audio as attempted speech and firing false partial transcripts. Gating below the VAD threshold before sending to Deepgram killed that.

Retry semantics on partial STT. The pipeline treats interim transcripts as the primary signal and finals as confirmation, not the other way around. The LLM starts generating on a stable interim, an interim transcript that has not changed for 200ms, and the generation is either confirmed by the final transcript or torn down and restarted if the final diverges materially. This is the retry logic that lets the LLM start thinking before the caller has finished talking. Tear-downs are rare; the interim-to-final delta is usually small.

Groq llama-3.3-70b LLM with tool use. The LLM is llama-3.3-70b-versatile on Groq. First-token latency of roughly 150ms at around 300 tokens per second is what makes the stage fit the budget. The model runs in tool-calling mode with three tools:

{
  "tools": [
    {
      "name": "leave_message",
      "description": "Save a message from the caller for Kaushik to review.",
      "input_schema": {
        "type": "object",
        "properties": {
          "caller_name": {"type": "string"},
          "reason_for_call": {"type": "string"},
          "callback_number": {"type": "string", "description": "E.164 preferred"},
          "urgency": {"enum": ["routine", "time-sensitive", "urgent"]}
        },
        "required": ["reason_for_call"]
      }
    },
    {
      "name": "book_call",
      "description": "Offer a specific window for Kaushik to call back.",
      "input_schema": {
        "type": "object",
        "properties": {
          "caller_name": {"type": "string"},
          "callback_number": {"type": "string"},
          "preferred_window": {"type": "string", "description": "Free-text; parsed downstream"}
        },
        "required": ["callback_number", "preferred_window"]
      }
    },
    {
      "name": "reject_caller",
      "description": "End the call politely, used for robodialers and clear sales scripts.",
      "input_schema": {
        "type": "object",
        "properties": {
          "reason": {"enum": ["robocall", "sales-script", "silence", "adversarial"]}
        },
        "required": ["reason"]
      }
    }
  ]
}

The tool interface is where the guard-rails get enforced structurally. There is no share_address tool, no confirm_meeting tool, no disclose_stack tool. The model can be talked into intending to do any of those; it cannot be talked into calling a tool that does not exist. This is a different discipline than prompt-based safety: the prompt describes what the model should say, the tool schema decides what the model can actually do.

Cartesia sonic-3 streaming TTS. LLM output streams into Cartesia sonic-3. Sonic-3's first PCM frame lands at roughly 200ms after the first LLM token, which is what lets the caller hear the response before the LLM is done generating. The chunking rule that matters: I flush a TTS request every complete phrase, sentence boundary or a comma-with-clause, rather than every token. Flushing per-token produced audible seams in the audio; flushing per-full-response added 800ms to first-audio. Per-phrase, with a minimum flush size of about 60 characters, was the shape that hit the sub-700ms first-audio budget.

LiveKit publish → caller phone. Cartesia's PCM streams back into the agent, which publishes it to the LiveKit room as an outgoing WebRTC track. The SIP gateway picks it up, transcodes to G.711, and it lands at the caller's ear. This last hop is the same 50-100ms as the inbound leg, and again, not a stage I optimize.

Cost per call. Per-minute billing at current published prices, for a typical Dyx call (roughly 50% caller talking, 50% agent talking, natural conversational density):

Roughly two to two-and-a-half cents per minute of call. A ten-minute conversation costs a quarter. At the calling volume Dyx sees, the monthly bill is bounded by the fixed number rental, not the per-minute variable, the whole thing runs on well under a Starbucks coffee per month. This is the number that made the unbundled architecture worth the coordination overhead. On OpenAI Realtime, the same call is roughly nine cents a minute; on Vapi with the same underlying vendors, it is closer to five once the platform fee is included.

What broke first

The first production failure was not latency. It was the tool-call schema on reject_caller.

Version one had a single end_call tool with no reason field. The intent was: the LLM decides the call is over, the tool fires, the agent hangs up. What happened in practice was that the model would occasionally decide a legitimate but weird call was "over", a friend calling from a very loud bar sounded to the pipeline like a synthetic voice with no addressee, and the LLM, doing exactly what its prompt told it to do, would call end_call. Real callers got hung up on. I found it in the logs because the transcript for those calls was six words long.

The fix was two-part. I added the reason enum so the model has to categorize the reason for ending the call, which changes the decision from "should I hang up" to "which category of hang-up is this," and I added a hard prompt rule that ambiguous cases must default to leave_message with a note. The reason field is more than observability, it forces the model to commit to a category that can be spot-checked, and if none of the categories fit, it should not be hanging up.

The deeper lesson was about tool-schema design. A tool without a reason field lets the model be vague about why it is calling the tool. A tool with a categorized reason forces a decision. This maps directly to the same discipline as the guard-rails post: the model does not negotiate with strangers, and the tool does not accept an un-justified action.

What I would do differently

Three things I would change in a v2, in order of what I think matters most.

Externalize session state. In-memory Python session objects are fine for one caller per day. They are the wrong shape the moment I want a warm pool that can migrate workers between calls, or the moment I want to persist mid-call context to survive an agent restart. A small Redis or DynamoDB store, keyed by call SID, with the transcript and turn state serialized every few seconds, would cost almost nothing and unblock everything downstream.

A learned endpointer. The 300ms static endpointing threshold is the same nprobe problem the latency post ended on, one number that works on average and thrashes at the tails. A small classifier that consumes the last three seconds of audio and predicts "turn continuing" vs. "turn ending" would let the pipeline endpoint aggressively for fast talkers and patiently for slow ones. It is a two-week project; I have not done it because at my scale the static number is good enough.

A dedicated pre-call greeting worker. The 1.5-second cold-start is masked by the warm pool, but the warm pool costs money to keep alive. A tiny pre-call greeting worker, a stripped-down process that plays "one moment" while the real pipeline boots, would let me drop the warm pool to zero and take the cold-start on the first turn instead of the ring. The greeting is where the "thinking sound" perceptual trick already lives; extending it to cover cold-start is the natural next step.

The transferable lesson

The right architecture for a voice agent is the one that matches the shape of your call volume and your appetite for control. A single-model realtime API is right at scale and wrong for a personal number that answers a handful of calls a day. A managed platform is right for a team shipping a product and wrong for an engineer who wants to tune the endpointer. An unbundled, hand-orchestrated pipeline is expensive in coordination and cheap in dollars, and it is the corner of the tradeoff space where a personal AI product actually gets to be your own. Pick the corner honestly, name the tradeoff you are making, and design the tool schema before you write the prompt.

See also

• The <700ms latency budget for Dyx, the stage-by-stage timing this architecture was built to hit.
• Guard-rails every personal AI should have, the six protocols that sit above this pipeline before the number is safe to publish.
• CipherStack: an LRU key-rotation state machine, where the Deepgram, Groq, and Cartesia keys are vended from.
• HNSW or IVF-PQ at 2M documents, the same "unbundle for stability, fuse for latency" shape in retrieval.

Dyx is live at voicemail.kaushik.cv. Or dial +1 (484) 270-7074 and it will answer.