Choose a voice AI stack in this order: set a latency budget (about one second from end of speech to first audio), pick speech-to-speech or a cascade based on whether you need to see inside the pipeline, then select STT by your callers’ audio conditions, the LLM by rules-per-millisecond, and TTS by your utterance length. Keep every component swappable; the rankings change monthly.

That is the whole framework. The rest of this guide is the evidence for each step, with measured numbers from benchmarks.speko.ai - a live, independent benchmark that runs every provider through the same rig, on the same audio, and publishes the method. Where a competitor’s model wins, we say so, because several of them do.

One warning before the framework: almost every number you will find while researching this decision was measured on a clip, not a call. Word error rate on clean read speech. MOS on a 30-second sample. Time-to-first-token on one prompt. Real calls have accents, phone-line audio, interruptions, invoice totals, and twenty turns of accumulated context - and the model that wins the clip routinely loses the call. Every decision point below cites the call-shaped measurement, not the clip-shaped one.

Step 0: Set your latency budget first

Everything downstream is constrained by one number: how long your caller waits between finishing a sentence and hearing the agent respond. Budget roughly one second end-to-end; under about 800ms feels responsive, and every component you pick spends part of that budget.

The measured landscape, from our boards:

Pipeline stage	Fast option (measured)	Slow option (measured)
STT final transcript	~180ms after end of speech (Smallest Pulse)	1-3s if the STT waits out its own silence timer
LLM first token	92ms p50 (gemma-4-31b-it on LiveKit)	10-66s (gpt-5 under a 200-rule prompt)
Full S2S turn	494ms p50 (gpt-realtime)	1104ms p50 (gpt-realtime-2.1)
Streamed cascade, end to end	~750-900ms	multi-second if any stage is non-streaming

Sources: Pulse STT (2026-07-07), the LiveKit LLM run (2026-07-05), the guardrail budget test (2026-07-10), the multi-turn S2S board rebuild (2026-07-08), the S2S vs cascade analysis (2026-05-18).

Two practical notes. First, latency budgets are spent in silence the caller hears, so a component that is 300ms slower is not “a bit slower” - it is 300ms of dead air on every single turn. Second, measure from a fixed vantage and trust the gaps between options, not the absolute milliseconds; your callers’ distance to the datacenter rides on top of every number.

Step 1: Speech-to-speech or cascade?

Pick a cascade (separate STT, LLM, TTS) if you need to inspect what the agent heard and said, forecast cost on long calls, or swap components; pick speech-to-speech (one audio-to-audio model) if latency and naturalness are the product and your calls are short. This is an operational decision now, not an intelligence decision.

The old objection to S2S - “it can’t reason” - is dead. The top S2S models score 93-98% on Big Bench Audio, level with frontier text LLMs (Can S2S Replace the Cascade?, 2026-05-18). What did not close are three operational gaps:

Observability. A cascade exposes exact text at every hop; you know what the agent heard and can block a response before the caller hears it. S2S is a black box - the only transcript is one you generate afterward by running STT on a recording. In healthcare, finance, and legal, this alone decides the architecture.
Cost shape. Cascaded cost is flat per turn: roughly $0.0095-$0.17 per minute depending on the stack, and a 12-minute call costs six times a 2-minute call. S2S re-accumulates the full audio context every turn, so cost grows super-linearly with call length - OpenAI Realtime measured at roughly $0.18-$0.46 per minute uncached, $0.05-$0.10 with prompt caching, and the back half of a long call costs more per minute than the front half. Budget off a 2-minute demo and a 12-minute real call will surprise you.
Component swap. A cascade lets you replace the STT, LLM, or TTS independently when something better ships (and, as the rest of this guide shows, something better ships constantly). S2S is one vendor for the whole pipeline.

What S2S genuinely wins: latency (roughly 250-500ms end to end versus 750-900ms for a well-built streamed cascade - a real 2x, though vendors get to “7x” by comparing against a non-streamed cascade, which is a strawman), naturalness (prosody and emotion never get flattened to text), and native barge-in.

Dimension	S2S	Cascade
End-to-end latency	~250-500ms	~750-900ms streamed
Reasoning	At parity (93-98% Big Bench Audio)	At parity
Observability	Post-hoc transcript only	Exact text at every stage
Cost on long calls	Super-linear growth	Flat per turn
Component swap	Locked to one vendor	Any STT / LLM / TTS
Naturalness / prosody	Preserved natively	Flattened at the STT boundary

The pattern winning in production is hybrid: S2S for fast, simple turns; a cascaded path for complex, regulated, or tool-heavy ones.

If you go S2S: which model

The current answer, measured on a live 10-turn conversation with a tool call, a mid-utterance barge-in, and a late memory probe (n=15 sessions from us-east4, 2026-07-08): gpt-realtime, at 494ms p50 conversational latency (p90 768ms, p95 870ms), 100% task success, and a clean sweep of all eight deterministic capability checks on a separate hard 20-turn concierge call. If cost matters more than the last half-second, gpt-realtime-2.1-mini scores 0.96 to the flagship’s 1.00 on that concierge call at roughly a third of the price - the catch is mid-pack latency at 1011ms p50.

Two findings from that rebuild worth more than the ranking (fastest and best value, both 2026-07-08):

Newer is not better. gpt-realtime-2 (1103ms) and gpt-realtime-2.1 (1104ms) run about 2x slower than the original gpt-realtime (494ms) with a rounding-error quality difference - and gpt-realtime-2 dropped the connection in 2 of 3 hard-scenario calls.
Do not let an LLM judge pick for you. When we graded the same nine models with an LLM judge - the way most voice leaderboards do - it rated the entire field 0.90-1.00 and gave a perfect 1.00 to a model that dropped two-thirds of its calls. Deterministic, code-level checks spread the same field across 0.55-1.00. If a leaderboard’s methodology says “judged by GPT-x”, discount it.

Live board: benchmarks.speko.ai/s2s.

Step 2: Choose STT by your callers’ conditions, not by clean WER

Pick your STT by testing on the audio your callers actually produce - accents, phone lines, background noise, domain vocabulary, spontaneous speech - because the ranking on clean read speech does not survive contact with any of them. There is no single best STT; there is a best STT for your distribution.

We ran six streaming STT models through the same gateway across five English production conditions, scoring word error rate on the streamed final - the exact text a live agent consumes, not a tidy batch re-transcription (What a Voice Agent Actually Hears, 2026-07-11; 1,800 streamed transcriptions). What it showed:

Qwen3-ASR and Smallest Pulse lead, tied at 9.9% average WER across all five conditions. Neither is the loudest brand in the category.
The model claiming the #1 streaming word error rate landed fourth on this mix. Cartesia’s Ink-2 is a genuinely strong model, and their launch post honestly tests on accented call-center audio - but “best on fourteen English accents” and “best on the full mess a phone agent sees” turn out to be different sentences.
The phone line and the noise floor barely matter. Telephony and noisy conditions stay close to accented for almost every model; Deepgram even ticks slightly down on the phone. The acoustic problems everyone worries about are the ones modern streaming models handle best.
Content is what separates the field. Medical vocabulary spreads the models from 8.2% WER (Qwen3-ASR) to 19.1% (Soniox). Spontaneous speech - a real human talking over themselves - roughly doubles every model’s WER and reshuffles the ranking.

Condition	What we measured	Spread
Accented (VoxPopuli)	baseline production condition	leaders ~9-10%
Telephony (same clips, 8kHz)	barely moves vs accented	near-flat for most models
Noisy (same clips, degraded)	barely moves vs accented	near-flat for most models
Medical vocabulary (MultiMed)	the vocabulary test	8.2% (Qwen3-ASR) to 19.1% (Soniox)
Spontaneous (People’s Speech)	the great equalizer	every model’s WER roughly doubles

Treat differences of about one WER point as noise at this sample size (60 clips per condition). Full interactive breakdown: benchmarks.speko.ai/streaming-stt; clean-speech board: benchmarks.speko.ai/stt.

The second STT clock: latency

Accuracy is half the decision. A caller never experiences your WER; they experience the pause after they stop talking. Watch two clocks: time to the first partial (how early the LLM can start reasoning) and time to the committed final after end of speech (the dead air before the turn even begins). Smallest AI’s Pulse is the rare model excellent at both: ~64ms to the first partial and ~180ms to the final transcript after the caller stops, while holding 5.1% WER on English (FLEURS, n=50) - versus the 1-3 seconds you eat when an STT waits out its own silence timer (Pulse, 2026-07-07). An STT that is 2 points more accurate but a second slower to finalize is usually the wrong trade for a live agent.

Step 3: Choose the LLM by rules-per-millisecond

For a voice agent, pick the model that holds the most instructions per millisecond, not the smartest one. The best reasoning model on the market is unusable on a phone call, and the model that aces the leaderboard may go silent or invent facts on a live turn.

Three measured findings drive this step.

1. Every model has a guardrail budget, and it is lower than you think. Voice agents carry the longest system prompts of anything - do not reveal you are an AI, never quote a price, one question at a time. We stacked 20 to 200 rules into a system prompt and counted how many each model still obeyed (Your System Prompt Has a Budget, 2026-07-10):

Model	Guardrail budget (@95% obeyed)	Latency under load
gpt-5	~200 rules	10-66s per reply - unusable live
cerebras gemma-4-31b	~150 rules	~400-540ms, flat to 200 rules
cerebras gpt-oss-120b	~50 rules	fast, erratic past 100
gpt-5-nano	~50 rules	fast; 19% obeyed at 150 rules

The winner for voice was the small open model: only gemma-4-31b was both accurate and fast. And when a weaker model runs out of budget, it drops the later rules first - at 150 rules, gpt-5-nano obeyed 51% of the first third of the list and 1% of the last third. On a cheap model, put the guardrails you cannot afford to lose at the top of the prompt.

2. First token speed varies 6x across serving hosts. On LiveKit’s inference host, gemma-4-31b-it returned a first token at 92ms p50 and gpt-oss-120b at 118ms - the same gpt-oss-120b served at ~195ms on Cerebras - while gpt-4.1 sat at 422ms and gpt-4.1-mini at 524ms (LiveKit Inference run, 2026-07-05). Where a model is served matters as much as which model it is.

3. A 100% leaderboard score hides call-breaking behavior. Two Cerebras models - gemma-4-31b and gpt-oss-120b - both post a perfect 100% overall, 100% tool-calling, 100% grounded on our LLM board. Stress-tested past the score (Both Models Scored 100%, 2026-07-09), they split hard: asked for store hours it was never given, gpt-oss-120b confidently invented “Mon-Fri 9-6, Sat 10-4” three runs out of three, while gemma refused and escalated 3/3. And across 27 adversarial turns, gpt-oss-120b fired its tools in total silence on 16 of them - dead air where gemma spoke a bridging line (“let me pull that up”) on all but 2. Both resisted every jailbreak and prompt-extraction attempt; neither leaked its reasoning. The lesson generalizes: a pass rate is a summary, and it drops whether the words were true and whether there were any words at all.

One more spend-your-skepticism note: on the LiveKit dial-discipline run, most models passed everything except one behavior - five of ten models hang up in silence instead of saying goodbye (0/3 on farewell-plus-end_call). It is a UX wart you patch downstream by enforcing a closing line in the worker, but it is the kind of thing no leaderboard number will ever show you.

Live board: benchmarks.speko.ai/llm.

Step 4: Choose TTS on three axes - naturalness, length, and content

Pick TTS last and pick it on three separate axes: how natural it sounds on a short clip (what every leaderboard measures), whether it stays the same voice at length (drift), and whether it can read your actual content - numbers, currency, confirmation codes - reliably. The #1 and #2 arena-ranked models each fail one of the axes the arena does not measure.

Axis 1: Short-clip naturalness is real, but it is 30 seconds

The Artificial Analysis Speech Arena - blind human A/B votes on ~30-second clips - currently ranks Cartesia Sonic 3.5 first (Elo 1218), Gemini 3.1 Flash TTS second (1209), and ElevenLabs Eleven v3 third (1184). The methodology is sound. The quiet assumption is clip length.

Axis 2: Length breaks models the arena ranks highly

We ran the same 9,184-character script through all three top-ranked models for roughly ten minutes and scored twelve acoustic and ASR signals per 3-second window (the 10-minute test, 2026-05-23):

Gemini 3.1 Flash TTS - the #2 model in the world on a 30-second clip - logged 353 severe-tier failure windows in an 8:34 take. The other two combined logged zero. It stops sounding like itself around minute one and never recovers; Whisper broke its audio into 249 segments versus 108 (Eleven v3) and 93 (Sonic 3.5). Excellent for short utterances, unproven past a minute.
Between the two that held: Sonic 3.5 wins voice-identity stability across the take (speaker-embedding cosine 0.82 +/- 0.04 versus ElevenLabs 0.72 +/- 0.06), ElevenLabs wins phrase-level phonation cleanliness (jitter 2.05% vs 2.26%, harmonic-to-noise 14.1dB vs 11.2dB). Pick by what you ship: single-call long-form favors Sonic, short expressive narration favors Eleven v3.
Operational footnote: ElevenLabs’s REST API caps input at 5,000 characters, so long takes mean chunk-and-stitch, and the chunk boundaries are visible in the analyzer.

And within one provider, the voice you pick matters more than the model. Cartesia Sonic 3.5 ships 378 English voices; we sampled 50 and ran a 90-second drift probe on each (Cartesia drift, 2026-05-25). Only 5 of 50 - 10% - stayed above the perceptual same-speaker threshold (cosine >= 0.70 at 60 seconds). The default voice drifts to 0.50 - audibly a different person by minute one - while “British Lady” holds 0.73 on identical text through the identical model. The catalog is not labeled for this. If a take can run past a minute, probe the specific voice you plan to ship.

For turn-based voice agents speaking 1-5 second utterances, drift never accumulates and this axis mostly drops out - which is exactly why you decide it per use case instead of from a brand ranking.

Axis 3: Content - make it read an invoice before you sign

The cheapest streaming TTS in our gateway is Alibaba’s qwen3-tts-flash at $0.0014 per minute - roughly a third of the next-cheapest streaming option and ~30x under ElevenLabs v3. On clean prose it is genuinely good. Then we asked it to read “$2,349.99” eight times (The .99 Problem, 2026-07-09): it never once read the amount correctly - the cents came back as “.999” five times, the dollar sign landed as a euro in half the takes - while xAI’s grok-tts, at 3x the price ($0.0045/min), read it correctly all four times. Under 20-way concurrency, Qwen completed 5 of 20 requests (the rest hit upstream 429s); Grok completed 20 of 20 in under two seconds. “Cheapest” is a real answer, but it comes with a sentence attached: do not hand it a price tag or a rush.

Also in the fine print: SSML is not uniformly safe. On Speechify’s simba-3.2, a modest <prosody pitch="+15%"> tag dropped speaker similarity from a 0.63 same-voice baseline to 0.28, and +/-40% collapsed it to ~0 - the tag does not raise the pitch of your branded voice, it silently swaps the speaker, on a one-second utterance (The Pitch Trap, 2026-07-07). Rate, volume, and break tags stayed on the same-voice baseline (0.60 and 0.69). If your agent has a branded voice, treat pitch tags as a voice change until you have probed your provider.

Live board: benchmarks.speko.ai/tts.

Step 5: Turn-taking is a component too - and the recommended setting loses

Do not accept your platform’s default or documented-as-smarter turn-detection mode; measure it against backchannels (“uh huh”, “mm hmm”), because that is where callers get cut off, and the marketing points the wrong way. Turn-taking is the component nobody lists on the invoice and everybody hears.

We ran 158 trials against gpt-realtime-2 across OpenAI’s two turn-detection modes and seven stimulus types (The Backchannel Trap, 2026-05-29). The docs recommend semantic_vad - a word-aware classifier - for natural conversation. Measured: server_vad with threshold 0.8, the dumb energy detector, absorbed every quiet backchannel we threw at it; semantic_vad cancelled the agent at every eagerness setting it exposes. Both modes correctly absorbed a cough, both correctly cancelled on a real “wait, stop” (in 350-430ms). The only regime where the mode choice changes the outcome, the recommended mode loses every cell.

The honest bottom line from that test: any backchannel spoken at conversational volume cancelled the agent at every configuration of both modes. If your agent will hold 1:1 dialogue with people who say “uh huh” while listening - most agents - neither built-in mode fully solves it, and you should evaluate purpose-trained turn-detection models on this exact distinction. Board: benchmarks.speko.ai/turntaking.

Step 6: Cost the solved call, not the components

Compare stacks on dollars per solved call - where “solved” means the booking tool actually fired, not that the agent said “you’re booked” - because per-minute rate cards and per-token prices do not predict the bill. Turn count, tool discipline, and STT accuracy dominate sticker price.

When we wired full stacks into a real booking scenario with a live scheduling tool and counted a solve only when the tool fired (Everyone Measures the Clip, 2026-07-05):

The cheapest-looking stack “resolved” in 1.7 turns because the model confidently said “you’re booked!” and hung up without ever calling the tool. Nothing was booked. The metric loved it.
The cheapest honest stack - Deepgram nova-3, Cerebras gpt-oss-120b, Cartesia sonic-3.5 - solved for $0.058 per grounded booking, 100% of the time.
The all-brand premium stack - ElevenLabs STT, GPT-4.1, ElevenLabs TTS - cost $0.703 per solve. About 12x more for the same outcome.

A “premium brain” that clarifies for ten turns costs more than a fast one that books in four, even at a lower per-token price. Cost your stack in units of solved calls or the number means nothing. Board: benchmarks.speko.ai/stacks.

Step 7: Keep the stack swappable

Whatever you pick today will be the wrong pick within months, so the most important property of your architecture is that any component can be replaced in a config line, not a rewrite. This is the one structural decision that outlives every ranking in this guide.

The evidence is this guide itself. Within eight weeks of measurements: the fastest S2S model changed when the probe got longer; the STT leader on production audio is not the STT leader on clean audio; the #2 arena TTS fails at length; the newer gpt-realtime tiers regressed against the original. If swapping a component means re-integrating an SDK, you will still be running February’s winner in September.

Practically, that means either building a thin abstraction over provider APIs yourself, or using a gateway that already normalizes them. (Disclosure and the one plug in this piece: Speko is a voice AI gateway - every measurement above ran through it, and on it a model swap is a one-line config change. That is also why we publish these benchmarks: routing decisions are only as good as the data behind them.) Whichever way you get it, the requirement is the same: STT, LLM, TTS, and turn-detection each replaceable independently, with your own eval suite as the gate.

How to run your own 10-minute test

Do not take our rankings - or anyone’s - on faith. The whole framework compresses into an afternoon of testing on your own traffic shape:

Collect 20-50 real audio samples from your actual channel: your phone line’s codec, your callers’ accents, your domain vocabulary. If you have no calls yet, record staff reading real scripts over a phone line, not a studio mic.
STT: stream those clips through 3-4 candidates and score WER on the streamed finals, not batch re-transcription. Log two latencies per clip: first partial, and final-after-end-of-speech. (This is exactly the harness behind /streaming-stt; our conditions and datasets - VoxPopuli, MultiMed, People’s Speech - are listed there if you want to mirror them.)
LLM: write your real system prompt, count its rules, then test each candidate at that rule count plus fifty. Ask it three questions it has no tool or context to answer and check whether it refuses or invents. Measure TTFT from your serving region, not from the vendor’s status page.
TTS: synthesize your ten hardest real utterances - amounts with cents, confirmation codes, names, your longest expected take. Round-trip them through an open ASR (we use faster-whisper large-v3) and diff. If any take can exceed a minute, run a 90-second generation on the exact voice ID you will ship and check it still sounds like the same person at the end (speaker-embedding cosine >= ~0.7 to the opening, or just use your ears).
The call test: wire the winning components together, give the agent one real tool, and run ten end-to-end conversations that include an interruption, a backchannel, and a request the agent must refuse. Count grounded solves - tool actually fired - and divide your total spend by that count.
Re-run monthly. Every number in this guide has a date on it because every number changes.

If a vendor’s number cannot survive this afternoon, it was a marketing number.

The framework on one screen

Decision	Rule	Key measured fact
Latency budget	~1s end of speech to first audio; allocate per stage	STT finals range 180ms to 3s; LLM TTFT 92ms to 66s
S2S vs cascade	Cascade for observability, long calls, swap; S2S for latency + naturalness on short calls	S2S ~250-500ms vs cascade ~750-900ms; S2S cost grows super-linearly
S2S model	gpt-realtime first; 2.1-mini if cost-bound	494ms p50, 8/8 checks vs 0.96 at a third of the price
STT	Test on your conditions; content beats channel	Qwen3-ASR / Pulse 9.9% avg; medical spreads 8.2-19.1%; spontaneous doubles WER
LLM	Rules-per-millisecond; stress past the pass rate	gemma-4-31b: ~150 rules at ~540ms; gpt-5: ~200 rules at 10-66s
TTS	Naturalness AND length AND content, per use case	#2 arena model: 353 severe windows at length; cheapest TTS: 0/8 on “$2,349.99”
Turn-taking	Measure against backchannels; distrust the recommended mode	server_vad 0.8 absorbs quiet backchannels; semantic_vad cancels at every setting
Cost	Dollars per grounded solve	$0.058 vs $0.703 for the same booking - 12x
Architecture	Every component swappable in config	The S2S speed leader changed within one month of measurement

FAQ

How do I choose a TTS provider for a voice agent?

Test three things on your own content: naturalness on a short clip, voice stability at your longest expected take, and reliability on numbers and currency. These rank differently - the #2 arena-ranked model (Gemini 3.1 Flash TTS) fails past one minute, and the cheapest streaming TTS (qwen3-tts-flash, $0.0014/min) misread “$2,349.99” in eight of eight takes. There is no single winner, only a winner per use case.

How do I choose an STT for voice agents?

Ignore clean-speech WER and test streaming accuracy on your callers’ actual conditions. On five production conditions, Qwen3-ASR and Smallest Pulse led at 9.9% average WER while a model claiming the #1 streaming WER placed fourth. Phone-line audio and noise barely move modern models; domain vocabulary and spontaneous speech are what separate them. Also measure final-transcript latency: 180ms versus 2s is the feel of the whole agent.

Which LLM is best for voice agents in 2026?

The one that holds your rule count within your latency budget. Measured: Cerebras gemma-4-31b held ~150 system-prompt rules at 400-540ms, the best rules-per-millisecond we have tested; gpt-5 held ~200 rules but took 10-66 seconds to answer; gpt-5-nano collapsed at 50 rules. Then stress-test the finalists for fabrication and silent tool calls - two models with identical 100% board scores split hard on both.

Is speech-to-speech better than a cascaded STT-LLM-TTS pipeline?

S2S is faster (roughly 250-500ms versus 750-900ms streamed cascade) and more natural, and its reasoning now matches text LLMs (93-98% on Big Bench Audio). Cascades still win observability (exact text at every hop), flat per-turn cost on long calls, and independent component swap. Regulated or long-call workloads stay cascaded; short latency-critical calls suit S2S; the emerging production pattern is a hybrid of both.

Can I trust vendor benchmarks for voice AI?

Treat them as claims, not measurements. Vendors choose the test that flatters them: 30-second clips hide long-form collapse, clean audio hides production WER, LLM-judged leaderboards rated nine S2S models 0.90-1.00 while a deterministic scorer spread them 0.55-1.00. Prefer independent benchmarks that publish method, dates, and raw failure cases - then spend one afternoon reproducing the two or three numbers your decision actually hangs on.

Every number in this guide comes from a dated, published run at benchmarks.speko.ai - method, caveats, and raw failure cases included. The boards are live and re-measured; when the data changes, the recommendations above change with it.

How to Choose Your Voice AI Stack (2026)