Field notes from
the voice layer
The parts of voice AI that fail in production — latency, endpointing, provider selection, benchmark methodology — and stories from teams shipping real agents.
Your Voice Agent Booked the Wrong Name. WER Said It Was 95% Accurate.
A voice agent only acts on a few words in any sentence — the ones that become tool-call arguments. We measured how often real speech-to-text corrupts them, why Word Error Rate hides it completely, and the verification layer that stops an agent from silently booking the wrong name.
Read the storyMore from the blog
Fast or Natural? Cartesia Sonic-3.5 Refuses to Pick.
In our TTS reliability probe, Cartesia Sonic-3.5 posted the lowest median first-audio of anything we measured — 126ms, network-free — while sitting at the top of the Artificial Analysis naturalness arena, above ElevenLabs v3. The rare part isn't the speed. It's that the speed comes with the quality. Here are the numbers, the caveats, and three clips so you can judge for yourself.
Your Voice Agent's LLM Speaks Spanish. That Doesn't Mean It Follows the Rules in Spanish.
Tool-calling and grounding are treated as language-agnostic — the model emits the same JSON whether the caller speaks English or Indonesian. The format is language-agnostic. The error rate is not. Published evidence and a preliminary look at where voice-agent LLMs quietly stop following the rules once you leave English.
How We Benchmark TTS: Gate, Profile, Rank
A single naturalness score hides the two failures that matter, so we don't publish one. Our TTS benchmark is three stages — a hard intelligibility gate, a multi-axis acoustic profile, and a nativeness ranking that now agrees with a native speaker at 0.99 Spearman. Here's the whole pipeline, including the stage that used to require a human.
Ranking TTS Nativeness with Content-Matched FAD
For years the only honest answer to 'which native-sounding voice is most native' was a human ear. This is the metric that changed that: a content-matched Fréchet Audio Distance over a commercial encoder ensemble that agrees with a native Thai speaker at 0.988 Spearman — the deep dive on Stage 3 of our TTS benchmark.
How Speko Benchmarks TTS: A Gate, Then a Profile
No single number ranks a synthetic voice, so we don't publish one. We run a two-stage pipeline — first prove the speech is intelligible with Whisper-large-v3, then profile how it actually sounds: harmonics, micro-stability, and prosody. Here's the method, in waveforms and spectrograms.
The audio context isn't really 128K tokens
It's about 5 minutes of conversation. Past that, the model accepts your audio and returns nothing. We pushed gpt-realtime-2 through a 60-turn English protocol; it crashed at turn 32, 5 minutes of accumulated assistant audio in. gpt-realtime ran the same prompts comfortably.
Semantic Was Supposed to Be the Smart One. It Lost.
OpenAI's Realtime API gives you two turn-detection modes. server_vad is the energy detector from the 1990s. semantic_vad is the word-aware classifier the docs recommend for natural conversation. We ran 158 trials against gpt-realtime-2 across both modes and seven stimulus types. server_vad with threshold 0.8 absorbs every quiet backchannel we threw at it. semantic_vad never absorbs anything. The recommended mode is worse, at every eagerness setting, in the only acoustic regime where the choice matters.
How Anglicized Is Your TTS? Measuring Phonological Authenticity Across 7 Providers
The anglicization index measures how often a TTS model substitutes a target-language phoneme with its nearest English neighbor. We applied it to 7 providers across 4 Southeast Asian languages and surface the per-language rankings, the largest spread we measured, and what the numbers sound like.
The Cartesia Drift: 10% of Voices Hold the Line for a Minute. The Other 90% Don't.
Cartesia Sonic 3.5 ships 378 English voices. We sampled 50 and ran a long-form drift probe. Only 5 of them stay above the perceptual same-speaker threshold at 60 seconds. The default voice Speko gateway pins is not one of them.
Artificial Analysis Ranks Gemini 3.1 Flash TTS #2. We Asked It for Ten Minutes.
Google's Gemini 3.1 Flash TTS sits at #2 on the Artificial Analysis Speech Arena — Elo 1209, behind only Cartesia Sonic 3.5 (1218) and ahead of ElevenLabs Eleven v3 (1184). The Arena scores blind 30-second clips. We ran a ten-minute take. At length, the model ranked second is the only one of the three that breaks.
Vendors Say They Support 99 Languages. They Don't.
Voice-AI vendors compete on language counts the way phone makers used to compete on megapixels. The published benchmarks, vendor docs, and developer forums tell a quieter story: 'supported' is a marketing word, and the floor it hides is lower than anyone says out loud.
Speech-to-Speech Got Smart. It Still Can't Replace the Cascade.
Speech-to-speech models closed the reasoning gap with text LLMs in 2026. The gap that's left — observability, cost predictability, component swap — is the one that actually decides production architecture.
Cutting voice agent latency to sub-500ms — a practical playbook
A latency budget for cascaded voice pipelines, why endpointing is the silent killer, and where the architecture itself has to change when you cannot push lower.
We Tried to Break Four Voice Agents with a Cough. We Failed.
We ran the same 400 ms cough into four production voice-agent stacks — OpenAI Realtime default and tuned, cascaded Deepgram Nova-3, cascaded ElevenLabs Scribe v2. Four for four absorbed it without yielding. Here are the clips, and the engineering that explains why.
Evaluating voice ai quality in production — beyond WER
Your benchmark says 5% WER. Your users say the agent can't understand them. Both are correct. The metrics that actually predict production failures.
A developer's framework for picking an stt provider
Six axes that decide whether your product ships — accuracy, latency, language coverage, cost, API ergonomics, vocabulary tolerance — with the tolerance thresholds we use to route traffic.
Streaming vs batch stt — when each one wins
Most of you shouldn't be using streaming STT. Four questions to answer honestly before you open another WebSocket.
Building voice ai for noisy real-world audio
Noise is not one thing — it's four. A field guide to suppression, SNR thresholds, and the model choice that survives where your users actually live.
No posts match that — try a different search or tag.
Ship your own voice agent.
Describe it in plain English — live in under five minutes.