Speech-to-Speech
Benchmarks
XYAction
higher is betterDid the agent book or add the thing only after the caller said yes, with the right city, date, time and amount, including corrections made mid-sentence.
Booked before yes
lower is betterShare of bookings written before the caller confirmed. The prompt says to read the date back and wait for a yes.
Call completion
higher is betterShare of the scripted turns that came back with speech or a tool call. A missing turn is a caller left waiting.
Task capability
higher is betterSix checks across one call: right tool (or a specific question when vague), no tool on small talk, honest about what happened, no invented prices, latest-value recall, brevity.
First sound
lower is betterWhen the model starts talking, measured from US East. On a duplex model that is often filler — "let me check that" — not the answer.
Answer
lower is betterWhen the caller hears the answer they asked for, measured from US East.
gpt-live-1 · full duplex
It listens while it speaks, so it answers the way a person on a phone does: it says something the moment it has something to say, then keeps talking once it knows more.
What we tested
| Scenario | The story it forces | Tool |
|---|---|---|
| default | book a flight to your 3pm-meeting city, then flip the day | book_flight |
| travel_support | infer the wedding city, book there, flip Thu→Sat, spell an email | book_flight |
| restaurant_calendar | infer the anniversary night, make the reservation, move 7→8pm | create_event |
| finance_currency | infer trip budget + currency, convert it, then bump the budget | convert_currency |
| refund_negative | negative-heavy support; infer the one free pickup day, schedule it, spell an order code | create_event |
| multiconstraint_schedule | infer the summit city, book a flight, then change destination Berlin→Munich | book_flight |
| Model | Mean | Default | Travel | Restaurant | Finance | Refund | Multi-con. |
|---|---|---|---|---|---|---|---|
| gpt-live-1 | 0.91 | 0.93 | 0.91 | 0.93 | 0.94 | 0.93 | 0.81 |
| grok-voice-3.0 | 0.90 | 0.99 | 0.70 | 0.90 | 0.98 | 0.82 | 0.99 |
| phonic-1-high | 0.87 | 0.77 | 0.85 | 0.90 | 0.98 | 0.91 | 0.81 |
| gpt-realtime-mini | 0.86 | 0.88 | 0.80 | 0.88 | 0.99 | 0.79 | 0.84 |
| gpt-realtime-2 | 0.86 | 0.86 | 0.82 | 0.91 | 0.96 | 0.74 | 0.87 |
| gpt-realtime-2.1 | 0.84 | 0.81 | 0.83 | 0.89 | 0.95 | 0.79 | 0.76 |
| grok-voice-think-fast-2.0 | 0.82 | 0.82 | 0.81 | 0.93 | 0.92 | 0.80 | 0.66 |
| gpt-realtime | 0.82 | 0.82 | 0.68 | 1.00 | 0.95 | 0.70 | 0.78 |
| eleven_realtime_v1_max (cascade) | 0.80 | 0.68 | 0.71 | 0.89 | 0.98 | 0.77 | 0.79 |
| gpt-realtime-2.1-mini | 0.78 | 0.78 | 0.60 | 0.87 | 0.89 | 0.75 | 0.76 |
| gemini-3.8-live | 0.78 | 0.69 | 0.66 | 0.96 | 0.94 | 0.71 | 0.69 |
| gemini-3.8-live-extended-thinking | 0.71 | 0.73 | 0.55 | 0.60 | 0.95 | 0.67 | 0.78 |
| eleven_realtime_v1_mini (cascade) | 0.69 | 0.52 | 0.69 | 0.72 | 0.98 | 0.57 | 0.69 |
| gemini-3.1-flash-live | 0.67 | 0.58 | 0.53 | 0.63 | 0.92 | 0.71 | 0.65 |