Did the agent actually finish what the caller asked for — look the order up, issue the refund, book the appointment?

OpenAIgpt-4.1-mini95.91%

OpenAIgpt-5.4-mini95.32%

Gemini3.5-flash-lite92.11%

BasetenDeepSeek-V4-Flash-073191.23%

OpenAIgpt-4.191.23%

AnthropicClaude Sonnet 589.18%

BasetenNemotron-3-Ultra88.60%

OpenAIgpt-5.6-terra87.72%

TogetherDeepSeek-V4.1-Flash86.84%

Gemini3.7-flash86.84%

xAIGrok-4.386.84%

CerebrasQwen3.8-27B86.55%

Basetengpt-oss-120b85.96%

Gemini3.8-flash85.09%

Cerebrasgpt-oss-120b84.65%

BasetenDeepSeek-V4.1-Flash82.46%

InceptionMercury Voice82.46%

OpenAIgpt-5.6-luna80.70%

OpenAIgpt-5.280.41%

OpenAIgpt-5-mini79.82%

AnthropicClaude Haiku 4.578.65%

OpenAIgpt-5-nano76.75%

OpenAIgpt-572.81%

TogetherLlama-3.3-70B49.56%

Alibabaqwen-turbo46.64%

0%25%50%75%100%

With no line in the prompt naming a language, how much of what the agent says comes back in the caller's language rather than English.

AnthropicClaude Sonnet 5100%≥82%

TogetherDeepSeek-V4.1-Flash100%≥82%

Gemini3.7-flash100%≥82%

Gemini3.8-flash100%≥82%

OpenAIgpt-4.1100%≥82%

OpenAIgpt-5-mini100%≥82%

OpenAIgpt-5.2100%≥82%

OpenAIgpt-5.4-mini100%≥82%

OpenAIgpt-5.6-terra100%≥82%

TogetherLlama-3.3-70B100%≥82%

InceptionMercury Voice100%≥82%

OpenAIgpt-5.6-luna98%

OpenAIgpt-596%

Gemini3.5-flash-lite93%

Alibabaqwen-turbo86%

CerebrasQwen3.8-27B85%

OpenAIgpt-4.1-mini74%

AnthropicClaude Haiku 4.573%

BasetenDeepSeek-V4.1-Flash69%

OpenAIgpt-5-nano53%

BasetenNemotron-3-Ultra20%

BasetenDeepSeek-V4-Flash-073111%

Basetengpt-oss-120b2%

Cerebrasgpt-oss-120b0%≤18%

xAIGrok-4.30%≤18%

0%25%50%75%100%

The agent says it will do something, then goes quiet without doing it. The caller has to ask again.

gpt-4.14%

gpt-4.1-mini6%

DeepSeek-V4-Flash-07317%

DeepSeek-V4.1-Flash7%

gpt-5.4-mini11%

gemini-3.5-flash-lite11%

gemini-3.7-flash11%

gpt-oss-120b11%

gpt-oss-120b11%

Nemotron-3-Ultra13%

Qwen3.8-27B13%

Mercury Voice13%

gpt-5.6-terra15%

gemini-3.8-flash15%

gpt-5-nano17%

Claude Sonnet 519%

Grok-4.319%

gpt-5-mini19%

DeepSeek-V4.1-Flash20%

gpt-5.6-luna20%

gpt-5.220%

gpt-526%

Claude Haiku 4.533%

qwen-turbo63%

Llama-3.3-70B70%

0%25%50%75%100%

How often the caller hears nothing at all — every turn counted. The small number is the narrower measure: the share of tool calls made in complete silence, which is pinned at 100% for about half the rows in every language and so cannot rank them.

Claude Haiku 4.50%0% of tool calls ≤19%0 of 148 turns · ≤18% · 19 independent

Claude Sonnet 536%56% of tool calls61 of 168 turns

qwen-turbo38%83% of tool calls43 of 112 turns

Llama-3.3-70B50%100% of tool calls ≥82%60 of 120 turns

DeepSeek-V4-Flash-073151%75% of tool calls92 of 182 turns

Qwen3.8-27B56%81% of tool calls112 of 199 turns

DeepSeek-V4.1-Flash60%85% of tool calls119 of 198 turns

gpt-4.1-mini61%99% of tool calls96 of 157 turns

Grok-4.361%100% of tool calls ≥82%93 of 153 turns

gpt-4.162%99% of tool calls100 of 161 turns

DeepSeek-V4.1-Flash62%91% of tool calls117 of 188 turns

gpt-5.4-mini64%100% of tool calls ≥82%106 of 166 turns

gemini-3.7-flash64%100% of tool calls ≥82%109 of 169 turns

gemini-3.5-flash-lite65%100% of tool calls ≥82%111 of 171 turns

gpt-5.6-terra65%100% of tool calls ≥82%112 of 172 turns

Nemotron-3-Ultra66%98% of tool calls120 of 183 turns

gemini-3.8-flash66%100% of tool calls ≥82%116 of 176 turns

gpt-5.6-luna68%98% of tool calls129 of 191 turns

gpt-568%99% of tool calls124 of 183 turns

gpt-5-mini69%100% of tool calls ≥82%130 of 188 turns

gpt-5.270%100% of tool calls ≥81%137 of 196 turns

gpt-oss-120b72%100% of tool calls ≥82%143 of 200 turns

gpt-oss-120b72%100% of tool calls ≥82%146 of 203 turns

Mercury Voice72%100% of tool calls ≥82%144 of 200 turns

gpt-5-nano76%100% of tool calls ≥82%143 of 189 turns

0%25%50%75%100%

The agent states a fact it was never given — an opening time, a delivery window, a policy. It sounds confident and it is invented. Measured on the run without the language line, the only pass that ran these probes.

6 models0 of 30≤31% · 10 independent0%

Claude Haiku 4.5Claude Sonnet 5DeepSeek-V4.1-Flashgpt-5.6-lunagpt-5.6-terraGrok-4.3

Nemotron-3-Ultra21 of 3070%

qwen-turbo21 of 3070%

gemini-3.5-flash-lite12 of 3040%

gpt-4.1-mini10 of 3033%

gpt-59 of 3030%

gpt-oss-120b8 of 3027%

gpt-oss-120b7 of 3023%

gpt-5-nano6 of 3020%

gpt-4.15 of 3017%

Qwen3.8-27B2 of 307%

gpt-5-mini2 of 307%

DeepSeek-V4-Flash-07311 of 303%