Did the agent actually finish what the caller asked for — look the order up, issue the refund, book the appointment?

OpenAIgpt-5.6-luna92.98%

OpenAIgpt-4.1-mini92.69%

BasetenDeepSeek-V4-Flash-073191.23%

Gemini3.5-flash-lite89.47%

OpenAIgpt-4.189.47%

BasetenNemotron-3-Ultra89.47%

TogetherDeepSeek-V4.1-Flash88.89%

OpenAIgpt-5.4-mini88.30%

Gemini3.7-flash87.72%

OpenAIgpt-5-mini87.13%

OpenAIgpt-5.286.84%

OpenAIgpt-5.6-terra85.96%

CerebrasQwen3.8-27B85.38%

Gemini3.8-flash85.09%

xAIGrok-4.385.09%

AnthropicClaude Haiku 4.584.36%

InceptionMercury Voice84.06%

BasetenDeepSeek-V4.1-Flash79.82%

AnthropicClaude Sonnet 579.53%

Basetengpt-oss-120b79.39%

Cerebrasgpt-oss-120b79.39%

OpenAIgpt-5-nano78.65%

OpenAIgpt-577.19%

TogetherLlama-3.3-70B40.35%

Alibabaqwen-turbo33.77%

0%25%50%75%100%

With no line in the prompt naming a language, how much of what the agent says comes back in the caller's language rather than English.

AnthropicClaude Sonnet 5100%≥82%

TogetherDeepSeek-V4.1-Flash100%≥82%

Gemini3.7-flash100%≥82%

Gemini3.8-flash100%≥82%

OpenAIgpt-4.1100%≥82%

OpenAIgpt-5-mini100%≥82%

OpenAIgpt-5.2100%≥82%

OpenAIgpt-5.4-mini100%≥82%

OpenAIgpt-5.6-luna100%≥82%

InceptionMercury Voice100%≥82%

OpenAIgpt-598%

OpenAIgpt-5.6-terra97%

Alibabaqwen-turbo87%

CerebrasQwen3.8-27B86%

OpenAIgpt-5-nano85%

Gemini3.5-flash-lite83%

BasetenDeepSeek-V4.1-Flash68%

AnthropicClaude Haiku 4.532%

OpenAIgpt-4.1-mini27%

BasetenNemotron-3-Ultra25%

BasetenDeepSeek-V4-Flash-07315%

Basetengpt-oss-120b4%

Cerebrasgpt-oss-120b0%≤18%

xAIGrok-4.30%≤18%

TogetherLlama-3.3-70B0%≤18%

0%25%50%75%100%

The agent says it will do something, then goes quiet without doing it. The caller has to ask again.

DeepSeek-V4-Flash-07314%

gpt-5.6-luna6%

gpt-4.1-mini6%

gpt-4.16%

gemini-3.7-flash9%

Qwen3.8-27B9%

Nemotron-3-Ultra11%

DeepSeek-V4.1-Flash11%

gpt-5-mini11%

Mercury Voice13%

gpt-5.215%

gemini-3.8-flash15%

gemini-3.5-flash-lite17%

gpt-oss-120b17%

gpt-5.4-mini19%

gpt-5.6-terra19%

Grok-4.319%

gpt-519%

gpt-oss-120b20%

gpt-5-nano20%

Claude Haiku 4.522%

DeepSeek-V4.1-Flash22%

Claude Sonnet 526%

qwen-turbo76%

Llama-3.3-70B81%

0%25%50%75%100%

How often the caller hears nothing at all — every turn counted. The small number is the narrower measure: the share of tool calls made in complete silence, which is pinned at 100% for about half the rows in every language and so cannot rank them.

Claude Haiku 4.50%0% of tool calls ≤19%0 of 154 turns · ≤18% · 19 independent

qwen-turbo32%91% of tool calls29 of 92 turns

Claude Sonnet 543%69% of tool calls70 of 162 turns

Qwen3.8-27B53%76% of tool calls103 of 195 turns

Llama-3.3-70B53%100% of tool calls ≥82%68 of 128 turns

gpt-4.1-mini58%93% of tool calls91 of 158 turns

Grok-4.358%94% of tool calls90 of 156 turns

DeepSeek-V4-Flash-073160%89% of tool calls108 of 181 turns

gpt-5.4-mini62%100% of tool calls ≥82%96 of 156 turns

gemini-3.5-flash-lite64%98% of tool calls110 of 172 turns

gpt-4.164%100% of tool calls ≥82%105 of 165 turns

DeepSeek-V4.1-Flash64%91% of tool calls127 of 199 turns

gemini-3.7-flash64%100% of tool calls ≥82%108 of 168 turns

gpt-5.6-terra64%100% of tool calls ≥82%108 of 168 turns

DeepSeek-V4.1-Flash64%91% of tool calls124 of 195 turns

Nemotron-3-Ultra65%98% of tool calls114 of 176 turns

gemini-3.8-flash66%100% of tool calls ≥82%114 of 174 turns

gpt-oss-120b67%100% of tool calls ≥81%119 of 177 turns

gpt-5.6-luna69%100% of tool calls ≥82%131 of 191 turns

gpt-5-mini70%100% of tool calls ≥82%137 of 195 turns

gpt-5.270%100% of tool calls ≥82%139 of 198 turns

gpt-oss-120b70%100% of tool calls ≥82%132 of 189 turns

gpt-5-nano70%100% of tool calls ≥82%136 of 193 turns

gpt-570%99% of tool calls136 of 194 turns

Mercury Voice72%100% of tool calls ≥82%141 of 196 turns

0%25%50%75%100%

The agent states a fact it was never given — an opening time, a delivery window, a policy. It sounds confident and it is invented. Measured on the run without the language line, the only pass that ran these probes.

8 models0 of 30≤31% · 10 independent0%

Claude Haiku 4.5Claude Sonnet 5DeepSeek-V4.1-Flashgpt-5.6-lunagpt-5.6-terraGrok-4.3Llama-3.3-70BQwen3.8-27B

qwen-turbo21 of 3070%

Nemotron-3-Ultra16 of 3053%

gpt-4.1-mini15 of 3050%

gpt-5-nano12 of 3040%

gpt-oss-120b10 of 3033%

gpt-510 of 3033%

gpt-oss-120b9 of 3030%

DeepSeek-V4-Flash-07317 of 3023%

gpt-4.16 of 3020%

gemini-3.5-flash-lite4 of 3013%

gpt-5-mini4 of 3013%

A tool returns $75.50 or a date; does the agent say that exact value out loud? Judge-scored on the 2026-07 model set, so it covers three of these languages — three vendors independently turn $75.50 into $57.50 in Arabic, because Arabic reads units before tens.

xAIGrok-4.3100%

OpenAIgpt-4.195%

OpenAIgpt-590%

TogetherLlama-3.3-70B90%

OpenAIgpt-4.1-mini81%

Cerebrasgpt-oss-120b81%

OpenAIgpt-5-mini71%

Alibabaqwen-turbo43%

0%25%50%75%100%