Did the agent actually finish what the caller asked for — look the order up, issue the refund, book the appointment?

BasetenDeepSeek-V4-Flash-073191.23%

Gemini3.5-flash-lite90.35%

OpenAIgpt-4.189.47%

BasetenNemotron-3-Ultra89.47%

CerebrasQwen3.8-27B89.18%

OpenAIgpt-4.1-mini88.60%

TogetherDeepSeek-V4.1-Flash87.13%

OpenAIgpt-5.6-luna87.13%

Gemini3.7-flash86.84%

OpenAIgpt-5.6-terra86.26%

AnthropicClaude Sonnet 585.96%

OpenAIgpt-5.285.96%

OpenAIgpt-5.4-mini85.67%

xAIGrok-4.385.23%

Gemini3.8-flash85.09%

InceptionMercury Voice85.09%

OpenAIgpt-5-mini84.06%

Cerebrasgpt-oss-120b83.63%

OpenAIgpt-5-nano81.73%

Basetengpt-oss-120b81.14%

BasetenDeepSeek-V4.1-Flash79.53%

AnthropicClaude Haiku 4.577.34%

OpenAIgpt-574.56%

TogetherLlama-3.3-70B58.33%

Alibabaqwen-turbo35.23%

0%25%50%75%100%

With no line in the prompt naming a language, how much of what the agent says comes back in the caller's language rather than English.

AnthropicClaude Sonnet 5100%≥82%

TogetherDeepSeek-V4.1-Flash100%≥82%

Gemini3.7-flash100%≥82%

Gemini3.8-flash100%≥82%

OpenAIgpt-5100%≥82%

OpenAIgpt-5-mini100%≥82%

OpenAIgpt-5.2100%≥82%

OpenAIgpt-5.4-mini100%≥82%

OpenAIgpt-5.6-luna100%≥82%

OpenAIgpt-5.6-terra100%≥82%

InceptionMercury Voice100%≥82%

OpenAIgpt-4.198%

CerebrasQwen3.8-27B94%

OpenAIgpt-5-nano93%

AnthropicClaude Haiku 4.591%

Alibabaqwen-turbo90%

BasetenDeepSeek-V4.1-Flash72%

Gemini3.5-flash-lite72%

BasetenNemotron-3-Ultra46%

TogetherLlama-3.3-70B19%

Cerebrasgpt-oss-120b16%

Basetengpt-oss-120b16%

OpenAIgpt-4.1-mini13%

xAIGrok-4.312%

BasetenDeepSeek-V4-Flash-073111%

0%25%50%75%100%

The agent says it will do something, then goes quiet without doing it. The caller has to ask again.

DeepSeek-V4-Flash-07312%

gpt-4.16%

Qwen3.8-27B7%

DeepSeek-V4.1-Flash7%

Mercury Voice9%

Nemotron-3-Ultra11%

gpt-4.1-mini11%

gemini-3.7-flash11%

gpt-oss-120b11%

gpt-oss-120b11%

gemini-3.5-flash-lite13%

gpt-5.6-luna13%

gpt-5-mini13%

gpt-5-nano13%

gemini-3.8-flash15%

gpt-5.217%

Grok-4.317%

gpt-5.6-terra19%

gpt-520%

Claude Sonnet 522%

gpt-5.4-mini22%

DeepSeek-V4.1-Flash22%

Claude Haiku 4.535%

Llama-3.3-70B65%

qwen-turbo74%

0%25%50%75%100%

How often the caller hears nothing at all — every turn counted. The small number is the narrower measure: the share of tool calls made in complete silence, which is pinned at 100% for about half the rows in every language and so cannot rank them.

Claude Haiku 4.51%1% of tool calls1 of 147 turns

Claude Sonnet 532%50% of tool calls54 of 168 turns

qwen-turbo35%94% of tool calls34 of 96 turns

Qwen3.8-27B49%70% of tool calls97 of 196 turns

Llama-3.3-70B50%100% of tool calls ≥82%60 of 120 turns

gpt-4.1-mini58%93% of tool calls92 of 159 turns

DeepSeek-V4.1-Flash58%81% of tool calls113 of 196 turns

DeepSeek-V4-Flash-073159%84% of tool calls110 of 187 turns

Grok-4.360%97% of tool calls94 of 157 turns

DeepSeek-V4.1-Flash61%88% of tool calls119 of 194 turns

gpt-5.4-mini62%100% of tool calls ≥82%99 of 159 turns

Nemotron-3-Ultra64%97% of tool calls114 of 177 turns

gpt-4.164%100% of tool calls ≥82%108 of 168 turns

gemini-3.7-flash64%100% of tool calls ≥82%109 of 169 turns

gemini-3.5-flash-lite65%99% of tool calls113 of 174 turns

gpt-5.6-terra65%100% of tool calls ≥82%112 of 172 turns

gemini-3.8-flash66%100% of tool calls ≥82%115 of 175 turns

gpt-5.6-luna69%100% of tool calls ≥82%136 of 196 turns

gpt-5.269%100% of tool calls ≥82%134 of 193 turns

gpt-oss-120b70%100% of tool calls ≥82%133 of 190 turns

gpt-570%100% of tool calls ≥82%132 of 189 turns

gpt-5-mini71%100% of tool calls ≥82%146 of 205 turns

Mercury Voice72%100% of tool calls ≥82%142 of 198 turns

gpt-oss-120b72%100% of tool calls ≥82%141 of 197 turns

gpt-5-nano73%100% of tool calls ≥82%141 of 193 turns

0%25%50%75%100%

The agent states a fact it was never given — an opening time, a delivery window, a policy. It sounds confident and it is invented. Measured on the run without the language line, the only pass that ran these probes.

6 models0 of 30≤31% · 10 independent0%

Claude Sonnet 5DeepSeek-V4.1-Flashgpt-5.6-lunagpt-5.6-terraGrok-4.3Qwen3.8-27B

qwen-turbo21 of 3070%

gpt-4.1-mini15 of 3050%

gpt-512 of 3040%

gpt-oss-120b11 of 3037%

Nemotron-3-Ultra9 of 3030%

gpt-oss-120b9 of 3030%

gemini-3.5-flash-lite6 of 3020%

gpt-5-nano6 of 3020%

gpt-4.15 of 3017%

gpt-5-mini5 of 3017%

DeepSeek-V4-Flash-07313 of 3010%

Claude Haiku 4.51 of 303%

Llama-3.3-70B1 of 303%

A tool returns $75.50 or a date; does the agent say that exact value out loud? Judge-scored on the 2026-07 model set, so it covers three of these languages — three vendors independently turn $75.50 into $57.50 in Arabic, because Arabic reads units before tens.

OpenAIgpt-4.1100%

OpenAIgpt-4.1-mini100%

OpenAIgpt-5100%

OpenAIgpt-5-mini100%

Cerebrasgpt-oss-120b100%

xAIGrok-4.3100%

TogetherLlama-3.3-70B100%

Alibabaqwen-turbo86%

0%25%50%75%100%