Did the agent actually finish what the caller asked for — look the order up, issue the refund, book the appointment?

Gemini3.5-flash-lite94.74%

OpenAIgpt-5.4-mini92.69%

OpenAIgpt-4.191.81%

BasetenDeepSeek-V4-Flash-073190.35%

CerebrasQwen3.8-27B88.89%

OpenAIgpt-5.6-terra88.60%

TogetherDeepSeek-V4.1-Flash87.72%

Gemini3.7-flash86.84%

Gemini3.8-flash86.84%

BasetenNemotron-3-Ultra86.84%

Basetengpt-oss-120b84.21%

AnthropicClaude Sonnet 583.63%

Cerebrasgpt-oss-120b82.60%

InceptionMercury Voice81.58%

BasetenDeepSeek-V4.1-Flash81.29%

OpenAIgpt-5.280.70%

xAIGrok-4.379.82%

OpenAIgpt-4.1-mini78.07%

OpenAIgpt-5-mini76.61%

OpenAIgpt-5-nano75.88%

AnthropicClaude Haiku 4.575.44%

OpenAIgpt-575.44%

OpenAIgpt-5.6-luna75.44%

TogetherLlama-3.3-70B52.19%

Alibabaqwen-turbo11.84%

0%25%50%75%100%

With no line in the prompt naming a language, how much of what the agent says comes back in the caller's language rather than English.

AnthropicClaude Sonnet 5100%≥82%

TogetherDeepSeek-V4.1-Flash100%≥82%

Gemini3.7-flash100%≥82%

Gemini3.8-flash100%≥82%

OpenAIgpt-4.1100%≥82%

OpenAIgpt-5.2100%≥82%

OpenAIgpt-5.4-mini100%≥82%

OpenAIgpt-5.6-luna100%≥82%

OpenAIgpt-5.6-terra100%≥82%

InceptionMercury Voice100%≥82%

OpenAIgpt-5-mini98%

CerebrasQwen3.8-27B97%

OpenAIgpt-596%

Gemini3.5-flash-lite93%

OpenAIgpt-5-nano62%

Alibabaqwen-turbo52%

BasetenDeepSeek-V4.1-Flash50%

BasetenDeepSeek-V4-Flash-073110%

AnthropicClaude Haiku 4.59%

OpenAIgpt-4.1-mini3%

Basetengpt-oss-120b0%≤18%

Cerebrasgpt-oss-120b0%≤18%

xAIGrok-4.30%≤18%

TogetherLlama-3.3-70B0%≤18%

BasetenNemotron-3-Ultra0%≤18%

0%25%50%75%100%

The agent says it will do something, then goes quiet without doing it. The caller has to ask again.

gemini-3.5-flash-lite6%

gpt-4.16%

DeepSeek-V4-Flash-07317%

Qwen3.8-27B7%

gpt-5.4-mini11%

gemini-3.7-flash11%

gemini-3.8-flash11%

DeepSeek-V4.1-Flash13%

gpt-oss-120b15%

gpt-5.6-terra17%

gpt-oss-120b19%

Mercury Voice19%

DeepSeek-V4.1-Flash20%

gpt-5-mini20%

Nemotron-3-Ultra22%

gpt-5.222%

gpt-4.1-mini24%

gpt-5-nano24%

Grok-4.326%

gpt-526%

gpt-5.6-luna26%

Claude Sonnet 528%

Claude Haiku 4.541%

Llama-3.3-70B76%

qwen-turbo94%

0%25%50%75%100%

How often the caller hears nothing at all — every turn counted. The small number is the narrower measure: the share of tool calls made in complete silence, which is pinned at 100% for about half the rows in every language and so cannot rank them.

Claude Haiku 4.50%0% of tool calls ≤19%0 of 140 turns · ≤18% · 19 independent

qwen-turbo20%100% of tool calls ≥48%15 of 75 turns

Claude Sonnet 527%43% of tool calls42 of 157 turns

DeepSeek-V4-Flash-073137%55% of tool calls71 of 190 turns

DeepSeek-V4.1-Flash46%67% of tool calls85 of 186 turns

Qwen3.8-27B50%71% of tool calls98 of 196 turns

Llama-3.3-70B50%100% of tool calls ≥82%60 of 120 turns

gpt-4.1-mini52%88% of tool calls75 of 145 turns

DeepSeek-V4.1-Flash55%79% of tool calls105 of 192 turns

Grok-4.359%98% of tool calls90 of 152 turns

gemini-3.5-flash-lite62%100% of tool calls ≥82%98 of 158 turns

gpt-5.4-mini62%100% of tool calls ≥82%98 of 158 turns

gpt-4.163%100% of tool calls ≥82%104 of 164 turns

gpt-5.6-terra64%100% of tool calls ≥82%105 of 165 turns

Nemotron-3-Ultra64%100% of tool calls ≥81%105 of 165 turns

gemini-3.7-flash64%100% of tool calls ≥82%108 of 168 turns

gemini-3.8-flash66%100% of tool calls ≥82%115 of 175 turns

gpt-567%99% of tool calls122 of 183 turns

gpt-5.6-luna67%100% of tool calls ≥82%120 of 180 turns

gpt-oss-120b68%100% of tool calls ≥82%124 of 183 turns

gpt-oss-120b68%100% of tool calls ≥82%127 of 186 turns

gpt-5-mini68%100% of tool calls ≥82%128 of 187 turns

Mercury Voice70%100% of tool calls ≥82%134 of 191 turns

gpt-5.271%100% of tool calls ≥82%140 of 198 turns

gpt-5-nano73%100% of tool calls ≥81%131 of 180 turns

0%25%50%75%100%

The agent states a fact it was never given — an opening time, a delivery window, a policy. It sounds confident and it is invented. Measured on the run without the language line, the only pass that ran these probes.

9 models0 of 30≤31% · 10 independent0%

Claude Sonnet 5DeepSeek-V4-Flash-0731DeepSeek-V4.1-Flashgemini-3.8-flashgpt-5.6-lunagpt-5.6-terraGrok-4.3Llama-3.3-70BQwen3.8-27B

Nemotron-3-Ultra21 of 3070%

gpt-4.1-mini21 of 3070%

gpt-5-nano13 of 3043%

gemini-3.5-flash-lite10 of 3033%

qwen-turbo9 of 3030%

gpt-56 of 3020%

gpt-4.15 of 3017%

gpt-oss-120b4 of 3013%

gpt-oss-120b4 of 3013%

gpt-5-mini4 of 3013%

Claude Haiku 4.51 of 303%

A tool returns $75.50 or a date; does the agent say that exact value out loud? Judge-scored on the 2026-07 model set, so it covers three of these languages — three vendors independently turn $75.50 into $57.50 in Arabic, because Arabic reads units before tens.

OpenAIgpt-4.1100%

OpenAIgpt-5100%

xAIGrok-4.3100%

TogetherLlama-3.3-70B100%

OpenAIgpt-4.1-mini95%

OpenAIgpt-5-mini90%

Cerebrasgpt-oss-120b86%

Alibabaqwen-turbo67%

0%25%50%75%100%