Did the agent actually finish what the caller asked for — look the order up, issue the refund, book the appointment?

OpenAIgpt-4.1-mini90.35%

BasetenNemotron-3-Ultra89.47%

BasetenDeepSeek-V4-Flash-073188.60%

Gemini3.7-flash85.96%

Gemini3.5-flash-lite85.09%

OpenAIgpt-4.184.21%

xAIGrok-4.382.46%

Gemini3.8-flash81.58%

OpenAIgpt-5.4-mini81.58%

CerebrasQwen3.8-27B81.29%

BasetenDeepSeek-V4.1-Flash79.24%

Cerebrasgpt-oss-120b77.19%

Basetengpt-oss-120b77.05%

InceptionMercury Voice76.75%

TogetherDeepSeek-V4.1-Flash74.27%

OpenAIgpt-5-mini73.98%

OpenAIgpt-5.6-terra73.68%

OpenAIgpt-5-nano73.10%

AnthropicClaude Sonnet 571.93%

OpenAIgpt-571.93%

OpenAIgpt-5.271.93%

OpenAIgpt-5.6-luna71.35%

AnthropicClaude Haiku 4.570.61%

TogetherLlama-3.3-70B52.49%

Alibabaqwen-turbo36.40%

0%25%50%75%100%

With no line in the prompt naming a language, how much of what the agent says comes back in the caller's language rather than English.

AnthropicClaude Sonnet 5100%≥82%

TogetherDeepSeek-V4.1-Flash100%≥82%

Gemini3.7-flash100%≥82%

OpenAIgpt-4.1100%≥82%

OpenAIgpt-5100%≥82%

OpenAIgpt-5-mini100%≥82%

OpenAIgpt-5.2100%≥82%

OpenAIgpt-5.4-mini100%≥82%

xAIGrok-4.3100%≥82%

InceptionMercury Voice100%≥82%

Alibabaqwen-turbo100%≥82%

Gemini3.8-flash98%

Cerebrasgpt-oss-120b98%

OpenAIgpt-5.6-terra97%

OpenAIgpt-5.6-luna95%

Gemini3.5-flash-lite92%

CerebrasQwen3.8-27B88%

OpenAIgpt-5-nano77%

BasetenDeepSeek-V4.1-Flash43%

AnthropicClaude Haiku 4.525%

BasetenNemotron-3-Ultra21%

BasetenDeepSeek-V4-Flash-07318%

TogetherLlama-3.3-70B6%

OpenAIgpt-4.1-mini1%

Basetengpt-oss-120b0%≤18%

0%25%50%75%100%

The agent says it will do something, then goes quiet without doing it. The caller has to ask again.

gpt-4.1-mini9%

Nemotron-3-Ultra11%

DeepSeek-V4-Flash-073113%

gemini-3.7-flash13%

gemini-3.5-flash-lite15%

gpt-4.117%

DeepSeek-V4.1-Flash20%

gemini-3.8-flash22%

Qwen3.8-27B22%

gpt-oss-120b22%

Grok-4.324%

gpt-oss-120b24%

Mercury Voice24%

gpt-5.4-mini26%

gpt-5-nano26%

DeepSeek-V4.1-Flash28%

gpt-5-mini28%

gpt-5.6-terra31%

gpt-533%

gpt-5.6-luna33%

gpt-5.235%

Claude Haiku 4.541%

Claude Sonnet 543%

Llama-3.3-70B74%

qwen-turbo87%

0%25%50%75%100%

How often the caller hears nothing at all — every turn counted. The small number is the narrower measure: the share of tool calls made in complete silence, which is pinned at 100% for about half the rows in every language and so cannot rank them.

Claude Haiku 4.50%0% of tool calls ≤18%0 of 152 turns · ≤18% · 19 independent

Claude Sonnet 528%44% of tool calls44 of 159 turns

Qwen3.8-27B38%55% of tool calls71 of 187 turns

qwen-turbo40%100% of tool calls ≥74%40 of 100 turns

DeepSeek-V4-Flash-073141%61% of tool calls74 of 182 turns

Llama-3.3-70B51%100% of tool calls ≥82%62 of 122 turns

DeepSeek-V4.1-Flash57%82% of tool calls111 of 194 turns

DeepSeek-V4.1-Flash61%86% of tool calls118 of 195 turns

gpt-4.163%97% of tool calls106 of 169 turns

Grok-4.363%99% of tool calls103 of 164 turns

gpt-5.4-mini63%100% of tool calls ≥82%100 of 160 turns

gpt-5.6-terra63%100% of tool calls ≥80%101 of 161 turns

Nemotron-3-Ultra64%96% of tool calls117 of 182 turns

gemini-3.7-flash64%100% of tool calls ≥82%109 of 169 turns

gemini-3.5-flash-lite64%99% of tool calls110 of 171 turns

gpt-4.1-mini65%99% of tool calls112 of 173 turns

gemini-3.8-flash66%100% of tool calls ≥82%114 of 174 turns

gpt-566%97% of tool calls124 of 188 turns

gpt-oss-120b69%100% of tool calls ≥82%130 of 188 turns

gpt-5.6-luna69%100% of tool calls ≥82%135 of 195 turns

gpt-oss-120b70%99% of tool calls138 of 196 turns

gpt-5-mini70%100% of tool calls ≥82%135 of 194 turns

gpt-5-nano72%100% of tool calls ≥82%136 of 189 turns

gpt-5.272%100% of tool calls ≥81%146 of 203 turns

Mercury Voice73%100% of tool calls ≥82%149 of 205 turns

0%25%50%75%100%

The agent states a fact it was never given — an opening time, a delivery window, a policy. It sounds confident and it is invented. Measured on the run without the language line, the only pass that ran these probes.

1 models0 of 30≤31% · 10 independent0%

DeepSeek-V4.1-Flash