Did the agent actually finish what the caller asked for — look the order up, issue the refund, book the appointment?

BasetenDeepSeek-V4-Flash-073192.98%

Gemini3.5-flash-lite92.11%

OpenAIgpt-4.1-mini92.11%

TogetherDeepSeek-V4.1-Flash90.35%

OpenAIgpt-4.189.47%

OpenAIgpt-5.4-mini88.60%

CerebrasQwen3.8-27B88.30%

BasetenDeepSeek-V4.1-Flash87.43%

Gemini3.7-flash86.84%

BasetenNemotron-3-Ultra86.84%

xAIGrok-4.386.26%

Gemini3.8-flash85.96%

OpenAIgpt-5.6-luna84.80%

InceptionMercury Voice84.06%

AnthropicClaude Sonnet 582.75%

AnthropicClaude Haiku 4.581.29%

OpenAIgpt-5.6-terra81.14%

Cerebrasgpt-oss-120b80.70%

OpenAIgpt-5.279.39%

OpenAIgpt-5-mini79.24%

Basetengpt-oss-120b78.07%

OpenAIgpt-573.68%

OpenAIgpt-5-nano73.25%

TogetherLlama-3.3-70B55.99%

Alibabaqwen-turbo48.98%

0%25%50%75%100%

With no line in the prompt naming a language, how much of what the agent says comes back in the caller's language rather than English.

AnthropicClaude Sonnet 5100%≥82%

TogetherDeepSeek-V4.1-Flash100%≥82%

Gemini3.7-flash100%≥82%

Gemini3.8-flash100%≥82%

OpenAIgpt-5-mini100%≥82%

OpenAIgpt-5.2100%≥82%

OpenAIgpt-5.4-mini100%≥82%

OpenAIgpt-5.6-luna100%≥82%

OpenAIgpt-5.6-terra100%≥82%

InceptionMercury Voice100%≥82%

OpenAIgpt-598%

CerebrasQwen3.8-27B98%

Alibabaqwen-turbo95%

Gemini3.5-flash-lite87%

OpenAIgpt-4.187%

OpenAIgpt-5-nano83%

BasetenDeepSeek-V4.1-Flash77%

AnthropicClaude Haiku 4.576%

BasetenDeepSeek-V4-Flash-073125%

BasetenNemotron-3-Ultra11%

OpenAIgpt-4.1-mini5%

Cerebrasgpt-oss-120b2%

xAIGrok-4.32%

Basetengpt-oss-120b0%≤18%

TogetherLlama-3.3-70B0%≤18%

0%25%50%75%100%

The agent says it will do something, then goes quiet without doing it. The caller has to ask again.

DeepSeek-V4-Flash-07312%

gpt-4.1-mini4%

gpt-4.16%

DeepSeek-V4.1-Flash7%

Qwen3.8-27B9%

gemini-3.5-flash-lite11%

gemini-3.7-flash11%

DeepSeek-V4.1-Flash13%

gemini-3.8-flash13%

gpt-5.4-mini15%

Mercury Voice15%

gpt-oss-120b15%

gpt-oss-120b15%

Nemotron-3-Ultra17%

gpt-5.6-luna17%

Claude Sonnet 517%

Grok-4.319%

gpt-5-mini19%

gpt-5-nano22%

gpt-5.224%

gpt-526%

Claude Haiku 4.528%

gpt-5.6-terra28%

Llama-3.3-70B63%

qwen-turbo63%

0%25%50%75%100%

How often the caller hears nothing at all — every turn counted. The small number is the narrower measure: the share of tool calls made in complete silence, which is pinned at 100% for about half the rows in every language and so cannot rank them.

Claude Haiku 4.50%0% of tool calls ≤19%0 of 152 turns · ≤18% · 19 independent

Claude Sonnet 533%51% of tool calls57 of 171 turns

qwen-turbo40%86% of tool calls44 of 111 turns

Qwen3.8-27B49%69% of tool calls97 of 199 turns

Llama-3.3-70B50%100% of tool calls ≥82%60 of 120 turns

DeepSeek-V4.1-Flash56%81% of tool calls108 of 193 turns

DeepSeek-V4-Flash-073157%84% of tool calls108 of 188 turns

gpt-4.1-mini59%92% of tool calls98 of 167 turns

DeepSeek-V4.1-Flash59%85% of tool calls115 of 196 turns

Grok-4.360%97% of tool calls96 of 159 turns

gpt-5.6-terra63%100% of tool calls ≥82%101 of 161 turns

gpt-4.164%100% of tool calls ≥82%109 of 169 turns

gemini-3.7-flash64%100% of tool calls ≥82%108 of 168 turns

gemini-3.5-flash-lite65%100% of tool calls ≥82%111 of 171 turns

gpt-5.4-mini65%100% of tool calls ≥82%110 of 170 turns

Nemotron-3-Ultra65%97% of tool calls117 of 181 turns

gemini-3.8-flash65%100% of tool calls ≥82%112 of 172 turns

gpt-5.6-luna69%100% of tool calls ≥82%133 of 193 turns

gpt-oss-120b69%100% of tool calls ≥81%127 of 184 turns

gpt-569%100% of tool calls ≥82%132 of 190 turns

gpt-5.270%100% of tool calls ≥82%142 of 202 turns

Mercury Voice72%100% of tool calls ≥82%148 of 205 turns

gpt-oss-120b72%100% of tool calls ≥82%145 of 201 turns

gpt-5-mini72%100% of tool calls ≥82%147 of 205 turns

gpt-5-nano74%100% of tool calls ≥82%142 of 191 turns

0%25%50%75%100%

The agent states a fact it was never given — an opening time, a delivery window, a policy. It sounds confident and it is invented. Measured on the run without the language line, the only pass that ran these probes.

8 models0 of 30≤31% · 10 independent0%

Claude Haiku 4.5Claude Sonnet 5DeepSeek-V4.1-Flashgpt-5.6-lunagpt-5.6-terraGrok-4.3Llama-3.3-70BQwen3.8-27B

qwen-turbo19 of 3063%

gpt-513 of 3043%

gpt-4.1-mini12 of 3040%

Nemotron-3-Ultra10 of 3033%

gpt-oss-120b6 of 3020%

gpt-oss-120b5 of 3017%

gpt-5-nano5 of 3017%

gemini-3.5-flash-lite4 of 3013%

gpt-4.13 of 3010%

gpt-5-mini3 of 3010%

DeepSeek-V4-Flash-07311 of 303%