Did the agent actually finish what the caller asked for — look the order up, issue the refund, book the appointment?

BasetenDeepSeek-V4-Flash-073191.23%

Gemini3.5-flash-lite89.47%

OpenAIgpt-5.4-mini88.60%

OpenAIgpt-4.183.63%

xAIGrok-4.383.63%

AnthropicClaude Sonnet 583.33%

Gemini3.7-flash83.33%

Gemini3.8-flash82.46%

BasetenDeepSeek-V4.1-Flash81.14%

TogetherDeepSeek-V4.1-Flash79.82%

OpenAIgpt-578.65%

OpenAIgpt-5-mini78.65%

OpenAIgpt-4.1-mini78.07%

CerebrasQwen3.8-27B78.07%

OpenAIgpt-5.276.75%

Cerebrasgpt-oss-120b74.85%

OpenAIgpt-5-nano74.42%

Basetengpt-oss-120b74.12%

OpenAIgpt-5.6-luna71.49%

AnthropicClaude Haiku 4.570.61%

BasetenNemotron-3-Ultra68.86%

OpenAIgpt-5.6-terra68.42%

TogetherLlama-3.3-70B46.05%

Alibabaqwen-turbo12.28%

0%25%50%75%100%

With no line in the prompt naming a language, how much of what the agent says comes back in the caller's language rather than English.

AnthropicClaude Sonnet 5100%≥82%

TogetherDeepSeek-V4.1-Flash100%≥82%

Gemini3.5-flash-lite100%≥82%

Gemini3.7-flash100%≥82%

Gemini3.8-flash100%≥82%

OpenAIgpt-4.1100%≥82%

OpenAIgpt-5100%≥82%

OpenAIgpt-5.2100%≥82%

OpenAIgpt-5.4-mini100%≥82%

OpenAIgpt-5.6-luna100%≥82%

OpenAIgpt-5.6-terra100%≥82%

OpenAIgpt-5-mini98%

TogetherLlama-3.3-70B98%

BasetenDeepSeek-V4.1-Flash97%

CerebrasQwen3.8-27B96%

OpenAIgpt-5-nano95%

Alibabaqwen-turbo90%

OpenAIgpt-4.1-mini59%

BasetenDeepSeek-V4-Flash-073133%

BasetenNemotron-3-Ultra22%

AnthropicClaude Haiku 4.515%

Cerebrasgpt-oss-120b3%

Basetengpt-oss-120b3%

xAIGrok-4.30%≤18%

0%25%50%75%100%

The agent says it will do something, then goes quiet without doing it. The caller has to ask again.

DeepSeek-V4-Flash-07317%

gemini-3.5-flash-lite17%

Grok-4.317%

gpt-4.119%

gemini-3.7-flash19%

gemini-3.8-flash20%

DeepSeek-V4.1-Flash20%

gpt-5.4-mini22%

Claude Sonnet 522%

DeepSeek-V4.1-Flash22%

Qwen3.8-27B22%

gpt-524%

gpt-oss-120b24%

gpt-5-mini26%

gpt-5-nano26%

gpt-5.228%

gpt-4.1-mini30%

gpt-oss-120b30%

gpt-5.6-luna30%

Nemotron-3-Ultra35%

Claude Haiku 4.539%

gpt-5.6-terra43%

Llama-3.3-70B78%

qwen-turbo96%

0%25%50%75%100%

How often the caller hears nothing at all — every turn counted. The small number is the narrower measure: the share of tool calls made in complete silence, which is pinned at 100% for about half the rows in every language and so cannot rank them.

Claude Haiku 4.50%0% of tool calls ≤20%0 of 136 turns · ≤18% · 19 independent

qwen-turbo13%100% of tool calls ≥29%9 of 69 turns

Claude Sonnet 537%58% of tool calls60 of 164 turns

Llama-3.3-70B50%100% of tool calls ≥82%60 of 120 turns

Qwen3.8-27B51%70% of tool calls100 of 198 turns

gpt-4.1-mini51%90% of tool calls70 of 138 turns

DeepSeek-V4-Flash-073161%90% of tool calls114 of 186 turns

gpt-4.161%98% of tool calls95 of 157 turns

Nemotron-3-Ultra61%97% of tool calls97 of 160 turns

gpt-5.6-terra61%100% of tool calls ≥81%93 of 153 turns

Grok-4.362%97% of tool calls102 of 165 turns

DeepSeek-V4.1-Flash62%93% of tool calls114 of 183 turns

DeepSeek-V4.1-Flash62%91% of tool calls116 of 188 turns

gpt-5.4-mini63%100% of tool calls ≥82%100 of 160 turns

gemini-3.5-flash-lite65%100% of tool calls ≥82%111 of 171 turns

gemini-3.7-flash66%100% of tool calls ≥82%119 of 179 turns

gemini-3.8-flash66%100% of tool calls ≥82%118 of 178 turns

gpt-5.6-luna67%100% of tool calls ≥81%121 of 181 turns

gpt-oss-120b68%100% of tool calls ≥82%127 of 186 turns

gpt-569%100% of tool calls ≥82%136 of 196 turns

gpt-5-mini69%100% of tool calls ≥82%136 of 196 turns

gpt-5.269%100% of tool calls ≥81%131 of 191 turns

gpt-oss-120b70%100% of tool calls ≥81%138 of 196 turns

gpt-5-nano73%100% of tool calls ≥81%122 of 168 turns

0%25%50%75%100%