Did the agent actually finish what the caller asked for — look the order up, issue the refund, book the appointment?

BasetenDeepSeek-V4-Flash-073191.23%

OpenAIgpt-4.1-mini87.72%

OpenAIgpt-5.4-mini87.72%

OpenAIgpt-4.186.26%

Gemini3.5-flash-lite85.96%

Gemini3.7-flash85.96%

Gemini3.8-flash85.96%

BasetenDeepSeek-V4.1-Flash85.09%

Basetengpt-oss-120b84.50%

TogetherDeepSeek-V4.1-Flash83.92%

OpenAIgpt-5.6-luna82.02%

OpenAIgpt-5-nano79.68%

AnthropicClaude Sonnet 579.53%

BasetenNemotron-3-Ultra78.80%

CerebrasQwen3.8-27B78.65%

OpenAIgpt-5.6-terra78.51%

Cerebrasgpt-oss-120b77.19%

AnthropicClaude Haiku 4.576.90%

OpenAIgpt-5.274.56%

OpenAIgpt-5-mini74.12%

OpenAIgpt-573.25%

TogetherLlama-3.3-70B48.39%

Alibabaqwen-turbo21.93%

0%25%50%75%100%

With no line in the prompt naming a language, how much of what the agent says comes back in the caller's language rather than English.

AnthropicClaude Sonnet 5100%≥82%

Gemini3.8-flash100%≥82%

OpenAIgpt-4.1100%≥82%

OpenAIgpt-5100%≥82%

OpenAIgpt-5-mini100%≥82%

OpenAIgpt-5.2100%≥82%

OpenAIgpt-5.4-mini100%≥82%

OpenAIgpt-5.6-luna100%≥82%

OpenAIgpt-5.6-terra100%≥82%

TogetherDeepSeek-V4.1-Flash99%

Gemini3.5-flash-lite98%

Gemini3.7-flash98%

OpenAIgpt-5-nano98%

CerebrasQwen3.8-27B94%

TogetherLlama-3.3-70B88%

Alibabaqwen-turbo88%

OpenAIgpt-4.1-mini87%

BasetenDeepSeek-V4.1-Flash68%

AnthropicClaude Haiku 4.556%

BasetenNemotron-3-Ultra41%

BasetenDeepSeek-V4-Flash-073131%

Cerebrasgpt-oss-120b12%

Basetengpt-oss-120b8%

0%25%50%75%100%

The agent says it will do something, then goes quiet without doing it. The caller has to ask again.

DeepSeek-V4-Flash-07317%

gpt-4.1-mini11%

gpt-4.113%

gemini-3.7-flash13%

gemini-3.8-flash13%

DeepSeek-V4.1-Flash13%

gpt-oss-120b15%

gpt-oss-120b15%

gemini-3.5-flash-lite17%

DeepSeek-V4.1-Flash17%

gpt-5.6-luna20%

gpt-5-nano20%

gpt-5.4-mini22%

Claude Sonnet 524%

Qwen3.8-27B24%

gpt-5-mini24%

gpt-528%

Nemotron-3-Ultra30%

gpt-5.6-terra31%

gpt-5.235%

Claude Haiku 4.537%

Llama-3.3-70B74%

qwen-turbo89%

0%25%50%75%100%

How often the caller hears nothing at all — every turn counted. The small number is the narrower measure: the share of tool calls made in complete silence, which is pinned at 100% for about half the rows in every language and so cannot rank them.

Claude Haiku 4.50%0% of tool calls ≤19%0 of 146 turns · ≤18% · 19 independent

Claude Sonnet 527%42% of tool calls47 of 172 turns

qwen-turbo29%93% of tool calls25 of 87 turns

Qwen3.8-27B39%55% of tool calls80 of 205 turns

DeepSeek-V4-Flash-073143%63% of tool calls80 of 187 turns

Llama-3.3-70B50%100% of tool calls ≥82%60 of 120 turns

Nemotron-3-Ultra60%95% of tool calls97 of 162 turns

DeepSeek-V4.1-Flash61%88% of tool calls117 of 192 turns

DeepSeek-V4.1-Flash61%85% of tool calls121 of 199 turns

gpt-4.1-mini62%95% of tool calls106 of 171 turns

gpt-5.4-mini62%99% of tool calls101 of 162 turns

gpt-4.163%100% of tool calls ≥82%101 of 161 turns

gemini-3.5-flash-lite63%100% of tool calls ≥82%104 of 164 turns

gpt-5.6-terra63%100% of tool calls ≥82%103 of 163 turns

gemini-3.7-flash66%100% of tool calls ≥82%116 of 176 turns

gemini-3.8-flash66%100% of tool calls ≥82%116 of 176 turns

gpt-5.6-luna67%100% of tool calls ≥82%123 of 183 turns

gpt-5.268%100% of tool calls ≥82%127 of 187 turns

gpt-oss-120b70%100% of tool calls ≥82%135 of 193 turns

gpt-oss-120b70%100% of tool calls ≥81%136 of 193 turns

gpt-5-mini70%100% of tool calls ≥82%137 of 197 turns

gpt-570%100% of tool calls ≥82%135 of 193 turns

gpt-5-nano72%100% of tool calls ≥82%137 of 190 turns

0%25%50%75%100%