LLM

Multilingual

The English board’s 19 scenarios and grader, run per language at 3 iterations.

Caller speaks

Chinese

Task done

higher is better

Did the agent actually finish what the caller asked for — look the order up, issue the refund, book the appointment?

95.91%

92.98%

91.23%

89.47%

87.72%

87.28%

86.84%

85.96%

84.80%

82.02%

81.58%

80.85%

80.70%

79.82%

79.39%

79.09%

78.51%

78.36%

73.54%

56.87%

42.54%

0%25%50%75%100%

Answered in the language (nothing asked)

higher is better

With no line in the prompt naming a language, how much of what the agent says comes back in the caller's language rather than English.

AnthropicClaude Sonnet 5100%≥82%

TogetherDeepSeek-V4.1-Flash100%≥82%

Gemini3.7-flash100%≥82%

Gemini3.8-flash100%≥82%

OpenAIgpt-4.1100%≥82%

OpenAIgpt-5-mini100%≥82%

OpenAIgpt-5.2100%≥82%

OpenAIgpt-5.4-mini100%≥82%

OpenAIgpt-5.6-terra100%≥82%

InceptionMercury Voice100%≥82%

OpenAIgpt-5.6-luna98%

Alibabaqwen-turbo91%

OpenAIgpt-5-nano88%

AnthropicClaude Haiku 4.587%

OpenAIgpt-586%

CerebrasQwen3.8-27B78%

Gemini3.5-flash-lite65%

BasetenDeepSeek-V4.1-Flash45%

OpenAIgpt-4.1-mini8%

BasetenDeepSeek-V4-Flash-07312%

Basetengpt-oss-120b2%

Cerebrasgpt-oss-120b0%≤18%

xAIGrok-4.30%≤18%

TogetherLlama-3.3-70B0%≤18%

BasetenNemotron-3-Ultra0%≤18%

0%25%50%75%100%

Stalled

lower is better

The agent says it will do something, then goes quiet without doing it. The caller has to ask again.

gpt-4.1-mini2%

gpt-4.16%

DeepSeek-V4-Flash-07317%

gemini-3.5-flash-lite13%

DeepSeek-V4.1-Flash17%

Claude Sonnet 517%

Mercury Voice19%

DeepSeek-V4.1-Flash20%

gpt-520%

gpt-5-mini20%

gpt-5-nano20%

gpt-5.222%

Grok-4.324%

gpt-5.6-luna24%

Claude Haiku 4.531%

Llama-3.3-70B69%

qwen-turbo70%

0%25%50%75%100%

Silence

lower is better

How often the caller hears nothing at all — every turn counted. The small number is the narrower measure: the share of tool calls made in complete silence, which is pinned at 100% for about half the rows in every language and so cannot rank them.

Claude Haiku 4.50%0% of tool calls ≤19%0 of 152 turns · ≤18% · 19 independent

Claude Sonnet 539%60% of tool calls66 of 170 turns

qwen-turbo43%100% of tool calls ≥77%46 of 106 turns

Qwen3.8-27B45%64% of tool calls87 of 194 turns

Llama-3.3-70B50%100% of tool calls ≥82%60 of 120 turns

DeepSeek-V4.1-Flash56%81% of tool calls111 of 197 turns

DeepSeek-V4-Flash-073157%84% of tool calls103 of 180 turns

DeepSeek-V4.1-Flash57%82% of tool calls109 of 192 turns

Grok-4.359%98% of tool calls91 of 153 turns

gpt-4.1-mini62%98% of tool calls103 of 165 turns

gpt-5.4-mini62%99% of tool calls101 of 162 turns

gpt-4.163%100% of tool calls ≥82%104 of 164 turns

gemini-3.7-flash64%100% of tool calls ≥82%108 of 168 turns

gpt-5.6-terra65%100% of tool calls ≥82%112 of 172 turns

gemini-3.5-flash-lite66%99% of tool calls121 of 182 turns

gemini-3.8-flash66%100% of tool calls ≥82%115 of 175 turns

Nemotron-3-Ultra67%100% of tool calls ≥82%123 of 183 turns

gpt-oss-120b68%100% of tool calls ≥82%125 of 183 turns

gpt-568%100% of tool calls ≥82%128 of 188 turns

gpt-5.6-luna68%100% of tool calls ≥82%129 of 189 turns

gpt-5.268%100% of tool calls ≥81%126 of 186 turns

gpt-oss-120b71%100% of tool calls ≥82%145 of 203 turns

gpt-5-mini72%100% of tool calls ≥82%147 of 205 turns

Mercury Voice73%100% of tool calls ≥82%149 of 203 turns

gpt-5-nano73%100% of tool calls ≥82%138 of 188 turns

0%25%50%75%100%

Fabrication

lower is better

The agent states a fact it was never given — an opening time, a delivery window, a policy. It sounds confident and it is invented. Measured on the run without the language line, the only pass that ran these probes.