LLM

Multilingual

The English board’s 19 scenarios and grader, run per language at 3 iterations.

Caller speaks

Korean

Task done

higher is better

Did the agent actually finish what the caller asked for — look the order up, issue the refund, book the appointment?

92.11%

90.64%

87.72%

85.38%

84.21%

83.92%

83.33%

82.46%

80.70%

79.97%

78.95%

78.51%

78.36%

75.44%

75.15%

73.68%

72.81%

72.51%

71.93%

69.30%

53.95%

43.13%

0%25%50%75%100%

Answered in the language (nothing asked)

higher is better

With no line in the prompt naming a language, how much of what the agent says comes back in the caller's language rather than English.

AnthropicClaude Sonnet 5100%≥82%

TogetherDeepSeek-V4.1-Flash100%≥82%

Gemini3.7-flash100%≥82%

Gemini3.8-flash100%≥82%

OpenAIgpt-4.1100%≥82%

OpenAIgpt-5.2100%≥82%

OpenAIgpt-5.4-mini100%≥82%

InceptionMercury Voice100%≥82%

Gemini3.5-flash-lite98%

OpenAIgpt-5.6-luna98%

OpenAIgpt-5-nano97%

OpenAIgpt-5-mini93%

OpenAIgpt-5.6-terra93%

OpenAIgpt-592%

CerebrasQwen3.8-27B91%

BasetenDeepSeek-V4.1-Flash79%

Alibabaqwen-turbo70%

AnthropicClaude Haiku 4.565%

BasetenNemotron-3-Ultra55%

TogetherLlama-3.3-70B6%

BasetenDeepSeek-V4-Flash-07311%

OpenAIgpt-4.1-mini0%≤18%

Basetengpt-oss-120b0%≤18%

Cerebrasgpt-oss-120b0%≤18%

xAIGrok-4.30%≤18%

0%25%50%75%100%

Stalled

lower is better

The agent says it will do something, then goes quiet without doing it. The caller has to ask again.

DeepSeek-V4-Flash-07319%

gpt-4.1-mini13%

gpt-oss-120b13%

gpt-4.117%

gemini-3.7-flash17%

gpt-5-nano17%

Grok-4.319%

gemini-3.8-flash19%

Mercury Voice19%

DeepSeek-V4.1-Flash19%

gemini-3.5-flash-lite20%

gpt-5-mini20%

Qwen3.8-27B28%

gpt-530%

gpt-5.235%

DeepSeek-V4.1-Flash35%

Claude Sonnet 541%

Llama-3.3-70B74%

qwen-turbo74%

0%25%50%75%100%

Silence

lower is better

How often the caller hears nothing at all — every turn counted. The small number is the narrower measure: the share of tool calls made in complete silence, which is pinned at 100% for about half the rows in every language and so cannot rank them.

Claude Haiku 4.50%0% of tool calls ≤18%0 of 154 turns · ≤18% · 19 independent

Claude Sonnet 529%47% of tool calls47 of 161 turns

Qwen3.8-27B41%58% of tool calls80 of 196 turns

qwen-turbo45%98% of tool calls50 of 111 turns

Llama-3.3-70B50%100% of tool calls ≥82%60 of 120 turns

DeepSeek-V4-Flash-073153%78% of tool calls97 of 182 turns

DeepSeek-V4.1-Flash53%75% of tool calls103 of 194 turns

Grok-4.360%94% of tool calls97 of 163 turns

DeepSeek-V4.1-Flash60%83% of tool calls121 of 201 turns

gpt-4.1-mini61%96% of tool calls99 of 163 turns

gpt-5.4-mini62%100% of tool calls ≥82%96 of 156 turns

gemini-3.5-flash-lite63%100% of tool calls ≥82%101 of 161 turns

gpt-4.164%100% of tool calls ≥82%105 of 165 turns

gemini-3.7-flash64%100% of tool calls ≥82%109 of 169 turns

gpt-5.6-terra65%100% of tool calls ≥82%110 of 170 turns

gemini-3.8-flash66%100% of tool calls ≥82%115 of 174 turns

Nemotron-3-Ultra67%98% of tool calls120 of 180 turns

gpt-568%99% of tool calls128 of 187 turns

gpt-5.6-luna69%100% of tool calls ≥82%135 of 195 turns

gpt-5.270%100% of tool calls ≥81%137 of 195 turns

gpt-oss-120b71%100% of tool calls ≥82%138 of 195 turns

gpt-5-mini71%100% of tool calls ≥82%140 of 197 turns

gpt-oss-120b72%100% of tool calls ≥82%146 of 203 turns

Mercury Voice72%100% of tool calls ≥82%146 of 203 turns

gpt-5-nano74%100% of tool calls ≥82%147 of 198 turns

0%25%50%75%100%

Fabrication

lower is better

The agent states a fact it was never given — an opening time, a delivery window, a policy. It sounds confident and it is invented. Measured on the run without the language line, the only pass that ran these probes.