LLM

Multilingual

The English board’s 19 scenarios and grader, run per language at 3 iterations.

Caller speaks

Thai

Task done

higher is better

Did the agent actually finish what the caller asked for — look the order up, issue the refund, book the appointment?

89.47%

87.72%

85.38%

84.21%

83.63%

81.58%

80.70%

80.12%

78.95%

78.36%

77.78%

77.34%

77.19%

74.71%

73.68%

73.25%

69.88%

69.15%

53.07%

43.42%

0%25%50%75%100%

Answered in the language (nothing asked)

higher is better

With no line in the prompt naming a language, how much of what the agent says comes back in the caller's language rather than English.

AnthropicClaude Sonnet 5100%≥82%

TogetherDeepSeek-V4.1-Flash100%≥82%

Gemini3.7-flash100%≥82%

Gemini3.8-flash100%≥82%

OpenAIgpt-4.1100%≥82%

OpenAIgpt-5100%≥82%

OpenAIgpt-5-mini100%≥82%

OpenAIgpt-5.2100%≥82%

OpenAIgpt-5.4-mini100%≥82%

InceptionMercury Voice100%≥82%

OpenAIgpt-5.6-luna98%

OpenAIgpt-5.6-terra97%

Alibabaqwen-turbo91%

OpenAIgpt-5-nano87%

CerebrasQwen3.8-27B83%

Gemini3.5-flash-lite79%

BasetenDeepSeek-V4.1-Flash57%

BasetenNemotron-3-Ultra32%

AnthropicClaude Haiku 4.521%

BasetenDeepSeek-V4-Flash-073114%

OpenAIgpt-4.1-mini12%

Basetengpt-oss-120b0%≤18%

Cerebrasgpt-oss-120b0%≤18%

xAIGrok-4.30%≤18%

TogetherLlama-3.3-70B0%≤18%

0%25%50%75%100%

Stalled

lower is better

The agent says it will do something, then goes quiet without doing it. The caller has to ask again.

DeepSeek-V4-Flash-073111%

gpt-4.1-mini13%

DeepSeek-V4.1-Flash15%

gemini-3.5-flash-lite17%

gpt-4.119%

gpt-5-nano22%

Qwen3.8-27B24%

DeepSeek-V4.1-Flash24%

gpt-5-mini24%

gpt-5.226%

gpt-5.6-terra26%

gpt-5.4-mini28%

Grok-4.331%

gpt-531%

qwen-turbo67%

0%25%50%75%100%

Silence

lower is better

How often the caller hears nothing at all — every turn counted. The small number is the narrower measure: the share of tool calls made in complete silence, which is pinned at 100% for about half the rows in every language and so cannot rank them.

Claude Haiku 4.50%0% of tool calls ≤18%0 of 147 turns · ≤18% · 19 independent

Claude Sonnet 536%57% of tool calls60 of 165 turns

Qwen3.8-27B44%63% of tool calls85 of 193 turns

qwen-turbo49%100% of tool calls ≥79%57 of 117 turns

Llama-3.3-70B50%100% of tool calls ≥82%60 of 120 turns

DeepSeek-V4-Flash-073155%80% of tool calls104 of 188 turns

gpt-4.1-mini57%90% of tool calls94 of 165 turns

Grok-4.361%100% of tool calls ≥82%92 of 152 turns

DeepSeek-V4.1-Flash62%88% of tool calls120 of 195 turns

gemini-3.5-flash-lite63%98% of tool calls104 of 166 turns

gemini-3.7-flash64%100% of tool calls ≥82%108 of 168 turns

gpt-4.164%100% of tool calls ≥82%109 of 169 turns

gpt-5.4-mini64%100% of tool calls ≥82%107 of 167 turns

gpt-5.6-terra65%100% of tool calls ≥82%111 of 171 turns

DeepSeek-V4.1-Flash65%92% of tool calls128 of 198 turns

Nemotron-3-Ultra66%99% of tool calls117 of 178 turns

gemini-3.8-flash66%100% of tool calls ≥82%114 of 174 turns

gpt-5.268%100% of tool calls ≥82%130 of 190 turns

gpt-5.6-luna68%100% of tool calls ≥82%126 of 186 turns

gpt-569%100% of tool calls ≥82%135 of 195 turns

gpt-5-mini70%100% of tool calls ≥82%135 of 193 turns

gpt-oss-120b71%100% of tool calls ≥82%140 of 197 turns

gpt-oss-120b71%100% of tool calls ≥82%141 of 198 turns

Mercury Voice75%100% of tool calls ≥82%157 of 210 turns

gpt-5-nano76%100% of tool calls ≥82%149 of 197 turns

0%25%50%75%100%

Fabrication

lower is better

The agent states a fact it was never given — an opening time, a delivery window, a policy. It sounds confident and it is invented. Measured on the run without the language line, the only pass that ran these probes.

9 models0 of 30≤31% · 10 independent0%

Claude Haiku 4.5

Claude Sonnet 5

DeepSeek-V4.1-Flashgpt-5-minigpt-5.6-lunagpt-5.6-terraGrok-4.3

Llama-3.3-70B

Qwen3.8-27B

qwen-turbo19 of 3063%

gpt-4.1-mini16 of 3053%

Nemotron-3-Ultra15 of 3050%

gemini-3.5-flash-lite10 of 3033%

gpt-oss-120b10 of 3033%

gpt-510 of 3033%