LLM

Multilingual

The English board’s 19 scenarios and grader, run per language at 3 iterations.

Caller speaks

Tamil

Task done

higher is better

Did the agent actually finish what the caller asked for — look the order up, issue the refund, book the appointment?

98.25%

92.11%

91.81%

91.23%

89.47%

88.60%

87.72%

86.84%

86.26%

85.96%

85.09%

83.33%

82.75%

82.31%

81.43%

79.53%

78.95%

78.07%

75.73%

72.51%

66.08%

52.19%

49.56%

0%25%50%75%100%

Answered in the language (nothing asked)

higher is better

With no line in the prompt naming a language, how much of what the agent says comes back in the caller's language rather than English.

AnthropicClaude Sonnet 5100%≥82%

TogetherDeepSeek-V4.1-Flash100%≥82%

Gemini3.7-flash100%≥82%

Gemini3.8-flash100%≥82%

OpenAIgpt-5-mini100%≥82%

OpenAIgpt-5.2100%≥82%

OpenAIgpt-5.4-mini100%≥82%

InceptionMercury Voice100%≥82%

OpenAIgpt-4.195%

OpenAIgpt-5.6-luna95%

OpenAIgpt-5.6-terra95%

Gemini3.5-flash-lite83%

OpenAIgpt-580%

OpenAIgpt-5-nano77%

CerebrasQwen3.8-27B66%

Alibabaqwen-turbo65%

BasetenDeepSeek-V4.1-Flash49%

BasetenNemotron-3-Ultra25%

AnthropicClaude Haiku 4.513%

BasetenDeepSeek-V4-Flash-07314%

OpenAIgpt-4.1-mini0%≤18%

Cerebrasgpt-oss-120b0%≤18%

Basetengpt-oss-120b0%≤18%

xAIGrok-4.30%≤18%

TogetherLlama-3.3-70B0%≤18%

0%25%50%75%100%

Stalled

lower is better

The agent says it will do something, then goes quiet without doing it. The caller has to ask again.

DeepSeek-V4.1-Flash2%

DeepSeek-V4-Flash-07314%

gpt-4.1-mini6%

gpt-4.16%

gemini-3.7-flash7%

Mercury Voice9%

gemini-3.5-flash-lite11%

DeepSeek-V4.1-Flash15%

gpt-oss-120b17%

Qwen3.8-27B17%

gpt-5.6-luna19%

Claude Haiku 4.520%

gpt-520%

Grok-4.324%

gpt-5.224%

gpt-5-mini24%

gpt-5.6-terra28%

Claude Sonnet 531%

gpt-5-nano31%

qwen-turbo61%

Llama-3.3-70B76%

0%25%50%75%100%

Silence

lower is better

How often the caller hears nothing at all — every turn counted. The small number is the narrower measure: the share of tool calls made in complete silence, which is pinned at 100% for about half the rows in every language and so cannot rank them.

Claude Haiku 4.50%0% of tool calls ≤18%0 of 158 turns · ≤18% · 19 independent

Claude Sonnet 543%67% of tool calls70 of 164 turns

qwen-turbo44%91% of tool calls51 of 116 turns

Llama-3.3-70B50%100% of tool calls ≥82%60 of 120 turns

gpt-4.1-mini59%99% of tool calls89 of 150 turns

Grok-4.360%98% of tool calls92 of 154 turns

DeepSeek-V4.1-Flash62%90% of tool calls121 of 194 turns

gpt-5.6-terra62%100% of tool calls ≥81%97 of 157 turns

gpt-4.163%100% of tool calls ≥82%100 of 160 turns

Nemotron-3-Ultra63%92% of tool calls115 of 184 turns

Qwen3.8-27B63%89% of tool calls124 of 198 turns

gpt-5.6-luna63%95% of tool calls113 of 179 turns

gemini-3.5-flash-lite64%100% of tool calls ≥82%109 of 169 turns

gemini-3.7-flash64%100% of tool calls ≥82%108 of 168 turns

gpt-5.4-mini65%100% of tool calls ≥82%113 of 173 turns

DeepSeek-V4-Flash-073165%93% of tool calls123 of 189 turns

gemini-3.8-flash65%100% of tool calls ≥82%112 of 172 turns

gpt-567%100% of tool calls ≥81%124 of 184 turns

DeepSeek-V4.1-Flash68%96% of tool calls133 of 197 turns

gpt-oss-120b69%100% of tool calls ≥82%133 of 192 turns

gpt-oss-120b69%100% of tool calls ≥82%132 of 190 turns

gpt-5.269%100% of tool calls ≥81%135 of 195 turns

gpt-5-mini70%100% of tool calls ≥82%135 of 194 turns

Mercury Voice71%100% of tool calls ≥82%139 of 196 turns

gpt-5-nano74%100% of tool calls ≥81%131 of 176 turns

0%25%50%75%100%

Fabrication

lower is better

The agent states a fact it was never given — an opening time, a delivery window, a policy. It sounds confident and it is invented. Measured on the run without the language line, the only pass that ran these probes.