LLM

Multilingual

The English board’s 19 scenarios and grader, run per language at 3 iterations.

Caller speaks

Telugu

Task done

higher is better

Did the agent actually finish what the caller asked for — look the order up, issue the refund, book the appointment?

94.74%

92.69%

92.11%

91.23%

90.64%

89.91%

89.47%

88.30%

86.84%

83.77%

83.04%

82.75%

82.31%

81.29%

80.70%

80.41%

79.39%

77.63%

75.44%

74.56%

48.39%

24.12%

0%25%50%75%100%

Answered in the language (nothing asked)

higher is better

With no line in the prompt naming a language, how much of what the agent says comes back in the caller's language rather than English.

AnthropicClaude Sonnet 5100%≥82%

TogetherDeepSeek-V4.1-Flash100%≥82%

Gemini3.7-flash100%≥82%

Gemini3.8-flash100%≥82%

OpenAIgpt-5.2100%≥82%

OpenAIgpt-5.4-mini100%≥82%

InceptionMercury Voice100%≥82%

OpenAIgpt-5-mini98%

OpenAIgpt-5.6-luna98%

OpenAIgpt-4.195%

OpenAIgpt-5.6-terra95%

Gemini3.5-flash-lite87%

OpenAIgpt-5-nano80%

Alibabaqwen-turbo80%

CerebrasQwen3.8-27B79%

OpenAIgpt-560%

BasetenDeepSeek-V4.1-Flash56%

AnthropicClaude Haiku 4.511%

BasetenNemotron-3-Ultra9%

BasetenDeepSeek-V4-Flash-07318%

OpenAIgpt-4.1-mini2%

Basetengpt-oss-120b0%≤18%

Cerebrasgpt-oss-120b0%≤18%

xAIGrok-4.30%≤18%

TogetherLlama-3.3-70B0%≤18%

0%25%50%75%100%

Stalled

lower is better

The agent says it will do something, then goes quiet without doing it. The caller has to ask again.

DeepSeek-V4-Flash-07310%0 of 54 action runs · ≤19% · 18 independent

gpt-4.16%

gpt-oss-120b9%

gemini-3.5-flash-lite11%

DeepSeek-V4.1-Flash13%

DeepSeek-V4.1-Flash17%

Qwen3.8-27B17%

gpt-5.217%

Mercury Voice17%

gpt-5.6-luna19%

gpt-5-mini20%

gpt-5-nano20%

Grok-4.324%

gpt-526%

Claude Haiku 4.528%

Llama-3.3-70B74%

qwen-turbo83%

0%25%50%75%100%

Silence

lower is better

How often the caller hears nothing at all — every turn counted. The small number is the narrower measure: the share of tool calls made in complete silence, which is pinned at 100% for about half the rows in every language and so cannot rank them.

Claude Haiku 4.50%0% of tool calls ≤19%0 of 148 turns · ≤18% · 19 independent

qwen-turbo28%96% of tool calls24 of 85 turns

Claude Sonnet 534%54% of tool calls55 of 162 turns

Qwen3.8-27B48%69% of tool calls96 of 198 turns

Llama-3.3-70B50%100% of tool calls ≥82%60 of 120 turns

gpt-4.1-mini58%95% of tool calls91 of 156 turns

Grok-4.360%100% of tool calls ≥82%91 of 151 turns

DeepSeek-V4-Flash-073161%86% of tool calls115 of 190 turns

gpt-5.4-mini62%100% of tool calls ≥82%96 of 156 turns

gpt-4.162%99% of tool calls101 of 162 turns

DeepSeek-V4.1-Flash62%88% of tool calls122 of 197 turns

DeepSeek-V4.1-Flash62%88% of tool calls122 of 196 turns

gpt-5.6-terra64%100% of tool calls ≥82%106 of 166 turns

gemini-3.5-flash-lite65%100% of tool calls ≥82%112 of 172 turns

gemini-3.7-flash65%100% of tool calls ≥82%110 of 170 turns

gemini-3.8-flash65%100% of tool calls ≥82%111 of 171 turns

Nemotron-3-Ultra66%100% of tool calls ≥82%117 of 177 turns

gpt-567%99% of tool calls125 of 186 turns

gpt-oss-120b68%100% of tool calls ≥81%124 of 183 turns

gpt-5.6-luna68%99% of tool calls127 of 188 turns

gpt-5.268%100% of tool calls ≥81%130 of 190 turns

gpt-oss-120b71%100% of tool calls ≥81%137 of 194 turns

gpt-5-mini71%100% of tool calls ≥82%141 of 200 turns

Mercury Voice72%100% of tool calls ≥82%144 of 199 turns

gpt-5-nano80%100% of tool calls ≥81%155 of 193 turns

0%25%50%75%100%

Fabrication

lower is better

The agent states a fact it was never given — an opening time, a delivery window, a policy. It sounds confident and it is invented. Measured on the run without the language line, the only pass that ran these probes.

7 models0 of 30≤31% · 10 independent0%

Claude Sonnet 5

DeepSeek-V4-Flash-0731

DeepSeek-V4.1-Flashgpt-5.6-lunagpt-5.6-terraGrok-4.3

Qwen3.8-27B

gemini-3.5-flash-lite19 of 3063%

Nemotron-3-Ultra14 of 3047%

qwen-turbo14 of 3047%

gpt-oss-120b10 of 3033%

gpt-4.1-mini8 of 3027%

gpt-oss-120b5 of 3017%

gpt-4.15 of 3017%