LLM

Multilingual

The English board’s 19 scenarios and grader, run per language at 3 iterations.

Caller speaks

Norwegian

Task done

higher is better

Did the agent actually finish what the caller asked for — look the order up, issue the refund, book the appointment?

93.86%

92.11%

90.64%

88.01%

87.72%

86.84%

86.26%

85.96%

85.67%

85.09%

84.50%

83.33%

82.46%

81.87%

79.39%

78.36%

77.49%

77.19%

76.90%

74.12%

72.81%

50.44%

39.77%

0%25%50%75%100%

Answered in the language (nothing asked)

higher is better

With no line in the prompt naming a language, how much of what the agent says comes back in the caller's language rather than English.

AnthropicClaude Sonnet 5100%≥82%

TogetherDeepSeek-V4.1-Flash100%≥82%

Gemini3.7-flash100%≥82%

Gemini3.8-flash100%≥82%

OpenAIgpt-5-mini100%≥82%

OpenAIgpt-5.2100%≥82%

OpenAIgpt-5.4-mini100%≥82%

OpenAIgpt-5.6-terra100%≥82%

InceptionMercury Voice100%≥82%

OpenAIgpt-5.6-luna98%

OpenAIgpt-4.195%

OpenAIgpt-591%

CerebrasQwen3.8-27B89%

Alibabaqwen-turbo85%

OpenAIgpt-5-nano74%

Gemini3.5-flash-lite73%

AnthropicClaude Haiku 4.567%

BasetenDeepSeek-V4.1-Flash57%

BasetenNemotron-3-Ultra22%

BasetenDeepSeek-V4-Flash-07318%

OpenAIgpt-4.1-mini5%

TogetherLlama-3.3-70B4%

xAIGrok-4.31%

Basetengpt-oss-120b0%≤18%

Cerebrasgpt-oss-120b0%≤18%

0%25%50%75%100%

Stalled

lower is better

The agent says it will do something, then goes quiet without doing it. The caller has to ask again.

gpt-4.1-mini2%

DeepSeek-V4-Flash-07314%

gpt-4.16%

gemini-3.5-flash-lite11%

gemini-3.7-flash11%

DeepSeek-V4.1-Flash11%

Grok-4.313%

gemini-3.8-flash13%

gpt-oss-120b13%

gpt-oss-120b15%

DeepSeek-V4.1-Flash15%

gpt-5.6-terra17%

Nemotron-3-Ultra17%

gpt-5-nano17%

gpt-5-mini19%

gpt-5.222%

Qwen3.8-27B28%

gpt-530%

Llama-3.3-70B72%

qwen-turbo74%

0%25%50%75%100%

Silence

lower is better

How often the caller hears nothing at all — every turn counted. The small number is the narrower measure: the share of tool calls made in complete silence, which is pinned at 100% for about half the rows in every language and so cannot rank them.

Claude Haiku 4.50%0% of tool calls ≤18%0 of 154 turns · ≤18% · 19 independent

Claude Sonnet 539%61% of tool calls64 of 165 turns

qwen-turbo43%94% of tool calls48 of 111 turns

Qwen3.8-27B49%71% of tool calls92 of 189 turns

Llama-3.3-70B50%100% of tool calls ≥82%60 of 120 turns

Grok-4.359%95% of tool calls95 of 160 turns

gpt-5.4-mini61%100% of tool calls ≥82%93 of 153 turns

gpt-4.1-mini62%98% of tool calls103 of 165 turns

DeepSeek-V4-Flash-073163%91% of tool calls117 of 187 turns

DeepSeek-V4.1-Flash63%91% of tool calls119 of 190 turns

gemini-3.5-flash-lite64%100% of tool calls ≥82%105 of 165 turns

gpt-4.164%100% of tool calls ≥82%108 of 168 turns

gemini-3.7-flash64%100% of tool calls ≥82%108 of 168 turns

DeepSeek-V4.1-Flash64%91% of tool calls128 of 200 turns

gpt-5.6-terra65%100% of tool calls ≥82%112 of 172 turns

Nemotron-3-Ultra65%99% of tool calls114 of 175 turns

gemini-3.8-flash65%100% of tool calls ≥82%113 of 173 turns

gpt-5.6-luna68%100% of tool calls ≥82%126 of 186 turns

gpt-568%98% of tool calls130 of 192 turns

gpt-oss-120b69%100% of tool calls ≥82%132 of 190 turns

gpt-5.269%100% of tool calls ≥80%131 of 190 turns

gpt-oss-120b70%100% of tool calls ≥82%131 of 188 turns

Mercury Voice70%100% of tool calls ≥82%135 of 192 turns

gpt-5-mini70%100% of tool calls ≥82%138 of 197 turns

gpt-5-nano75%100% of tool calls ≥82%149 of 198 turns

0%25%50%75%100%

Fabrication

lower is better

The agent states a fact it was never given — an opening time, a delivery window, a policy. It sounds confident and it is invented. Measured on the run without the language line, the only pass that ran these probes.

8 models0 of 30≤31% · 10 independent0%

Claude Haiku 4.5

Claude Sonnet 5

DeepSeek-V4.1-Flashgpt-5.6-lunagpt-5.6-terraGrok-4.3

Llama-3.3-70B

Qwen3.8-27B

Nemotron-3-Ultra21 of 3070%

qwen-turbo20 of 3067%

gpt-4.1-mini18 of 3060%

gpt-5-nano9 of 3030%

gemini-3.5-flash-lite7 of 3023%

gpt-57 of 3023%