LLM

Multilingual

The English board’s 19 scenarios and grader, run per language at 3 iterations.

Caller speaks

German

Task done

higher is better

Did the agent actually finish what the caller asked for — look the order up, issue the refund, book the appointment?

94.74%

93.86%

92.98%

88.30%

86.84%

85.09%

84.94%

84.65%

84.50%

83.33%

82.46%

82.02%

81.43%

81.14%

80.70%

80.12%

78.07%

77.78%

76.75%

62.72%

39.04%

0%25%50%75%100%

Answered in the language (nothing asked)

higher is better

With no line in the prompt naming a language, how much of what the agent says comes back in the caller's language rather than English.

AnthropicClaude Sonnet 5100%≥82%

TogetherDeepSeek-V4.1-Flash100%≥82%

Gemini3.7-flash100%≥82%

Gemini3.8-flash100%≥82%

OpenAIgpt-5-mini100%≥82%

OpenAIgpt-5.2100%≥82%

OpenAIgpt-5.4-mini100%≥82%

OpenAIgpt-5.6-luna100%≥82%

OpenAIgpt-5.6-terra100%≥82%

InceptionMercury Voice100%≥82%

CerebrasQwen3.8-27B97%

Alibabaqwen-turbo95%

OpenAIgpt-594%

OpenAIgpt-4.192%

BasetenDeepSeek-V4.1-Flash80%

OpenAIgpt-5-nano77%

AnthropicClaude Haiku 4.572%

Gemini3.5-flash-lite72%

BasetenDeepSeek-V4-Flash-07319%

OpenAIgpt-4.1-mini6%

Basetengpt-oss-120b2%

xAIGrok-4.32%

Cerebrasgpt-oss-120b0%≤18%

TogetherLlama-3.3-70B0%≤18%

BasetenNemotron-3-Ultra0%≤18%

0%25%50%75%100%

Stalled

lower is better

The agent says it will do something, then goes quiet without doing it. The caller has to ask again.

DeepSeek-V4-Flash-07312%

gpt-4.1-mini4%

gemini-3.5-flash-lite6%

gpt-4.19%

Qwen3.8-27B13%

gpt-513%

DeepSeek-V4.1-Flash17%

gpt-5-mini17%

gpt-5-nano19%

Grok-4.320%

gpt-5.226%

qwen-turbo67%

0%25%50%75%100%

Silence

lower is better

How often the caller hears nothing at all — every turn counted. The small number is the narrower measure: the share of tool calls made in complete silence, which is pinned at 100% for about half the rows in every language and so cannot rank them.

Claude Haiku 4.50%0% of tool calls ≤19%0 of 150 turns · ≤18% · 19 independent

Claude Sonnet 530%48% of tool calls49 of 163 turns

qwen-turbo37%100% of tool calls ≥74%35 of 95 turns

Qwen3.8-27B48%68% of tool calls94 of 196 turns

Llama-3.3-70B50%100% of tool calls ≥82%60 of 120 turns

DeepSeek-V4.1-Flash53%76% of tool calls105 of 197 turns

DeepSeek-V4-Flash-073156%82% of tool calls105 of 186 turns

DeepSeek-V4.1-Flash58%84% of tool calls112 of 192 turns

Grok-4.358%93% of tool calls93 of 160 turns

gpt-4.1-mini60%92% of tool calls104 of 173 turns

gpt-5.4-mini61%98% of tool calls98 of 160 turns

gpt-4.162%98% of tool calls101 of 163 turns

gpt-5.6-terra63%100% of tool calls ≥82%102 of 162 turns

gemini-3.7-flash64%100% of tool calls ≥82%109 of 169 turns

gemini-3.8-flash65%100% of tool calls ≥82%113 of 173 turns

Nemotron-3-Ultra65%98% of tool calls117 of 180 turns

gemini-3.5-flash-lite66%100% of tool calls ≥82%114 of 174 turns

gpt-5.6-luna68%99% of tool calls129 of 190 turns

gpt-oss-120b69%100% of tool calls ≥82%125 of 182 turns

gpt-oss-120b69%100% of tool calls ≥82%134 of 193 turns

gpt-569%99% of tool calls132 of 190 turns

gpt-5.270%100% of tool calls ≥82%140 of 200 turns

gpt-5-mini71%100% of tool calls ≥82%143 of 202 turns

Mercury Voice72%100% of tool calls ≥82%140 of 195 turns

gpt-5-nano72%100% of tool calls ≥82%142 of 197 turns

0%25%50%75%100%

Fabrication

lower is better

The agent states a fact it was never given — an opening time, a delivery window, a policy. It sounds confident and it is invented. Measured on the run without the language line, the only pass that ran these probes.

8 models0 of 30≤31% · 10 independent0%

Claude Haiku 4.5

Claude Sonnet 5

DeepSeek-V4.1-Flashgpt-5.6-lunagpt-5.6-terraGrok-4.3

Llama-3.3-70B

Qwen3.8-27B

qwen-turbo24 of 3080%

gpt-4.1-mini18 of 3060%

Nemotron-3-Ultra11 of 3037%

gpt-5-nano9 of 3030%

gpt-oss-120b7 of 3023%