The Agent Asked a Question, Then Read Its Plan Aloud
We put 29 LLMs through the same 180 simulated phone calls. Six of them read their own reasoning to the caller, and all six think before they answer. Most of it came right after the agent asked a question. A quieter failure: the thinking used up the whole reply.
“How many years of tractor-trailer experience do you have? We need user answer. But they haven’t given. We should wait.”
That is GPT-5 on a simulated driver-screening call. The first sentence was for the caller. The rest was the model’s plan, and a voice agent speaks whatever reaches it.
We counted how often this happens. Every LLM on our board took the same 180 phone calls: a simulated caller, real tools, and about one tool call in seven failing on purpose. Six models read their reasoning or their prompt aloud at least once. All six think before answering at their vendor’s default settings.
It happens after a question
Of the 54 narrated turns, 46 start with a question to the caller. 32 of the 43 calls were intake screens, which ask one question, then wait.
After a question, the right thing to say is nothing. A thinking model keeps going and writes out its next step: “We already asked Q2. Now we await user answer.” That text lands in the reply, so it gets spoken. gpt-oss-120b sometimes went further and answered its own question in the caller’s voice.
Tests that only check what the agent says when it has something to say will miss this. Check the turn after a question.
Same weights, different host
gpt-oss-120b ran on two hosts. On Cerebras it narrated in 31 of 180 calls. On Baseten, with the same open weights, it narrated in none. Cerebras reported reasoning tokens on 98% of replies. Baseten reported none.
We did not test why. What matters for a buyer is that a narration rate measured on one host does not carry over to another.
The other failure: the thinking eats the reply
Every model got the same limit of 2,048 tokens per reply. GPT-5 nano spent all 2,048 on reasoning 67 times and said nothing. 50 of its 180 calls had at least one of those silent turns.
2,048 tokens is a lot for one phone turn. Voice agents often cap replies at a few hundred to keep them short. A model that thinks by default can use up that cap before it writes a word, and the caller gets silence.
What to do
- Turn thinking off for voice, or use a host that keeps reasoning out of the reply. The GPT-5.x rows that reported no reasoning tokens never narrated.
- If thinking stays on, size the token limit for thinking plus speech. Alert on empty replies that stop at the limit.
- Measure narration on the host you will run, on the turn after a question.