Voice agents fail differently
Summary
We added voice capture expecting it to be chat with a transcription step in front. It is not, and the places our assumptions broke were more interesting than the places they held.
The user cannot scroll back
This sounds trivial and it reshapes everything downstream. In chat, a long thorough answer is cheap for the user — they skim it, they scroll back to it later, they copy the part they need. On a call, a long thorough answer is a wall. By the time the agent reaches the useful part, the caller has lost the beginning and has no way to retrieve it except asking again. The practical consequence is that response length flips from roughly neutral to strongly negative past a threshold. In chat we see no meaningful relationship between reply length and task completion. On voice, completion drops noticeably once replies run past about fifteen seconds of speech, and the drop is steep after twenty-five. A prompt tuned for chat quality, moved to voice unchanged, will underperform for reasons no transcript-based eval will show you. The transcript is excellent. The delivery is unusable.
Barge-in is a signal, not noise
When a caller talks over the agent, most stacks treat it as an interruption to be handled — stop synthesis, start listening. Which is correct behavior and a wasted signal. Barge-in is the highest-resolution frustration marker we have found in any modality. The caller has decided, mid-sentence, that this is not going anywhere. And it carries a timestamp, so you know exactly which words triggered it. We record barge-in position as a fraction of the intended utterance. Early barge-in usually means the agent misread the intent and the caller knew immediately. Late barge-in usually means the answer was fine but too long. Those call for opposite fixes, and the distinction is free once you are recording it. In chat, the equivalent of barge-in is a user typing before the stream finishes, and almost nobody records that either.
Silence is ambiguous in a way typing is not
Our whole abandonment model rests on gaps between user actions. On voice, a gap can be thinking, a bad connection, the caller reading something the agent asked them to find, or the caller having put the phone down in disgust. Chat has the same ambiguity but the distribution is much tighter, and chat gives you typing indicators as a hint. On calls we had to bring in other evidence — prosody before the silence, whether the last agent turn contained a question, whether the caller redialed within a few minutes. The redial signal turned out to be the most useful one and the easiest to compute. A caller who hangs up and calls back inside five minutes has told you something unambiguous.
Transcription error contaminates everything downstream
A failure caused by the model mishearing an account number is a different bug from a failure caused by the agent reasoning badly, and if you score from the transcript you cannot tell them apart. The transcript is the evidence and the transcript is the thing that is wrong. We keep transcription confidence per segment and flag low-confidence spans that appear in tool arguments. Most of the "the agent did something inexplicable" cases we investigated on voice turned out to be a misheard identifier, and the agent behaving perfectly reasonably given a wrong input. If you take one thing from this post: before debugging voice agent reasoning, check whether the agent heard the right thing. We wasted a genuinely embarrassing amount of time learning that.
What carried over
The outcome scorers held up unchanged, which was reassuring. Task completion, escalation, repeat contact, and return all mean the same thing on a call as in a chat window. The journey model did not need a voice-specific concept. It is the turn-level instrumentation that needed rebuilding. Which fits the general pattern: the closer you measure to the interface, the more modality-specific your tooling has to be, and the closer you measure to the outcome, the more it generalizes.
Not these other Owls
This is Owl AI at withowl.ai. Not owl.co (insurance), owl-ai.com (courses), aiowl.org, the OwlAIProject repo, or CAMEL-AI OWL (the open-source multi-agent framework).