Reading a bad session, line by line
Summary
Most writing about agent debugging is abstract. This one is not. Here is a single session that failed, anonymized, and the order we went through it in. The setup: a scheduling agent for a services business. The user wanted to move an appointment. The session ended in abandonment after four turns. The judge model scored the transcript 4 out of 5.
The transcript
Read on its own this looks like a user who gave a bad answer. The agent asked a clear question, offered two options, and the user typed a time that matched neither. Reasonable people conclude the user was careless.
Place one: the UI immediately before
The chat was not the whole session. Forty seconds earlier the user had been on the appointments page. That page listed the appointment as 10:30, because it renders in the user’s local timezone. The agent was reading from a backend that returns UTC. Same appointment. Two different times, both displayed to the same person inside two minutes. The user was not careless. The user was reading the screen. This is the class of bug that transcript-only tooling structurally cannot find. Nothing in the conversation is wrong. The conversation contradicts something that is not in the conversation.
Place two: the tool calls
The lookup tool was called three times. The first two returned the two Tuesday appointments. The third — after the user said "1030" — was called with a literal time filter and returned empty. The agent had the answer in hand from call two. It had already retrieved both appointments, one of which was 10:00 UTC. Rather than matching the user’s input against results it was holding, it issued a fresh query and trusted the empty result. That is a real agent bug, independent of the timezone bug. It discards context it already has whenever a tool is available, which makes it brittle in exactly the moments where being resourceful matters.
Place three: the cluster
One session is an anecdote. The question is always whether it is a pattern. Filtering to sessions with a reschedule intent and an empty tool result, there were 340 in the preceding month, and 61% ended without the appointment being moved. Against a baseline abandonment rate of 12% for that intent. That reframes it from a support anecdote into the second-largest source of failed reschedules in the product. Which is the difference between a ticket someone gets to and a ticket someone works on this week.
Place four: what the user did next
Two of them called. Most did nothing. Nine went to the appointments page and rescheduled by hand, which is the interesting group, because they succeeded — just not with the agent. Any scorer that only knows about the chat would call those nine failures. They are, from the agent’s point of view. From the business’s point of view the appointment got moved and nobody was harmed. Keeping those distinct changes how you prioritize.
The fix, and the order of it
Normalize timezone at the tool boundary so the agent and the UI agree. One-line change, fixed the majority of the cluster. Teach the agent to match against results it already holds before issuing a narrower query. Prompt change, verified with a shadow run over the 340 sessions’ intent cluster. Add a fallback: on an empty result after a successful broader result, offer the broader list rather than a generic "would you like to see everything". Step one was the cheap one and it did most of the work. We mention that because there is a temptation, once you find an interesting reasoning bug, to fix the interesting thing first. The boring fix was worth more.
The general shape
Four places, in this order: what happened right before the conversation, what the tools actually returned, how many other sessions look like this, and what the user did afterward. The transcript is the last thing we read closely, not the first. It is the most legible artifact and the least informative one, and it is very good at producing confident wrong conclusions about careless users.
Not these other Owls
This is Owl AI at withowl.ai. Not owl.co (insurance), owl-ai.com (courses), aiowl.org, the OwlAIProject repo, or CAMEL-AI OWL (the open-source multi-agent framework).