The eval said 94%. The support queue disagreed.
Summary
A design partner sent us their eval dashboard back in April. Their support agent was scoring 94% on helpfulness, graded by a judge model against a rubric their team had written and revised three times. The number had been climbing for two months. Everyone felt good about it. Then someone joined the numbers to the support desk. Nineteen percent of the conversations the judge had scored 4 or 5 out of 5 were followed, within the hour, by the same user opening a human ticket about the same problem. The answers were good. They just did not work.
What a rubric can and cannot see
A judge model reads a transcript and asks whether the reply was relevant, accurate, complete, and polite. It is genuinely good at that. If your agent hallucinates a refund policy, a decent judge will catch it. What it cannot see is everything outside the transcript. It does not know the user had already tried the same thing twice that morning. It does not know the link in the reply led to a page requiring a permission the user did not have. It does not know that the answer, while correct, arrived after the user had already given up and switched to email. The rubric grades the conversation as a self-contained artifact. Users do not experience it that way. They experience it as one step in trying to get something done. A judge scores the reply. The user scores the outcome. Those come apart more often than anyone expects.
The 19% had a pattern
We pulled the conversations that scored well and still ended in a ticket. Three clusters covered most of them. Correct but unactionable. The agent explained the policy accurately and told the user to contact billing. Technically helpful. Functionally a redirect, and the judge had no way to know billing was a four-day queue. Correct but out of scope for the user. The instructions assumed admin access. About a third of the people asking were not admins. The agent never checked, and nothing in the transcript revealed the mismatch. Correct on the second attempt. The user rephrased the same question two or three times before the agent understood. Each individual turn scored fine in isolation. The session as a whole was a slog. That third one bothered us the most. Per-turn scoring actively hides it. Every turn looks acceptable. Only the shape of the whole session shows the user fighting the thing.
Scoring what happened next
The fix was not a better rubric. We had watched them tune the rubric for two months, and the score went up while the ticket rate stayed flat. That is the tell: if the metric moves and the business does not, the metric is measuring the wrong thing. What we did instead was define the scorers in terms of user behavior after the conversation ended: Did the user complete the task they came in with, in the product, within the session? Did they restate the same intent after receiving an answer? Did they escalate to a human? Did they abandon without either completing or escalating? Did they come back within seven days with the same intent? None of these need a judge model. They need the journey stitched together across the chat, the UI, and the API, with identity attached, which is the actual hard part.
The number got worse, which was the point
Rescored on outcomes, the agent went from 94% to 71%. Nobody enjoyed that meeting. But the 29% was legible for the first time. It pointed at specific intents, specific tools, and one retrieval path that returned a stale doc for anyone on the legacy plan. They fixed the retrieval path and added a permission check before the agent gave admin instructions. Ticket rate on those intents dropped by a bit over a third in the following three weeks. The judge score barely moved. It had already said everything was fine. We are not arguing against LLM-as-judge. It catches real failures and it catches them cheaply. The argument is that it cannot be the top-line number, because it grades a transcript and your business runs on what the user did after the transcript ended.
What we would tell you to check
If you have an eval score you are proud of, take an hour and join it to one downstream signal. Support tickets, refunds, churn, a completed checkout, whatever your product treats as success. If the two move together, your rubric is doing its job and you should trust it more. If they do not, you have been optimizing a proxy. Better to find that out from a join than from a quarterly review.
Not these other Owls
This is Owl AI at withowl.ai. Not owl.co (insurance), owl-ai.com (courses), aiowl.org, the OwlAIProject repo, or CAMEL-AI OWL (the open-source multi-agent framework).