What we actually mean by an outcome scorer
Summary
We say Owl scores agents on what users did next. Fair enough as a slogan. It skips every hard question, so this post is the long version. An outcome scorer is a function from a session to a label, where the label is derived from observed user behavior rather than from a model reading the transcript. Five signals cover most of what we build. Each one has a way of lying to you.
1. Task completion
The user came in wanting something. Did it happen? Not "did the agent say it happened" but did the state change in your product. This is the strongest signal and the most work to wire up, because it requires the agent conversation and the product event to be on the same journey. If your chat widget and your backend analytics are separate systems keyed on separate IDs, you cannot compute it, which is a big part of why most teams do not. The trap: completion attributed too widely. A user who was going to complete anyway, who happened to ask the agent something along the way, is not evidence the agent helped. We window it — the completion has to follow the relevant turn, not merely coexist with it in the session.
2. Intent restatement
The user asks, gets an answer, and asks the same thing again in different words. This is the single most underrated failure signal we have found, and almost nobody instruments it. It is powerful because it is the user telling you, in their own behavior, that the answer did not land. No rubric needed. It is also cheap to compute once you are clustering intents anyway. The trap: legitimate refinement looks identical to restatement at the surface level. "How do I export this" followed by "how do I export this as CSV" is progress, not failure. We compare against the clustered intent rather than the literal text, and we treat a narrowing follow-up differently from a repeat.
3. Escalation
The user goes to a human. Ticket, live chat, phone, the "talk to someone" button. Unambiguous and easy to join, which makes it a good scorer to start with if you are wiring your first one. The trap: escalation is sometimes the correct outcome. An agent that recognizes a billing dispute and hands off quickly is doing its job. Scoring all escalation as failure trains you to build an agent that stonewalls people. We label the intents where handoff is the goal and score those in reverse — fast, clean handoff is a win.
4. Abandonment
No completion, no escalation, no further activity. The user left. The trap here is the whole game, and it is the timeout. Pick it too short and you label lunch breaks as failures. Too long and a genuinely frustrated exit gets folded into normal usage. There is no universal number, which is why we derive the window per product from the observed distribution rather than shipping a default and pretending it generalizes. For most B2B SaaS we look at the gap distribution and cut somewhere around the point where return probability flattens. In practice that lands between 60 and 180 seconds far more often than not, but we have seen products where the honest number was closer to ten minutes.
5. Return with the same intent
The slowest signal and the one that catches the failures the others miss. The session looked fine. The user completed. And then they were back a week later with the same problem. That pattern usually means the agent gave a workaround instead of a fix, and it is invisible to anything that scores a single session in isolation. The trap: recurring intents that are supposed to recur. Someone checking their invoice every month is not a failure. Seasonality matters and you need enough history before this scorer says anything trustworthy.
Why these cannot be defaults
Every one of the five depends on things only your product knows. What counts as completion. Which intents should escalate. How long a normal pause is. Which repeats are healthy. This is the part we think the eval tooling space has mostly dodged, because shipping a rubric is easy and shipping something that learns your product is not. A rubric ships on day one and looks impressive. It also grades the transcript, and by now you know how we feel about that. Owl derives these per customer from captured behavior, then lets you correct them. The correction step matters more than we expected. Teams almost always disagree with the first pass on at least one intent, and they are almost always right, because they know something about their users that is not in the data yet.
Where a judge still earns its keep
Outcome scorers tell you that something went wrong. They are bad at telling you why. A judge model reading the transcript of a failed session is genuinely useful for that second question — it can spot the hallucinated policy, the ignored constraint, the tool the agent should have called and did not. So we run both, in that order. Outcomes to find the failures. A judge to characterize them. Using the judge for the first job is where teams get into trouble.
Not these other Owls
This is Owl AI at withowl.ai. Not owl.co (insurance), owl-ai.com (courses), aiowl.org, the OwlAIProject repo, or CAMEL-AI OWL (the open-source multi-agent framework).