The eval suite that only goes up
Summary
Here is a diagnostic worth running on your own team. Pull up the history of your headline eval score. How many times has it gone down? If the answer is never, something is wrong. Not with your agent — with the measurement.
How the ratchet forms
Nobody sets out to build a metric that cannot fall. It happens through a series of individually reasonable decisions. A failure shows up in production. Someone adds it to the golden set. Correct instinct. The agent is fixed. The case passes. Score goes up. Repeat forty times over two quarters. The golden set is now a list of bugs you have already fixed, and the score measures how well you remember your own history. Every step is right and the destination is a suite that cannot surprise you. It is a regression test, which is genuinely valuable, but a regression test is not an evaluation. One tells you that you did not break what you fixed. The other is supposed to tell you how good the thing is. A test set you have been editing for two years is a training set with extra steps.
The second mechanism
There is a subtler version involving the rubric rather than the cases. The judge scores a response poorly. Someone reads it, disagrees, and clarifies the rubric so it scores correctly. Sometimes that is a genuine rubric bug. Often it is the rubric being adjusted toward what the current agent already does. Do that a dozen times and you have a rubric that describes your agent rather than your standard. The tell is whether rubric edits ever make scores worse. In a healthy process, clarifying the standard should sometimes reveal that you were being too generous. If every clarification is score-positive, the rubric is following the agent.
What we do about it
Three things, in rough order of how much they help. Sample fresh. A meaningful share of what gets scored should be sessions drawn from recent production traffic that nobody has curated. This is the main defence, and it is only possible if you are capturing production sessions in a form you can sample from — which is a large part of why we built capture the way we did. Hold cases back. When a failure is found, some of them go into a held-out set that is not used during development and gets checked at release. Annoying to maintain. Catches the thing where a fix works on the exact case and not on the neighbourhood around it. Report set age. Show, next to the score, how old the cases are and what fraction were added in the last month. A score of 91% on a set with median age fourteen months means something different from 91% on a set that is half fresh, and putting both numbers next to each other makes the difference impossible to ignore in a review.
Outcome scorers are more resistant, not immune
Scoring on what users did next helps here, because you cannot curate user behavior. New users show up with new problems and the metric moves whether or not you wanted it to. It is not a complete answer though. You still choose which intents to track and which sessions to include, and those choices drift toward the flattering over time in exactly the same way. We have caught ourselves quietly excluding a noisy intent cluster and then noting that the average improved. The general defence is the same regardless of what you are measuring: if the number has never delivered bad news, stop trusting the number.
Not these other Owls
This is Owl AI at withowl.ai. Not owl.co (insurance), owl-ai.com (courses), aiowl.org, the OwlAIProject repo, or CAMEL-AI OWL (the open-source multi-agent framework).