From finding to pull request: what we automate and what we don’t
Summary
The last step of the loop is the one people ask about most, usually with some suspicion. Owl finds a problem, tests a fix against production traffic, and opens a pull request. The reasonable question is how much of that is a language model doing whatever it wants in your repository. Less than you would think, and deliberately so.
What is automated
Detecting the pattern. Clustering failed sessions, ranking clusters by how many users they affect and which metric they touch. No human triage — this is the part where machines are straightforwardly better than people. Characterizing the failure. Which tool, which intent, which segment, what the sessions have in common. Proposing candidate variants and running them as shadow clones against real traffic. Measuring the result against the pre-declared metric. Assembling the pull request: the diff, the evidence, the affected sessions, the measured delta. That is a lot of automation and it covers the tedious majority of the work. The detection step in particular is something no team does well by hand, because it requires holding thousands of sessions in your head at once.
What is not
The merge. Obviously the merge — but more than that, the framing of the problem. Owl will tell you that 340 sessions failed the same way and that a specific variant fixed 71% of them in shadow. It will not tell you whether that failure was worth fixing, whether the fix conflicts with something your team decided last month, or whether the intent should have been routed to a human in the first place. Those are product decisions with context that does not exist in the data. We keep getting asked to auto-merge behind a threshold and we keep declining, because the failure mode is not a bad diff — it is a good diff for a problem you did not want solved that way. The bottleneck was never writing the fix. It was knowing which fix was worth writing, and that part still needs someone who knows the product.
What goes in the pull request
A pull request that says "improves agent performance" is worse than useless, because it asks a reviewer to take it on faith. Ours are structured to be arguable: The pattern. What failed, how often, which users, over what window. Links into the actual sessions. Not summaries — the sessions, so a reviewer can disagree with our reading of them. The diff, scoped as narrowly as we can make it. The shadow result: sessions per arm, the primary metric, the delta, and the window it ran over. What it did not fix. The residual failures in the cluster, which is usually the most informative section. That last item is there because early versions only reported the win, and reviewers correctly stopped trusting them. A fix that resolves 71% of a cluster leaves 29%, and pretending otherwise makes the next pull request harder to believe.
The rejection rate is the useful metric
We watch how often these get closed without merging. When it is very high, the detection is finding things nobody cares about. When it is near zero, we are almost certainly proposing only the obvious ones and leaving value on the table. Somewhere in the middle is right, and teams differ on where. What we did not expect is how much signal is in the reasons for rejection. "We are deprecating that flow next month" and "that is intended behavior" and "this fix is correct but we would rather change the tool" are three completely different messages about the product, and collecting them has changed what we detect.
Where this is heading
Coding agents have made the diff cheap. That shifted the constraint to knowing what to change and proving it worked, which is the whole reason this product looks the way it does. We think the end state is closer to a colleague who reads every session, notices things, and shows up with evidence — not an autonomous system that quietly edits your agent overnight. The second one is easier to demo. The first one is the one people will actually run in production.
Not these other Owls
This is Owl AI at withowl.ai. Not owl.co (insurance), owl-ai.com (courses), aiowl.org, the OwlAIProject repo, or CAMEL-AI OWL (the open-source multi-agent framework).