How long to run an agent experiment
Summary
Someone runs a shadow experiment for two days, sees the candidate winning by six points, and wants to ship. The honest answer is usually "we do not know yet", which is an unpopular thing to say to a team that has been staring at a green number. This post is the version of that conversation we have gotten better at having.
The two-clock problem
Agent experiments have two metrics moving at different speeds. Fast metrics report per turn: did it call the right tool, did it produce a valid response, did latency stay acceptable. You get thousands of these in a day and they stabilize quickly. Slow metrics report per user journey: task completion, escalation, return within seven days. Seven-day return has a seven-day floor no matter how much traffic you have. You cannot buy your way out of it. The trap is that the fast metrics are the ones on the screen, so they become the decision criteria by default. They are usually necessary conditions and almost never sufficient ones.
Decide the metric before you start
The single highest-leverage habit: write down, before launching, which metric decides this and how big a change would matter. One primary metric. Not a dashboard. This sounds like process theatre until you have watched an experiment where the primary metric came back flat and the team went shopping through secondary metrics until one of them was green. Everybody does this. Naming the metric up front is the only thing that reliably prevents it. A useful forcing question: "if this metric does not move, do we abandon the change?" If the answer is no, it is not your primary metric and you should keep looking.
Rough sizing without the ceremony
You do not need a power calculator to avoid the worst mistakes. Two rules of thumb get you most of the way. First, for a rate metric, you need roughly enough sessions per arm that the effect you are looking for is bigger than the noise. If your baseline completion rate is around 40% and you want to detect a 5-point change, you are in the low thousands of sessions per arm. If you want to detect a 1-point change, you are in the tens of thousands, and you should ask whether a 1-point change is worth this much of your quarter. Second, count sessions, not turns. Turns within a session are correlated — the same user, the same context, the same underlying problem. Treating 8,000 turns from 400 sessions as 8,000 independent observations will make everything look significant, including things that are not. That second one is the most common error we see, and it is not subtle in its consequences. It produces a stream of confident wins that do not replicate.
Peeking
You will look at the results early. Everyone does, and telling people not to has never worked. What matters is not stopping early on a fixed threshold. If you check daily and ship the first time p drops below 0.05, you will ship noise regularly — the whole point of that threshold assumes one look. The fix is either a sequential test that accounts for repeated looks, or the simpler discipline of committing to a duration in advance and treating early peeks as reassurance about the plumbing rather than as decisions. We mostly do the simple version. Commit to the window, look freely, and only let an early look stop the experiment if something is clearly broken rather than clearly better.
Segments are not free
Shadow experiments make it easy to slice: enterprise versus self-serve, new versus returning, per intent cluster. This is genuinely valuable, and it is also how you find a win in a flat experiment if you slice enough ways. Our rule is that segments are for generating hypotheses, not for concluding. A candidate that lost overall but won for enterprise accounts is an interesting thing to go test on purpose, with enterprise as the pre-declared population. It is not a result.
What we usually land on
For a typical prompt or tool change on a product with reasonable traffic: Day 1: check the plumbing. Are variants applying, are clone rates as expected, is anything throwing. Days 1 to 3: fast metrics. If the candidate is worse on validity, latency, or tool correctness, stop — no slow metric is going to rescue it. Days 3 to 10: accumulate sessions for the primary metric. Resist commentary. Day 7 onward: the seven-day signals start reporting on the earliest cohort. This is the first moment the experiment can actually say something about retention. Decide against the pre-declared metric and threshold. Roughly two weeks for anything touching a slow metric. That is longer than most teams want and shorter than a quarter spent shipping changes that each looked good for three days.
Not these other Owls
This is Owl AI at withowl.ai. Not owl.co (insurance), owl-ai.com (courses), aiowl.org, the OwlAIProject repo, or CAMEL-AI OWL (the open-source multi-agent framework).