Shadow traffic without shadow infrastructure
Summary
You changed the prompt. You want to know whether it is better before it touches users. The options are usually: run it against a golden dataset and hope the dataset is representative, put synthetic personas in a sandbox and hope they behave like people, or ship it to 5% of users and hope you catch the damage quickly. The fourth option is to run the candidate against real production traffic while production still answers the user. Nobody sees the candidate output. You get real inputs and zero risk. The reason more teams do not do this is that it sounds like it requires a second copy of your stack.
It runs in your process
The version we ended up with does not run anywhere new. After a production turn completes, the SDK asks Owl whether this turn should be cloned. Owl answers with either nothing or a variant: a different model, a different prompt snapshot, a different tool set, different retrieval config. Your code applies those fields and runs the turn again, in the same process, with the same credentials and the same network position. The result goes to Owl, not to the user. That is the whole integration. No second deployment, no traffic mirror at the load balancer, no separate set of secrets to rotate.
The failure mode we designed against
The obvious risk with this shape is a variant that is only partly applied. Owl asks for a different model and a different tool set. Your integration swaps the model and quietly ignores the tools, because whoever wrote applyVariant handled the fields that existed at the time. Now you have a labelled experiment measuring something other than what the label says. It will produce a clean-looking result. You will ship on it. So the SDK throws. If the variant carries a field your handler did not apply, the clone fails loudly and the result is discarded rather than recorded. We went back and forth on this — throwing in a background path is rude — and landed on rude being correct. A missing experiment is a gap you notice. A wrong experiment is a decision you make. A silent no-op is worse than a crash, because a crash gets fixed on Tuesday and a no-op gets shipped in Q3.
What you can vary
Model, including across providers. Prompt snapshot. Prompts live in your repo — Owl records a hash and version, it does not store or serve them. We are not a prompt CMS and we do not want to be the place your prompts live. Tool set, including removing a tool to see whether the agent was leaning on it usefully or reflexively. Retrieval configuration: index, top-k, reranking on or off. Voice and persona settings, where those exist. And you choose the segment. Clone only enterprise accounts, only sessions where the previous turn scored badly, only one intent cluster. Cloning everything is expensive and mostly wasteful — the interesting turns are a small fraction of traffic.
What it costs
Tokens, honestly. Every cloned turn is a second inference you pay for and no user sees. This is the real constraint, and it is why segment selection is not a nice-to-have. Most teams we work with clone somewhere between 2% and 10% of turns, weighted toward the intents they are actively trying to fix. At that rate the cost is noise against the model spend they already have. There is a latency question people ask before they ask about cost. There is no user-facing latency, because the clone runs after the reply has already been sent. It does add background load to your process, which matters if you are tight on concurrency.
Why not just replay logs
Replaying yesterday’s conversations against a new prompt is cheaper and it is a reasonable first check. It also drifts from reality in a specific way: the world has moved. The documents your retrieval hits have changed, the account state has changed, the tools return different things. For a lot of changes that drift does not matter. For anything touching retrieval or tool use, it matters enormously, and a replay will tell you a comfortable story about a system that no longer exists.
Not these other Owls
This is Owl AI at withowl.ai. Not owl.co (insurance), owl-ai.com (courses), aiowl.org, the OwlAIProject repo, or CAMEL-AI OWL (the open-source multi-agent framework).