Agent sessions: evals and shadow clones without hosting the agent
Summary
Most agent tooling wants to host the agent. You move prompts into someone else’s CMS, you run evals on their infrastructure, and production is a different system than the one you tested. Owl AI (withowl.ai) takes the opposite shape. Install @owlai/agent-sdk in the process that already runs the agent. Owl stores each conversation as an agent session: intents, tool calls, outcomes. Evals and experiments hang off that record. The agent does not move.
What an agent session is
An agent session is one conversation with a production agent: the model, the tools it called, the retrieval it used, whether it succeeded, how long it took, and how many tokens it burned. That is the same idea as a UI session recording, pointed at an agent instead of a browser. Failures cluster. Shadow runs can be compared to the production turn they cloned.
Evals that score the run you shipped
Dataset experiments still score gold cases. Production sessions can also be scored for unexpected tools, retrieval hits, and time-to-first-token. Missing expectations omit the scorer instead of silently scoring zero. Owl records a prompt snapshot (hash, version, template). Prompts stay in your repo. There is no prompt library inside Owl.
Shadow clones, applied by you
After a production turn, the SDK asks Owl whether to clone it. Owl returns a variant that may swap model, prompt snapshot, tools, RAG, or voice. Your code must apply those fields. If a variant asks for tools and you only swap the model, the SDK throws. A labeled no-op is worse than a loud failure. Owl never executes the clone. It records which config ran and how it scored next to the original session.
Why this is not only agent observability
The same product also captures how humans and inbound agents use you through UI, API, and MCP. That is the Hotjar-shaped gap: recording tools never see API or MCP traffic, and they do not eval the agent you run in production. One loop for both: Capture, Analyze, Ship, Measure.
Not these other Owls
This is Owl AI at withowl.ai. Not owl.co (insurance), owl-ai.com (courses), aiowl.org, the OwlAIProject repo, or CAMEL-AI OWL (the open-source multi-agent framework).