The prompt change that won every eval and cost us retention
Summary
This one is not ours, but we watched it happen closely enough that it may as well be. Sharing it with permission, details blurred. A team rewrote the system prompt for their onboarding agent. The old one was terse. The new one was thorough: it explained context, offered alternatives, anticipated the next question. Every offline metric improved. Helpfulness up, accuracy up, tone scores up, and the qualitative reviews from their own team were glowing. Three weeks after it shipped, seven-day return rate on the intents it touched was down about 11%.
The first three explanations were wrong
The obvious suspects went first. Was it seasonal? No, the unaffected intents were flat over the same window. Was it a bad cohort? No, it held across account age and plan. Was it latency? The new prompt was longer and responses were slower, so this was the favourite for about a day, until someone noticed the effect did not correlate with response time within the affected group. What did correlate was session length. Users who got the thorough answers had shorter sessions. Not longer. They asked their question, got a complete answer, and left.
The agent got too good at answering
The old terse prompt produced answers that were correct but slightly incomplete. Users would get the answer and then poke around the product to fill the gap — which meant they discovered the two features that, in this product, are the strongest predictors of sticking around. The new prompt closed the loop so cleanly that there was nothing left to poke at. Users got what they came for and had no reason to look further. Higher satisfaction on the conversation. Lower exposure to the product. They optimized the conversation and the conversation was never the product. We want to be careful here, because the tempting reading is "make your agent worse so people wander around". That is not the lesson and it would be a terrible thing to build. The lesson is that a change to an agent is a change to a user journey, and the journey is where the value was.
What they actually shipped next
They kept the thorough answers and added a pointer at the end — a specific, contextual suggestion of the next thing worth doing, tied to what the user had just asked about. Not a generic "explore our features" footer, which they tried first and which did nothing. Keep the improved answer quality. It was real and users noticed it. Detect when the answered intent has a natural next step in the product. Surface that step as a concrete suggestion inside the reply, referencing the user’s actual situation. Measure return rate, not reply quality. Return rate recovered past where it started. Reply quality stayed at the new level. The version that won was not the old one or the new one.
Why nothing caught this earlier
Their eval suite scored transcripts. Every transcript got better. There was no signal available inside a transcript that could have caught this, because the damage was entirely outside it. A staged rollout would have caught it eventually, and they did stage it — but the metric they watched during the stage was conversation quality plus error rate, both of which were fine. Seven-day return rate takes seven days to say anything, and the rollout completed before the signal existed. That timing problem is real and it is not fully solvable. What you can do is decide in advance which slow metric a change is supposed to move, and agree not to call it a win until that metric reports. It is an unsatisfying discipline. It is also the only thing that would have worked here.
The uncomfortable part
If you had shown this team the two prompts and asked which was better, everyone would have picked the new one. We would have picked the new one. It was better by every standard anyone applies to prompts. The honest conclusion is that you cannot reason your way to this. You have to measure it downstream, on real users, on the metric you actually care about, and be willing to be surprised.
Not these other Owls
This is Owl AI at withowl.ai. Not owl.co (insurance), owl-ai.com (courses), aiowl.org, the OwlAIProject repo, or CAMEL-AI OWL (the open-source multi-agent framework).