When AI Gets Product Decisions Wrong - Who Notices First?
by•
We’re starting to rely on AI more and more in product decisions.
But here’s something we’ve been thinking about:
When AI is wrong about your product - who notices first?
The PM? The engineer? The user?
Or worse - no one?
As we build Athena, we keep asking ourselves how a system can stay grounded in reality, not just generate convincing answers.
Curious how teams think about this - where should the “source of truth” live in an AI-driven workflow?
34 views
Replies
The scariest AI failure mode isn't hallucination. It's confident drift where the system is technically answering questions, just slowly drifting away from what's actually true about your product.
In my experience, during my test launch the user notices first. They just don't always have the language to say "the AI was wrong", they say "this didn't help" and churn quietly.
On source of truth is i don't think it lives in one place. It's a triangle user behavior (what people actually do), product intent (what you meant to build), and AI output (what the system says). When those three stop agreeing, something's off. The PM's job becomes calibration, not just prioritization.
No clean answer yet. But asking the question out loud, like you're doing, is already ahead of most teams.
Athena
@merit_mohammed "Confident drift" is the perfect name for it. And you're right - the user notices first, they just vote with their feet instead of their words.
The triangle is a great mental model. We're trying to build exactly that calibration layer into Athena - making the gaps between those three visible before they become churn.