We stopped measuring engagement and our product got better
For the first year of building Murror, we optimized for the same metrics every other app optimizes for: daily active users, session length, screens per visit. The dashboard looked healthy. Usage was growing. We felt good about it.
But something was off. Our most engaged users were not our happiest users. People who spent the most time in the app were often the ones who left the harshest feedback. Meanwhile, users who opened the app twice a week for five minutes were writing us emails about how it changed how they handle difficult conversations.
We were measuring activity when we should have been measuring impact.
So we ran an experiment. For one quarter, we replaced our engagement metrics with what we called "outcome metrics." Instead of tracking how long someone stayed, we tracked whether they reported feeling more clarity after a session. Instead of measuring return frequency, we measured whether people said they applied something from Murror in a real life situation.
The results were counterintuitive. Some of our most "engaging" features scored terribly on outcomes. A beautiful interactive visualization that users loved to play with was not actually helping them understand anything about themselves. And a simple, almost boring two-question prompt that most people finished in under a minute was producing the highest outcome scores we had ever seen.
We started making product decisions based on outcomes instead of engagement. We removed three features that quarter. We simplified two screens. Our session length dropped. Our DAU dipped slightly. And our NPS went from 34 to 61.
The hardest part was trusting the process. Every instinct from years of building products told us that declining engagement metrics meant something was wrong. But we had to keep reminding ourselves: the goal is not to keep people in the app. The goal is to help them understand themselves better and take that understanding into the real world.
We are still early in this shift. We do not have it all figured out. But I genuinely believe that the next generation of AI products, especially ones dealing with something as personal as emotions and self-awareness, will need to rethink what success looks like. Not every product should optimize for time spent.
Curious if anyone else has experimented with outcome-based metrics instead of engagement. What did you measure, and how did it change your product decisions?
Replies
Really appreciate this framing, Mona. We're seeing the exact same tension in marketing automation. I'm building AI marketing agents for solo founders (ad-vertly) and the temptation is to optimize for volume: more posts published, more ads running, more campaigns active. But that's just the marketing version of DAU. The metric that actually matters is whether the founder got a qualified lead, closed a deal, or learned something about their market. We've started building our agent feedback loops around outcomes (did this campaign actually move revenue?) rather than activity (did we post 4x this week?). It completely changes which channels you prioritize and what content you create. Your NPS jump from 34 to 61 after removing features is such a powerful signal. Less noise, more signal. Applies to both product and marketing.
Murror
@gaurav_singh91 Love that you're applying the same thinking to marketing automation. "Did this campaign actually move revenue?" vs. "Did we post 4x this week?" is such a clear parallel to what we went through on the product side. The temptation to optimize for visible output over real impact seems universal. Really cool to see ad-vertly building feedback loops around actual founder outcomes — that's the kind of tool I'd want as a founder.
This is a great reminder that engagement is only useful when it correlates with user success. I’ve seen products where time spent looked healthy, but it was really a proxy for friction or unresolved intent. Curious what metric became your best leading signal after you stopped optimizing for session depth.
Murror
@luca_ardito You nailed it — time spent can absolutely be a proxy for friction rather than value. Our best leading signal became the 3-day follow-up question: "Did you apply something from your last session in a real-life situation?" Users who answered yes had dramatically higher long-term retention and referral rates. It turned out that real-world application was the strongest predictor of both satisfaction and organic growth. Much more reliable than any in-session metric.
This hit hard. We did the exact same thing six months ago and it felt terrifying at first. Swapped session time and DAU for “did this actually move the needle on their emotional state?” and “did they use it in a real conversation this week?” Killed two of our prettiest features that people played with forever but got zero real value from. Session length tanked, but the messages we started getting went from “cool app” to “this changed how I show up with my team.” NPS jumped 28 points.
Murror
@second_son_of_god This is so validating to hear, Adrin. The shift from "cool app" to "this changed how I show up with my team" is exactly the kind of feedback that makes all the discomfort worth it. A 28-point NPS jump is huge — sounds like you're seeing the same pattern we did. The scary part is always the initial dip, but once you see the qualitative shift in how users talk about your product, there's no going back.
any reason you spent so long (a quarter) testing this?
Murror
@daniel7789 Good question! A quarter felt like the minimum to get reliable signal. Outcome metrics are inherently noisier than engagement metrics — they depend on user mood, context, and willingness to respond. The first few weeks were especially messy. We needed at least 6-8 weeks before patterns started emerging, and then a few more weeks to validate that the patterns held. Shorter experiments would have tempted us to abandon the approach before the data had time to tell its story.
This is one of the most useful product lessons because engagement can be a very comfortable metric to hide behind.
If the product promise is clarity, confidence, or better decisions, then measuring time spent can easily reward the wrong behavior.
Murror
@luca_ardito Completely agree. When the product promise is about clarity and better decisions, engagement metrics can actively mislead you. Someone spending 20 minutes might be struggling, not thriving. We had to learn that the hard way before the shift clicked for us.
This is one of the more important posts I've read on PH in a while.
We ran into a version of this at ad-vertly too. Early on, we were tracking "campaigns launched" as our primary success metric — it looked great, numbers going up. But campaigns launched is an input metric, not an outcome one. When we switched to tracking "marketing tasks founders stopped doing manually," we got very different signal about what was actually working.
The distinction you're drawing between activity and impact is especially hard in SaaS because investors, benchmarks, and your own instincts all push you toward engagement. Switching to outcome metrics takes real conviction because they often look worse in the short term.
To answer your question: what changed for us is that we started asking users "did this do something useful for your business this week?" instead of looking at logins. The answers were humbling but way more useful for roadmap decisions.
Murror
@gaurav_singh91 Really appreciate you sharing the ad-vertly example — that shift from "campaigns launched" to "marketing tasks founders stopped doing manually" is such a clear illustration of the difference between activity and impact. You're right that investors and benchmarks push you toward engagement, and it takes conviction to move away from metrics that look good on a dashboard. For us, the moment that made it click was realizing our happiest users weren't our most active ones. Once we saw that, we couldn't unsee it. Would love to hear more about how that shift changed your roadmap priorities at ad-vertly.
Hello Aria
This is a really important observation. Engagement metrics often optimize for stickiness over actual value — they measure how often people come back, not whether the product is making their life better.
We have been thinking about this a lot with Hello Aria. It is an AI assistant that runs inside WhatsApp/Telegram and manages your day. The temptation is to track DAU, session length, message count. But the metric that actually matters is: did someone accomplish what they set out to do that day? That is much harder to measure but infinitely more meaningful.
When you stopped tracking engagement, what did you start tracking instead?
Murror
@sai_tharun_kakirala Really appreciate this perspective, Sai — and Hello Aria sounds like you're dealing with the exact same tension. To answer your question: we replaced session length and DAU with three things. First, a post-session clarity score ("do you feel more clarity about what you were reflecting on?"). Second, a 3-day follow-up asking if they applied something from the session in real life. Third, we started tracking referral behavior as a proxy for genuine value — people don't recommend things that didn't actually help them. The "did someone accomplish what they set out to do" framing you described is spot on. It's harder to measure, but it's the only metric that actually tells you if your product is working. Would love to compare notes sometime!