Stop vanity, choose product metrics that matter

“No totals, only actionable numbers” is now my daily mantra when I open dashboards and start doing analytics. With it in mind, one day I found myself relentlessly kicking out all the charts which showed ‘bad’ metrics for the product I was working on.

What do I mean by ‘bad’ metrics? They are so called ‘vanity metrics’, which look so good and make you feel happy and proud, but actually mean nothing. Tracking those metrics is almost as tracking nothing at all.

How did I figure them out?

Let’s start with what our dashboard looked like when we first set up analytics, with our bad metrics, for product overall and for each feature:

1. Total users
2. New users
3. MAU/WAU/DAU
4. Number of transactions
5. Different button clicks and screen views

The whole team could see all of these big numbers growing even bigger all of the time, which was super exciting at first.

Let’s say we had 10K of active users and 3K of new users each month. Looking at these numbers for some time, I started asking “Now what? Do we know what actions we took in the past that drove those users and converted them, and do we know which actions to take next?”. Then I realized these metrics weren’t really helpful. We still had no understanding what to focus on, because it seemed like everything was already perfect.

The thing is, these total numbers will always go up, especially if you’re doing some acquisition efforts. Everyone loves their vanity dashboards and it might feel tempting to keep them. Don’t. They can misguide decision-making, which you probably wouldn’t want happening. A good dashboard puts failure or success up in big letters on the wall shows whether your product growth is on track or needs adjustment.

After a lot of iteration on our key product metrics, I’ve worked out 2 key things that help define which metrics are ‘good’:

1. They have context
2. They tell you what to focus on

And applying these principles help to transform vanity metrics to actionable ones:

1. Don’t use totals, measure rates and relative numbers instead (funnels, conversions, shares, etc.)
2. Don’t look just at numbers of current periods, look on the historical dynamic of metrics to compare growth over time. For example, we have activation rate of 30% this month. Okay, is it good or bad? Bad — if all the previous months you had 50%. Good — if earlier you had 20%.
3. Use breakdowns: by marketing channels, platforms, types, or any other parameters — to find behavioral patters and impacts.
4. Look at user segments/cohorts separately. For example, instead of looking at the total number of active users in a given month, consider looking at the % of new and returning users in this activity.
5. Look at the context. Don’t measure intermediate metrics like the click-through rates of the buttons. Care about the user behaviours that lead to something useful, whether purchase, retention, or some other measurable “success” particular to your product.
6. Review regularly. No matter how hard you fight with vanity metrics, they still may appear sometimes. Review your dashboards regularly and adjust them if you feel they no longer help you understand what to focus on.

The new and improved dashboard

So, what I’ve got after applying the above principles:

1. Total users → Number of users in different user segments
2. New users → Activation rate (% of new users who did some meaningful action in the product) and its growth rate compared to previous periods
3. MAU/WAU/DAU → Daily/weekly/monthly retention rates by acquisition cohorts
4. Number of transactions → Number and frequency of transactions per users
5. % of users adopted (used) a feature, with breakdown by share of new and returning users

This is a whole different picture now.

Measure what matters.

Always happy to connect and chat: email, linkedin, twitter