The Product Metric – a Product tale

Introduction

Metrics that help you understand user engagement or the resonance of a new feature, to high level metrics that indicate the health of the overall business

Tracking the performance of a product, be it digital or physical, tells a lot about the product, the business and the health of it. From a Digital Product Manager lens, we believe it tells us a deeper story than what meets the eye. From specific metrics that help you understand user engagement or the resonance of a new feature, to high level metrics that indicate the health of the overall business.

In this post, we’ll cover some key product metrics across a handful of categories and then dwell into industry specific ones:

• NPS – Net promoter score using the question ‘How likely are you to recommend [your product] to a friend? 0-10’. Promoters (9-10) and detractors (0-6). Subtract the detractors from the promoters as a %. Common mistake: Your NPS is not a % itself; it’s a whole number based upon the calculation of the 2 percentages. The NPS is generally seen as an overall indicator of how happy your customers are with your product. Revisit it on a monthly or quarterly basis. When a user gives you a low score ask why. Use the feedback to improve your product.
• Revenue-The amount of revenue generated by your product directly or by your overall business. Measure the direct impact of your product decisions on the bottom line. More difficult in larger organisations but easy in startups
• Actions– actions that are specific to your product, over a set time period Your product will probably allow users to perform an action of sort. Your action will be linked to the inherent value your product is offering to its audience. For example, if your product allows customers to book cinema tickets, number of cinema tickets booked would be an action. If it allows merchants to print labels for ecommerce orders, your action would be labels printed.
• Magic metric / NORTH STAR– A metric that matters most to your product goals. A magic metric which, through your own analysis, has proven to have an impact on some of your most important product goals, such as increasing revenue, sales or NPS. Your magic metric will be unique to your product. It may take some experimentation to identify what yours is. For example, a music product may find that after 3 song previews are played the user is more likely to buy a track and recommend the service to a friend. A content publisher may find after 10 articles are viewed a month for 2 months, the user is more likely to buy a subscription. A language learning product may discover that after 3 live classes with a teacher, the student is more likely to stay for 6 months but after 0 live classes they want to cancel their subscription. Experiment to identify your magic metric and craft your product to help users experience the magic moment. Re-evaluate your magic metric every quarter to see if the correlation is still true. As your product evolves your magic metric may lose its magic over time.
• Monthly recurring revenue (MRR) – The amount of revenue generated per month. Every business should be tracking monthly revenue. Splice it by all sorts of dimensions (numbers of staff, package) to uncover the financial health of your business.
• Churn – The % of customers who stop using your product (typically per month). Critical for monthly subscription products and an overall indicator of product health.
• Cost per acquisition – how much it costs to acquire a customer. The lower the CPA, the better. Typically compared against the customer life time value (LTV)
• Virality / Referrals – The % of customers who refer a friend. One way to grow your business is through referrals. Measure how many people are  sharing your product and telling their near and dear ones be it family, friends, followers or even colleagues
• Life time value (LTV) – the value of a customer of their lifetime. A baseline metric that all consumer facing products should measure to understand how much value is derived from a customer

For Mobile Apps

To track metrics for a Mobile App some key factors to consider the following

• Downloads – The number of downloads of your app.
• Activations– The number of people who have ‘activated’ your app where activate is defined by you. Activation can be defined as different things for different products.
• Uninstalls– The number of people who have uninstalled your app. Monitor uninstallations to understand how many users are waving goodbye each month
• Daily active users– Number of users who are using your product on a daily basis. Keep on top of your most important numbers.
• Monthly active users – Number of users who are using your product on a monthly. Keep on top of your most important numbers
• Session length– The length of a session where session is defined as time between app open and app close. Gives you a sense of how engaged your users are. Generally, the higher the session length, the better. However, an app which anticipates a user’s need and ensures a speedy transaction would mean less time spent with the app but a happier user who is more likely to return. An example could be a flight booking app. Your goal could be to reduce session time and increase transactions at the same time.
• Session interval- the time gap between the last app open and the next app open. Gives you an insight into how ‘sticky’ your app is
• Ratings– Your app’s ratings in the app store (typically a star rating). Less stars reduces the likelihood of your app being downloaded.
• In-app purchases– The number of in-app purchases made. Measure purchases and revenues to project the current and future financial projects.

For Ecommerce platform

When the product is an Ecommerce platform selling physical goods some meaningful metrics to track are

• Orders – The number of orders sold in a given time period. Running an ecommerce business is all about the orders, baby.
• Average order value – The average monetary value of 1 order (what the user has in their basket). Gives you a snapshot of order values which can be used in projections and updating your stakeholders / investors. Enables you to assess and project revenue based on visitor numbers
• Shopping cart abandonment– Number of users who have filled a cart with items but haven’t reached the checkout. It’s fairly self explanatory. If people abandon their baskets, you’re losing revenue. You need to a) try and figure out why and b) put together some experiment ideas for increasing it.
• Checkout abandonment rate – Number of users who have filled a cart, reached the checkout but not completed the purchase Is there something confusing about your checkout process? If customers are putting stuff in their basket and dropping off before checking out, what’s occurring? Measuring this helps you to run experiments to address this.
• Product searches– The number of times a user searched for a product in a set time period. Measure how engaged users are e.g. if you’re selling train tickets, a search is a vital part of the funnel. Discover what users are searching for.
• Average shipping costs– How much are customers paying on average on shipping? Allows you to factor in delivery into pricing models. What is the point at which delivery becomes prohibitively expensive so as to prevent sales? Can delivery costs be used as a lever for driving conversion i.e. offering free delivery over a specific threshold?
• Repeat customers – The % of customers who are repeat customers where repeat customers means they have placed at least 1 order in the past. Single purchase customers are not good news for your life time value or your cost per acquisition. Allows you to measure the health of your LTV and CPAs.
• Revenue – Total revenue for set time period. Without revenue you don’t have an e-commerce business.
• Visits – A unique visitor to your site. Self explanatory baseline metric
• Visit to sale conversion %– The % of all visitors who make a purchase. The visit to sale conversion gives you a healthy overall conversion % benchmark to play with. If you work on a large e-commerce site this will best be split into categories, else it doesn’t make much sense to provide an overall average if you sell thousands of products.
• Shipping time– The time it takes to deliver an item from order date to delivery date (excl. bank holidays / weekends). Shipping is increasingly important to ecommerce companies. Efficient shipping drives conversion.
• Cost per acquisition (CPA)– How much it costs to acquire a customer. If your average order value is $50 but you spend $80 to acquire a customer, things don’t look pretty. But bear in mind the LTV; if the LTV is higher than the CPA, that makes things healthier. In e-commerce profitability is tough since you’re selling physical goods – either through a vertically integrated model or wholesale model – each with their own margins.
• Customer life time value – The value of a customer of their lifetime. Compare your LTV against your CPA for a true test of profitability per customer.
• Mailing list effectiveness– The effectiveness of your mailing list, as defined by you. Measure how effective your email marketing list is in driving sales and return visits to your site. Open rate, click through rate and sales conversion may make up the funnel of your mailing list effectiveness
• Virality – The number of referrals. Monitor how referrals are helping your product to grow

For API’s

When the product is API’s the metrics to tangibly track performance are

• API calls (requests / responses) – total number of API calls in a given time period. Prove that your API is being used. You could call this a vanity metric, but as your usage increases, you’re continually proving to your team, your stakeholders (and potentially your investors) that a) your product is actually being used by real customers and b) that your API is scaling effectively (if the numbers are large enough).
• Failure rate – number of failed API calls. Prove to stakeholders you’re on top of your game. If you track it, you can measure it. If you measure it, you can improve it.What happens if you don’t monitor this? You don’t know how well your API products are performing. Failed API calls might mean missed revenue or dropping your service levels below an agreed rate, resulting in the loss of revenues / clients.
• Actions per month – your API will perform an action e.g. sending a text message, booking a hotel room, playing a music track. Total number of these actions for your product. Measuring API calls is helpful but defining explicit actions is better. APIs can perform meaningful actions, which result in revenue or engagement. Demonstrate to your stakeholders the value of your by explicitly showing the actions performed by the API.
• Revenue – revenues as a direct result of your API product. Actions can be linked to revenue. Put a $ figure against a set of API calls / actions and suddenly the business will perk up its ears.
• Support tickets – number of support tickets raised as a result of API problems. A sense check from your customer service teams to ensure support tickets are manageable. If the number is high, dig into them to find out what’s going on and why.
• Uptime – total uptime for a particular API. Demonstrate that your API is working at scale
• Downtime – total downtime for a specific API – Identify problems and diagnose bottlenecks in your API by measuring down time at specific periods.
• Client retention – month on month retention of external API customers Retention is an indicator of how useful / how much impact on your API is having on your customers business.
• Customers – number of external customers using a particular API Customers = revenues = growth = success = happy faces. An external API with zero customers is a sad thing indeed.

For Content site

When the product is a Content site providing internally created content or through external creators some metrics to track are

• Advertising click through rate (CTR)-The click through rate of an advertisement running on your content site. Measure how effective advertising formats are across your content site.
• Advertising cost per thousand (CPM)– The cost of an advert running on your content site, sold to the client. CPM = cost per thousand page impressions where M = roman numeral for a thousand. Get a deeper understanding of the business metrics / economics behind your content site if it’s advertising led.
• New vs. returning visits– % split between new visitors and returning visitors. Are visitors coming back to your site? Do you care? Repeat visitors means spending less to acquire new visits and demonstrates long term interest / engagement in your product.
• Bounce rate %– % of visitors who land on your page, click on nothing and leave. Not always helpful and often misunderstood, but particularly useful for measuring the performance of specific landing pages. Looking at a site wide bounce rate isn’t much help.
• Pages viewed per user– Number of pages viewed by 1 user. If your content site is selling ads against impressions, you’ll want impressions to be high – and the average number of pages viewed by a user to be high.
• % of users who share content– % of users who share at least 1 piece of content in a given time period. Content doesn’t have to be shared. If your growth plan is built upon viral growth, this is a key metric, if it isn’t, it’s not as important. Content sites should focus on building content for their users and not dupe them into sharing stuff for the sake of it.
• Most viewed content– Your most viewed piece of content over a specific time period Some content sites can get a major % of their traffic from 1 post alone. Segmenting your traffic by posts allows you to see what your most viewed post / content is. You could make product decisions based upon this. For example, if you know 80% of your visits from Australia are interested in 1 post, you could promote an upcoming Austrlia event on that page.
• % of total visitors who sign up for an email newsletter. Overall visit to email sign up can be an indicator of a) how relevant your content is to your audience and b) how effective your newsletter / email offering is. Running experiments will help you to monitor this metric.
• Revenue per visit– Amount of revenue generated per visitor. Helps to formulate revenue projections for your business plans.

For SaaS products

For SaaS products (parts of subscription service covered here) some meaningful metrics are as follows

• Cost per acquisition – Cost for acquiring a new paying subscriber. It’s a beautiful thing to acquire customers with zero marketing costs. Rarely does it happen – so it’s best to keep track of it. Your CPA should be below your LTV to factor in other costs. Some suggest around 1/3 of your LTV
• Leads –The number of leads where leads is defined by you / your business. Monitor the people interested in your product. Leads may be: people who have downloaded an e-book, an email subscriber, someone who has requested more information or something else. Agree to a definition of a lead and monitor the number to measure your performance.
• Visit to lead % – The % of your visitors who become leads. Gives you a flavour of how well your marketing channels are targeted to specific audiences. Helps you measure the potency of your sign up forms / process
• Visit to subscription – the % of your visitors who become subscribers/. Get a sense of how willing your audience is to pay for your product. Can be monitored and fed back to your marketing teams or analysed on a campaign specific level.
• Lifetime value of a subscriber – The amount of revenue 1 subscriber brings to the business. LTV gives you a sense of long term health of your product and how engaged your customers are over period of time. LTV is linked to your CPA. If you’re acquiring customers at a cost of $500 and your LTV is $400, your microeconomics are not working. You’ll need to work to increase LTV and decrease CPA at the same time.
• Subscription package breakdown – Identify which packages are more popular. Explore bundling opportunities by combining 2 or more packages together.
• Churn rate – The % of customers who leave your product / cancel their subscription e.g. in month 1 you have 100 subscribers and in month 2 you have 95 of those subscribers remaining. Your churn rate is 5%. Your retention is the opposite of churn. In this case it would be 95%. Measure the long term health of your subscription product. Measure happiness of your customers. Calculate customer life time value. Identify problems which may have contributed to churn
• Actions per month – Number of actions per month (where action is relevant for your product). You can define what your actions are based on the function of your product. Users will derive value from your subscription product which will typically be based upon a set of actions. Examples could include: printing labels, attending classes, listening to songs, viewing movies, downloading books. Monitoring actions allows you to monitor engagement and build rich sets of data about your users.
• % Activations – the number of people who have ‘activated’ their subscription where ‘activate’ is defined by your product. In subscription products, activate could mean many things. If you offer a trial of your product, activate may mean ‘to become a paying customer’.
• % Active users – the number of people actively using your product where you define what ‘active’ means. Measure the engagement of your subscribers. Pre-empt cancellations by tracking lapsed users. Build mechanisms to increase engagement
• Monthly recurring revenue – the amount of revenue your product generates on a monthly basis. Monthly revenue is critical for all products and businesses, but since the subscription / SAAS business model is based upon paying customers, your monthly revenue is particularly pertinent in this case.

Though there are many listed but these are just the tip of the iceberg. There are many nuances when we want to track specific actions on features and functionality. Having these as L1 and L2 we can have plethora of finer metrics telling us specifically how certain things are doing.

Final Thoughts

The above list provides a good starting point for products to track on the onset and dig further to get to improving the product overall.