This is Part 18/18 in the series “How to Build an Innovative New Product or Company” on the topic of how to measure the value and ROI that your product is having for your buyers and users
Consider a product that helps prevent patient falls in a hospital. It is bed alarm that alerts the nurse when a patient is shifting their weight so the nurse can assist the patient who may be trying to stand up, thus reducing the risk that a fall will occur.
Further, imagine that, before implementing ‘Fall Safe’, a particular hospital had 5% of patients fall during their hospital stays. Then that hospital implemented ‘Fall Safe’ in 2015 and falls went down in many of the departments. Over time, falls across the hospitals went down to 2.5% and held steady.
How do you take credit for holding the fall rate at 2.5%? If your product wasn’t in use, wouldn’t the rate go back up to 5%? What if the hospital claims they’ve done other things since 2015 that contribute to the 2.5% level? What if the hospital is set to renew their subscription with you and wants to know what improvement you’ll make beyond the current 2.5%? In other words, their expectation might be that 2.5% is now the baseline and your product will need to move beyond that to be considered a good investment.
What if the severity of patients increased during the past year and their fall rate went up to 7.5%, but you believe it would have gone much higher had it not been for your product?
If you want to impact an outcome metric, you need to communicate that outcome metric to the buyers and users of your product. Ideally, you’d compare it to a target. In this example, how many falls has the hospital avoided this year? What’s the cost avoided from doing so? For each department, how are they doing? Are they meeting their target? Can the product itself communicate this or is it better to have a monthly email or meeting where this is shared and explained?
Create an index that your customers will need to rely on you to provide, that they will refer to often, and that the use of your product is the primary mechanism they will use to improve it
You may have to create your own metric that your buyers believe is a leading indicator of the actual outcome metric; this is especially useful if the real outcome metric is rarely measured or conflated with noisy events.
Customer acquisition via marketing is a good example. Companies spend a lot of money on marketing efforts (e.g., visiting tradeshows or running magazine ads) but struggle to know whether current efforts are putting them on a path towards the future sales they aspire to. So marketing firms have created proxy metrics, such as the open rate of email advertisements that they send. It works great as marketing team members can focus on increasing it every day. They know that if they do that will, it will improve total sales, which is the ultimate outcome, but may not respond on a daily basis to their efforts.
Proxy metrics should (a) be easy to measure, (b) be concrete and stable, (c) correlate to / predictive of the main metric, (d) show you how to hone your actions and efforts in the near-term, (e) adjust as you take those actions, (f) be a clear measurement, not a “black box”, and (g) ideally be on a scale of 0-100 where 100 is best. Net Promoter Score (NPS) is a good example; it has been shown to predict lifetime customer value and renewal rates. Can you create a proxy metric that becomes the de facto standard for your industry?
Can you aspire to continuous improvement? Should you?
Continuous improvement, such as the Toyota Production System or airline safety reporting system enjoy, is a noble goal for a product. If you’re ambitious, you don’t just want your users to try your product; you want them to use it, see an improvement in their goals, then use it more, and over time see it revolutionize a major part of their lives or careers for the better. This can’t happen from a point solution; rather, only from a cultural change.
If you aspire to continuously improve a metric, I think you then need to consider how your product fits into the larger ecosystem. You may also need a larger client care team to make sure that culture and behaviors are being built and reinforced around the product.
What if your product improves outcomes … but other forces at work muddy up the outcome metric so it’s hard to tell for sure?
One thing we do at my current company is to split our user base into two or three categories at each client, based on usage: superstars vs. on-track vs. lower users. We then look at the changes in the outcomes metrics for each of the three cohorts every few months. If the overall outcome metric is improving for the highest users, is stagnant for the middle users, and is deteriorating for the lowest users, you can easily see the impact that your product is having, assuming all other forces are affecting the three groups equally.
What if your product improves outcomes … but your users revert that improvement by using the improvement as an excuse to relax other efforts?
Malcolm Gladwell shares the insight that more pedestrians are killed crossing the street at crosswalks than they are at other locations. Overlooking the mathematics of it (e.g., part of the answer is that many more people try to cross at crosswalks), part of the reason is that the crosswalks provide a false sense of security. The crosswalks reduce risk, but that risk reduction is then lost due to more careless actions by the pedestrians.
The crosswalk is a “product” and it is only half of a pedestrian safety solution. It needs public awareness and other supporting systems. For example, statistics on a signs posted at each crosswalk warning of the number of pedestrians killed at crosswalks might help reinforce the need for continued vigilance.
In conclusion…
You have a great product, you have happy users, and you’re making an impact on the world. The final major step is to document that impact and prove it to the world. Focus on the outcome metric that you want to improve. Make sure that your product and customer success team collectively can own as much of the influences on that metric as possible. Work back from that larger outcome metric to create proxy metrics that provide proof of real-time progress. Find ways to show that your highest users are making the most progress. Most of all, keep it simple: you may spend all of your life thinking about the details of these outcomes, but the impact has to be clear and unequivocal to any skeptical stakeholder taking a cursory glance.