I’ve been reading tons of posts in the last few years about outputs and outcomes. Most of them have very similar messages. There’s the consistent mantra of “outcomes over outputs” and even talk of “outcome-based roadmaps”. But what are these things (outputs and outcomes)? How can I get leverage by using both? Is this “model” complete”? In this post, I’ll try to get down to basics and provide a framework for reasoning over these concepts.
Metric Result Types: Outputs, Outcomes, and Impact
I consider outputs and outcomes to be “metric result types”. Metrics, simply put, are units of measure like revenue, churn, and active daily users. They are the foundation of goals, which are general things we’d like to achieve, and objectives, which are things we'd like to achieve that are time-bound and measurable. Goals and objectives represent “ends” we would like to achieve. We can classify these ends based on the associated metrics result type.
Outputs
Outputs are the product of our work. In a product context, one of the most important outputs we measure is features. The existence of outputs can usually be easily verified, by counting for example. We might wish to translate our mobile app into 5 non-English languages. We can invest in these translations and actually verify that the translation has been done by simply counting the translations.
Monitoring this type of result (an output) is valuable — it can give us an indication that progress is being made and can help us make decisions about the work as it's in progress, e.g., how close are we to completing it. However, this output-oriented metric (number of translations) lacks some critical information: are these outputs generating desirable business outcomes like increased revenue, less churn, or more satisfied customers. That’s why virtually all output-oriented metrics should have one too many corresponding outcome-oriented metrics, i.e., metrics that directly or indirectly indicate a positive effect on our business.
Outcomes
Outcomes are important because they demonstrate that associated outputs are actually making a difference in our business. The risk they address is delivering a feature (an endeavor that requires enormous effort) but not realizing a reasonable return on our investment. For example, if we provide the capability for users to visualize their data in our product but no one uses it, we haven’t moved the needle on revenue, customer satisfaction, our bottom line, and potentially other metrics. That means we’ve made an investment but it isn’t helping our business! As a matter of fact, this type of misfire often has a negative effect on the listed metrics and overall business performance.
You’ll find many voices in product management circles claiming that we should focus on outcomes and not outputs. As previously stated, my position is more nuanced: you should be careful of having output-oriented metrics that are NOT tied to corresponding outcomes. Both result types are valuable.
Impact
The third return type is impact, which is a desirable effect on customer outcomes. That means that customers feel or can prove they are getting value from the product. In a B2B context, this value may come in the form of more revenue, greater employee productivity, or even greater peace of mind.
Impact is different from the other two types because we, as the vendor creating the product, usually don’t have direct access to the data required to quantify it: we need to ask our customers about the product’s impact, which can be difficult and time-consuming. Assessing impact is, however, critical. Defining goals and objectives based on impact-oriented metrics early in the development process is important because we will likely need to do some planning regarding how we’ll collect the data.
I have seen at least one alternate definition for “outcome”: impact on customers’ behavior. I’m not a fan of it because it overlooks the criticality of tying the inputs we generate to our business performance. It most closely approximates the idea of impact I’ve shared in this post but lacks the emphasis on customer satisfaction/business impact.
Outcome-based Roadmaps
You may come across some folks talking about replacing traditional market value-oriented product roadmaps with "outcome-based roadmaps". The latter shows what ends you would like to achieve in the future. While reasoning over our goals and objectives is important, because these artifacts don't communicate the delivery of market/customer value, I don't consider them roadmaps. I'll elaborate on these thoughts in a separate post.
Conclusion
There is a lot of discussion in product circles about outputs vs. outcomes. Unfortunately, there's a lack of alignment on the definition of these terms and little discussion of the relationship between them. Measuing outputs, the things we deliver (like features) is valuable but incomplete: we should also be measuring the business impact of these outputs, i.e., the outcomes they generate. Ultimately, we should measure our product's impact, which is the effect our product has on our customers' goals and objectives.