Saturday, February 6, 2010

How to Measure Success

In Australia over the Christmas break I was asked by the CEO of a startup company how to tell if he had the right product strategy. Some companies believe (falsely) that you can analyze success into a product by doing greater and greater amounts of market research and analysis.

The true answer is extremely straightforward - you gain multiple paying customers.

We live in a capitalist economic system, and money is our store of value.  Therefore if a customer buys your product, they are clearly demonstrating that you are delivering them value at least equal to the purchase price. Selling the same product multiple times without customization proves you have built something repeatable.

All other ways of measuring product success, whether it be market share, market recognition, analyst awards etc, are irrelevant if you fail to sell the product. One large software company in particular sets goals around market share, but it does this because as a certain % of market share translates to a certain % of market revenue.

Therefore, any company that wishes to minimize product risk, needs to get to market rapidly, with a minimal viable product (MVP), and gain paying customers during the development process.  Finding 2-3 customer willing to commit to buy your product and engage during development dramatically reduces product risk.  If you can't get multiple customers to commit to buying your product before you start developing, this may be a signal that you are building the wrong thing.

As an example, while at Citrix we developed the Citrix Secure Gateway (CSG), that later morphed into the Access Gateway and quickly became number #1 in the SSL VPN market, and over $100M revenue. CSG was built by a small Citrix skunkworks team in only seven months, and very early in the process Lehman Brothers engaged as our development partner.  They actively helped during the development process by reviewing our specs, meeting with the team, detailing their requirements, and committing to using the product.

Compare this to a project at a different company, where after two years of defining requirements, development, testing, marketing, sales and SE training, the product didn't sell - it was essentially DOA.  There were a few main differences:
  • No customer(s) who had committed to buying the product on release.
  • No focus on delivering the minimum viable product (MVP).  Requirements blew out as "must haves" kept on being added.
  • No development customer partners.
We've all worked with development teams who get motivated by solving cool or difficult problems, and view financial goals as non-motivating.  As a Product person, you need to explain carefully to these people that the financial goal is simply a measurement of the success in meeting customer needs.  To meet the financial goal we still need them to do their magic and deliver great products.


  1. I'd offer one more angle: Customers who spend time using a product are also making an investment and establishing "value" for a product.

    Gmail has 176M non-paying users. Facebook exceeded 300M users in 2009 and studies have shown many users are spending 4-5 hours per week on the site. Clearly both have value. Both are indirectly monetizing through ads. Also, Facebook seems to have an inherent valuation of $14B (based on shares trading privately on "SecondMarket" at $32 each).

    With enterprise products, early adopters who are highly interactive with the product teams in shaping the release version see value. Otherwise, these over-worked individuals would invest their time elsewhere.

    Different measures of success exist depending on the stage of the company or product. Revenue is always the ultimate goal but it may not be a near-term milestone. Many measures of success can exist leading up to profitability. The trick is to pick metrics that show product acceptance and adoption are increasing.

  2. Great post Tim, you raise some very thoughtful questions. There used to be an accepted VC business model of going for subscribers first, and then working out monetization (Google and Facebook being notables), however this is very much out of vogue on Sandhill road.

    The way that companies like Facebook and Google are proving value is from creating advertising revenue. There are thousands of startups following in their path, and I would argue that advertising makes it very quick and easy to get development "customers". If your initial release is being used and gaining subscribers, and making advertising revenue, then you have early validation. The more customers, the more advertising revenue, and therefore, more validation of value.

    Interestingly, it could be argued that the real customer is the advertising service, and by providing customers you are delivering them value, from which a financial transaction ensues.

    Gmail is an interesting use case - without a doubt it creates huge value for users (including me), but AFAIK it doesn't create revenue for Google. Google Apps (their paid offering) still has a long road to climb to challenge Microsoft in the Enterprise. So it will be very interesting to see how this plays out for Google.

  3. I tend to agree with the comments here, but I think there's still a valuable aspect to market research that's being overlooked. Market research helps you prioritize the limitless range of opportunities and projects that you could theoretically pursue.

    In large technology companies, there are literally dozens of joint development opportunities to pursue sponsored by a surprising number of willing customers ready to help fund the effort. I believe that market research is critical to ascertain whether you're building a one-off for a tiny population of nagging customer or truly developing something that's strategic and going to be valued by a broad swath of the market.

    I'm familar with Phil's Secure Gateway case study and I also know that there were some pretty clever technologists and product managers who recognized the strategic value of what Lehman Brothers was asking for before Citrix even pursued CSG development. Somebody realized and articulated that the concept of a CSG was valuable to more than just the single development partner/customer. That's the power of market research!

    I'm still amazed at the willingness of so many enterprises to sink large sums of money into custom development efforts with their technology vendors. Yet, these efforts often yield deliverables with little-to-no strategic, let alone reuse, value for the vendor. I can't even begin to count the times when technology vendors that I've worked for have brought in large customers as development partners, yet we ended up producing a one-off deliverable with no value beyond the one customer's highly archaic requirements.

    Of course, if you can overcome the aforementioned issues, then there's nothing better than having a constructive, vocal, paying customer as a partner in the development process. If empowered, then they often feel like they have a stake in the product's long-term success.

  4. I think part of the art of Product Management is engaging a "keystone" customer but also making sure the product doesn't turn into a "custom" just for that customer. How to manage the interaction between base concept (from the company), keystone customer input and requirements that will allow the product to be successful at multiple customers is the key. I agree however that not having a keystone customer is real trouble!