Sunday, April 5, 2009

How to effectively measure product management

One of the most difficult issues that organizations deal with is how to measure the success of Product Management (and Product Marketing), especially in the short term.

The financial system forces companies to focus on short term quarterly results, and we are all used to the matching company quarterly business review. Our VP wants metrics to track the effectiveness of Product Management from quarter to quarter - just like manufacturing and support. Easy? No. Let's discuss...

Product Management isn't a discipline that can be measured scientifically - the inputs don't necessarily correlate to the outputs (results).  Writing code is part art, part science.  Deciding candidate features is a complex business planning process involving a huge number of dimensions.  As Alan Cooper mentioned last week, would the outcome for Google have been any different if it cost $50K, $500K or $5M to get to market - the answer is a resounding NO.  They would still be the powerhouse that they are today. 
So what we end up with is Product Management being tracked against a project schedule e.g. was the PRD submitted on time, did the product ship on time, was the SE training completed on time. The problem is that this type of measurement is not only completely invalid, it's actually counter productive. Defining the entire project dates up front is a waterfall technique, and as we move to agile it clearly doesn't make sense anyway.

The only measurement that matters is did the product release meet the established market objectives. These could be:
  • Revenue from new customers
  • Additional revenue from existing customers
  • Increased Market share
  • Increased user satisfaction
  • Decrease support
  • More successful evaluations
  • Getting to market before a competitor
Track and measure success against these market objectives, no matter what the time frame. Don't complain that you are dependent on other groups like sales & marketing - your role is to ensure that their plans line up against the release goals you defined. Product Management is the "product CEO" and should be held responsible for the ultimate success or failure.  Yes, this may need to be over many quarters, but it gets the team focused on the long term strategy.

Throughout my career, we've measured the most successful projects based on revenue increase - mostly new projects, so revenue was what we were after - nobody particularly cared about how closely we hit or missed project dates, as long as we kept the exec team posted on any slippage on the release data (and didn't cross a financial quarter, which has revenue recognition implications). Sure we had quarterly MBOs, but these were always qualitative, but measurable and made sure we were focusing on the right things.

When an organization focuses completely on measuring achievement against a project schedule, the team will modify their behavior to hit the dates. I have seen features dropped and products released that are complete crap, just so the team  can hit the date and get their bonus. So the product team stands up and praises how wonderful they are, even if the product dies in the marketplace. NetWare 4 is a great example - I can remember many engineers and managers who were proud that they "got it out the door", but never mind that it was extremely buggy, and NDS was too large a leap for many NetWare 3 users (this is the subject of a future post). Would Microsoft regard Windows Vista as a success in the marketplace?

This isn't an excuse to be loose and not care about dates - dates matter, especially if you have a specific target market window to beat a competitor, hit a financial period etc.  But what really matters is that your product is successful in the marketplace, and you measure and reward Product Management to hit this goal.

1 comment:

  1. When I saw your post a few days back I thought this has to be a hot issue! I led a panel discussion at P-Camp Silicon Valley on this topic last month. It was a crowded room with a lot of issues and questions, but not a lot of answers yet. Senior execs are cutting budgets. Today, conserving cash and reducing risk are considered strategies by many tech companies.
    So how does a VP of Product Management thrive or even survive without knowing what his/her team's specific contribution is to the most important corporate targets?
    In the panel discussion we talked about metrics with a dual edge. Most of these metrics have one thing in common; they are all LAG indicators. They can work for us in good times, but they can be hard for product management to impact in difficult times.
    Some of the best techniques I have seen include aligning the product management practice with a single high priority operational goal. It is easier to break down the goal and track it. There is also a reduced risk of driving behavior among product managers may confuse the purpose of the role. The one you have I really like is "Getting to market before a competitor" it is a great example of a single high priority operational metric that drives the right behavior. To succeed, product managers must get regular feedback from the total market and conduct on-going competitive strategy. These are valuable things.