The financial system forces companies to focus on short term quarterly results, and we are all used to the matching company quarterly business review. Our VP wants metrics to track the effectiveness of Product Management from quarter to quarter - just like manufacturing and support. Easy? No. Let's discuss...
Product Management isn't a discipline that can be measured scientifically - the inputs don't necessarily correlate to the outputs (results). Writing code is part art, part science. Deciding candidate features is a complex business planning process involving a huge number of dimensions. As Alan Cooper mentioned last week, would the outcome for Google have been any different if it cost $50K, $500K or $5M to get to market - the answer is a resounding NO. They would still be the powerhouse that they are today.
So what we end up with is Product Management being tracked against a project schedule e.g. was the PRD submitted on time, did the product ship on time, was the SE training completed on time. The problem is that this type of measurement is not only completely invalid, it's actually counter productive. Defining the entire project dates up front is a waterfall technique, and as we move to agile it clearly doesn't make sense anyway.
The only measurement that matters is did the product release meet the established market objectives. These could be:
- Revenue from new customers
- Additional revenue from existing customers
- Increased Market share
- Increased user satisfaction
- Decrease support
- More successful evaluations
- Getting to market before a competitor
Track and measure success against these market objectives, no matter what the time frame. Don't complain that you are dependent on other groups like sales & marketing - your role is to ensure that their plans line up against the release goals you defined. Product Management is the "product CEO" and should be held responsible for the ultimate success or failure. Yes, this may need to be over many quarters, but it gets the team focused on the long term strategy.
Throughout my career, we've measured the most successful projects based on revenue increase - mostly new projects, so revenue was what we were after - nobody particularly cared about how closely we hit or missed project dates, as long as we kept the exec team posted on any slippage on the release data (and didn't cross a financial quarter, which has revenue recognition implications). Sure we had quarterly MBOs, but these were always qualitative, but measurable and made sure we were focusing on the right things.
When an organization focuses completely on measuring achievement against a project schedule, the team will modify their behavior to hit the dates. I have seen features dropped and products released that are complete crap, just so the team can hit the date and get their bonus. So the product team stands up and praises how wonderful they are, even if the product dies in the marketplace. NetWare 4 is a great example - I can remember many engineers and managers who were proud that they "got it out the door", but never mind that it was extremely buggy, and NDS was too large a leap for many NetWare 3 users (this is the subject of a future post). Would Microsoft regard Windows Vista as a success in the marketplace?
This isn't an excuse to be loose and not care about dates - dates matter, especially if you have a specific target market window to beat a competitor, hit a financial period etc. But what really matters is that your product is successful in the marketplace, and you measure and reward Product Management to hit this goal.