Is It Time to End the Software Metric Madness?

We have all heard the phrase “What gets measured, gets managed,” often attributed to Peter Drucker. (Interesting note about this quote, Drucker didn’t say it.) In the DevOps space, or application development and delivery generally, this quote is often used as a justification to measure things like deployment velocity, bug fixes, lead time, and so on, as a way of gauging success.

One can find countless articles with titles such as “X Key Metrics to Track DevOps Success.” Many of these use a version of this quote to justify measuring attributes that are all essentially related to the speed and/or quality of work.

Deployment velocity, lead time, bug fixes, and the like, are effective ways to show that an organization is producing software rapidly, and with a degree of quality. But, are these really the best ways to measure success?

In almost any other industry, the answer would be “no.” These traditional software delivery metrics are a measure of work progress. They do not account for whether that work progress results in meaningful value, or business success.

Take an automobile manufacturer, for example. If Acme Cars can produce hundreds of thousands cars annually (speed), and they all drive down the road perfectly (quality), would they be a successful business?

What if no one wants to buy all those cars? GM was able to produce many thousands of Pontiac Aztecs, but they were condemned by journalists and consumers (Walter White notwithstanding) as the ugliest car ever created, and demand was far short of projections.

What if the cars sold, but they all sold for a loss? In the case of the Aztec, Pontiac needed to sell 30,000 annually to break even, but they never came close to that mark. Many today consider the Aztec as being at least partially responsible for the demise of the Pontiac brand.

Measuring the speed and quality of production is essential but not sufficient. Yet, for the most part, that’s what software organizations are tracking.

This could be why Forrester Research wrote that “CEOs must end the madness of software metrics” in its report “CEOs: Foster the Software Teams and Talents Your Firm Needs.” Forrester goes on to say that “arcane, indirect measures of success and ROI…confound conventional management techniques.”

Instead, there needs to be a correlation to business outcomes, such as profit margin, ROI, and customer acquisition, according to Forrester. Highly successful organizations are already doing this. Others need to take steps to ensure that “everyone in software delivery gauge success with business measures.”

This same philosophy is what’s driving increased interest in the concept of BizOps. BizOps is a framework for optimizing software development and delivery to foster the achievement of business objectives. Essentially, BizOps helps connect the dots between traditional measures of software progress and metrics of business success. It’s what is needed to end the software metrics madness.