Do You Mangle What You Measure? Eight Pitfalls to Avoid

“You can’t manage what you can’t measure.”

Grrrrrr! It sounds authoritative. Catchy, too. I like it!

I wanted to learn who originated this hallowed maxim, and my search led to none other than W. Edwards Deming, the quality guru. He must have known what he was talking about. Thanks to Deming’s pioneering work in statistical process control, the seals on my car doors don’t leak when it rains, and when I put the vehicle in drive, it goes forward, and not into reverse.

But it turns out that Ed never made that statement. In fact, his original thought got garbled over the years. What Deming did say was “The most important figures that one needs for management are unknown or unknowable . . . but successful management must nevertheless take account of them.” That’s too long to Tweet, and it looks clunky on a PowerPoint slide. No wonder his verbose sentence mutated into those seven presentation-friendly words.

Deming’s circumspect voice has spawned a font of finger-wagging spinoffs:

• “You treasure what you measure.”
• “You can expect what you inspect.”
• “You can’t control it, unless you measure it or model it.”

Measuring and metrics are smoking hot right now, with nearly 63 million search results when I last checked a few seconds ago – more than double the number I got when I entered crime prevention. And for good reasons:

• uncertainty and risk confront us every day
• humans love explanations
• we have boundless capabilities for keeping track of stuff

Not to mention, we have pragmatic business needs. As Deb Calvert wrote in a recent blog, “With the emphasis on results, plus the accolades and rewards you’ve received for producing results, you may be singularly focused on the numbers, the volumes, the productivity, and the bottom line.” But even wonky Deming cautioned that one of the seven deadly diseases of management is running a company on visible figures alone. Danger Will Robinson! You might mangle what you measure!

Steer clear of these pitfalls:

1) Measuring what’s not meaningful.
Number of outbound calls made. Number of demos given. Quantity of Facebook likes. Satisfaction indices. For managing revenue risk, do these measurements matter? For some companies, maybe. But for others, no. A problem that’s compounded if employee compensation is based on measurements for things unrelated to delivering value. At one company I worked with, the district manager announced to the sales force that he was going to track Windshield Time Ratio – or WTR. He described this metric as monthly revenue divided by total miles driven. The salesperson who covered suburban Philadelphia had a more efficient ratio than the rep who covered the entire state of Nebraska. Much more. But so what?

2) Succumbing to the flaw of averages.
“The Pollyanna way of forecasting the future is to take averages from the past and project them forward,” Steve Parrish wrote in Forbes (October 3, 2012). Adding, “It is not necessarily wrong to use averages in making financial decisions, but it is dangerous to rely on this measuring tool alone. Computers are powerful tools; let’s put them to work. Why not look at various assumptions and scenarios to get a feeling for possible outcomes?”

3) Putting too much credence in “what the numbers say.”
Otherwise known as Dashboard Love. “Monitor individual departments, multiple websites and anything else using dashboards,” one website proclaims. Anything else? Impressive, if it was possible.

4) Not questioning the accuracy of the measurements. Or their origin.
“66% of the buying process is complete by the time a salesperson gets involved.” (I’ve also read 60%, and 67%). “Two thirds of all marketing content is never used by salespeople.” A separate article squawks, “70% of sales enablement content is never used.” Hmmmmm. We bandy these numbers about like they are de facto standards, without defining terms, or clarifying the context when these measurements were allegedly made.

In current public policy debates, such as racial profiling and law enforcement, we hew to metrics that support our points of view. But, as The Wall Street Journal reported in December, 2014, “It is nearly impossible to determine how many people are killed by the police each year . . . Three sources of information about deaths caused by police – the FBI numbers, figures from the Centers for Disease Control and data at the Bureau of Justice Statistics – differ from one another widely in any given year or state.”

5) Injecting biases. Confirmation bias, anchoring, bandwagon effect.
Biases are always present when using metrics for decision making. According to Bob Hayes, (The Hidden Bias in Customer Metrics), “generally speaking, better decisions will be made when the interpretation of results matches reality” – an outcome no one should ever take for granted. He illustrated with an example: “we saw that a mean of 8.5 really indicates that 64% of the customers are very satisfied (% of 9 and 10 ratings); yet, the [Customer Experience] professionals think that only 45% of the customers are very satisfied, painting a vastly different picture of how they interpret the data.”

6) Observer effect.
The measurement perturbs the results. This idea goes back to 1927 when physicist Werner Heisenberg developed his uncertainty principle, in which he proved that the way scientists observe objects changes the way the objects behave. The same occurs in business development. For example, the reason that opt-in and opt-out online surveys yield different results. With opt-in, respondents could be motivated to receive a reward. That could create a fundamental misperception of reality, a point that James Mathewson made in an article, How to Measure Digital Marketing with Observer Effects. He wrote, “we have to accept that digital marketing management is not all science. It’s almost as much art as science. If we embrace the art, we can get better information on our users, and ultimately serve them better without violating their privacy.”

7) Compiling faulty indices.
Many indices are flawed and easy to manipulate, The Economist reported in an article, How to Lie with Indices (November 8, 2014). Their sardonic expose reveals some commonly used tricks. “Above all, remember that you can choose what to put in your index – so you define the problem and dictate the solution.”

8) Embedding unneeded complexity.
Seth Goldman, Co-founder of Honest Tea (now part of Coca Cola), wrote in an article, Way too Many Metrics, “At Honest Tea, we’ve pondered many different metrics when trying to quantify the impact of our mission driven business, including:

• The reduced calories for each bottle, can or drink pouch we sell
• The increase in organic acreage fueled by our expanded demand, which helps support a less chemical-intensive approach to agriculture
• The community investment dollars that we are able to generate with our Fair Trade premiums, such as support for schools, ambulances or eye care for villagers in a tea garden
• The influencer/ripple effects of our success, when we create pressure for competitors to expand their low/no-calorie or organic options

But instead we keep it simple – we evaluate the impact of our mission by counting the number of bottles we sell.” At the time he wrote the article, he knew the number: 930,601,802 bottles from inception.

There’s unlimited room for mangling what’s measured. The faulty insight that yields the misdirected strategy or tactic. The hubris that results after statistical output is anointed as having predictive validity. “The threat is that we will let ourselves be mindlessly bound by the output of our analyses even when we have reasonable grounds for suspecting something is amiss,” wrote Viktor Mayer-Schonberger and Kenneth Cukier in their 2013 book, Big Data. Nicholas Carr, author of a recent book, The Glass Cage, gives similar caution: ” . . . templates and forumulas are necessarily reductive and can too easily become straightjackets of the mind.”