Premise:

We live in a world (now more than ever) filled with data. Collecting data, all kinds has become fashionable. What this has also led to is the proliferation of KPI’s.

We measure all kinds of things and it has slowly but steadily become common practice to move to pay-for-performance models in every sphere of life including businesses, policing, healthcare, education, non-profits, governments, etc.

The assumption is that if the KPI’s are being achieved and over a period of time are trending in the right direction (usually up and to the right corner), everything is good.

In reality the assumption is far from real. When people are paid for performance and the way if the performance is measured via metrics or KPI’s, that someone far removed from reality uses to dole out rewards or punishment, people tend to find creative ways to game the system.

In the short term it appears that everything is good because the goals measured by the KPI’s are met and they are trending in the right direction (up and to the top right of any graph), but in the mid-to-long term, they have an adverse impact on the organisation.

We all have known this intuitively for long. We have many examples of this being true – Enron, Wells Fargo, Volkswagen, spiraling cost of healthcare or education, policing (where bigger crimes are reported as lesser crimes because the frontline policing staff want to ensure that they are able to reduce the crime rates (as their jobs might depend on it).

We also see this happening even in our houses. If you have teenage kids, you will know that they can be extremely creative in achieving the goals that you set for them, without actually doing what you want them to do.

So, what can we do about it? We still need metrics to measure how we are performing so that we can continue to monitor & improve performance. And yet, we don’t want to fall in the trap of measuring either the wrong things or pushing people to game the metrics.

I recently read a book titled – “The Tyranny of Metrics” by Jerry Z. Mueller. In the book he elegantly explains all the different ways that metrics fixation has negatively impacted organisations and the society in large and how metrics fixation can create chaos.

I have written about KPI’s or metrics here, here and here. I knew that KPI’s need to be defined with a lot of intention and in partnership with the people who will be held responsible and I also knew about gaming of KPI’s in business contexts but was not aware of the prevalence of gaming of KPI’s to be so prevalent across all kinds of organizations.

Data vs Judgement:

There are two ways to measure any particular performance – via data or through judgement of someone who knows the job or work being measured.

In the recent past, as it has become easier to collect data, the preference in all kinds of settings has been moving towards collecting a lot of data and as being the primary way to measure performance.

What we need to understand is that measured data is almost never an alternative to judgement. In fact, measurement demands judgement – judgement about whether to measure, what to measure, how to evaluate the significance of what is being measured and decide if rewards and penalties need be attached to the results. Lastly, judgement about who should have access to the measured data.

Guidelines on measurements:

At the end of the book, Jerry does share a checklist on when and how to create performance metrics and what we need to keep in mind while analyzing these metrics.

Below are some guidelines based on his checklist added with my perspectives.

What do you plan to measure:

The kind of information you plan to measure has an important impact on how you design your metrics. The more the object to be measured resembles inanimate matter, the more likely it is measurable (natural sciences or engineering).

When the object being measured are influenced by the process of measurement, then the measurement becomes less reliable. Even more unreliable if the measurement is about human activity.

People are conscious and will respond to the process of being measured and if rewards and punishments are involved, they will react in a way to skew the validity of the measurements.

What will the measures be used for:

Metrics used by practitioners to track performance to compare themselves with their peers and identify potential areas of improving performances are more useful.

Metrics that will be used by external parties (includes senior management, government or any other external party) who might not recognize the limitations of the metrics and use it to dish out rewards or punishments leads to behavior that will lead to gaming the system and are more harmful in the mid-to-long term.

Offering recognition to those who excel and assistance to those who are behind is good.

KPI’s that help accentuate the internal motivations of the people being measured so that they can continue to improve tend to be better than if they are designed to get people to behave in ways that they are not intrinsically motivated or appeal to extrinsic motivations.

Low stakes metrics are almost always better than high stakes metrics.

If the performance being measured does not require or has the potential for any intrinsic rewards, pay for performance might work really well.

How useful is the measurement:

Just because something is measurable doesn’t mean that it is worth measuring. In some cases, the ease of measurement could be inversely proportional to significance of what is being measured.

Ask yourself – “Is what you are measuring a proxy for what you really want to know?”. If the answer is no for both (usefulness or a proxy for what you really want to know”, you are better off not measuring it at all.

How useful are more metrics

Measured performance, when useful are more effective in finding outliers, especially poor performers or true misconduct. They might not be very useful in distinguishing people in the middle or on the top of the ladder.

The more you measure, the marginal cost of measuring will become more than the marginal improvements through measurement. So, the fact that metrics are helpful doesn’t mean that more metrics are more helpful.

In fact, it is easier, simpler and faster to use less metrics, but one’s that is relevant to drive behavior and continue to improve performance. One can change the KPI depending on the maturity of performance instead of having the same KPI to measure the performance of everyone (irrespective of how mature the person or process doing the activity is).

Costs of Measurement:

There is always a cost of acquiring the data for the metrics. Every moment that we devote to production of metrics, it is time not spent on doing what is being measured. So, before we initiate a new metric, we need to identify the costs of measuring.

We can also look if there are other sources of information about performance that can be used instead, based on the judgement and experience of customers or someone else.

Who develops the KPI’s and what they mean:

How and by whom are the measures of performance developed. Measurements are more likely to be beneficial if developed bottom up with inputs from the people who will be measured (experience and judgement).

KPI’s defined by people who don’t have the context in which the performance being measured typically tend to be the one’s that are gamed or manipulated as they might neither add value nor increase the difficulty of performing the activity.

In conclusion:

KPI’s are effective to the extent that the people who are being measured believe in its worth and effectiveness. We need to remember that even the best KPI’s are subject to gaming and corruption. Keep an eye of the result of what is being measured.

Sometimes, recognizing the limits of the possible is the beginning of wisdom. Not all problems are soluble and even fewer are soluble by metrics. It’s not true that everything can be improved by measurement and whatever can be measured can be improved.

We need to exercise judgement, which comes from experience, before introducing any new metrics. Metrics should inform judgement and vice versa.

As Jerry concludes in his book, sometimes the best use of metrics is not to use them at all.