Metrics, and Getting Them Right


Share on LinkedIn

In one of my sessions at SpeechTEK a couple weeks ago we got involved in a lively discussion of metrics for IVR and speech systems.

I’ve been arguing for a long time that “containment” is the wrong metric for measuring the performance of an IVR system. Not only does containment put the emphasis on blocking customers rather than serving them, it doesn’t really measure the cost of serving a customer either. Customers who are “contained” without being served often call right back, driving up costs and frustration. My recommendation is to use a “self-service” metric instead, counting success only if the IVR actually gave the customer what she needed on the call.

A couple members of the audience took issue with this and argued in favor of containment instead. Their reasons (if I can take the liberty of summarizing someone else’s statements) are:

  1. The purpose of an IVR is to reduce cost by handling calls without sending them to an agent. Containment is a direct measure of this, and therefore is the right metric.
  2. Containment is easy to measure, and self-service is hard because it requires surveying customers to find out if they got what they wanted.
  3. Containment is so well-established in some organizations as the key metric of IVR performance that change is impossible.

In my view, these arguments are not persuasive. In fact, they are emblematic of all the reasons why containment is such a bad metric and needs to be abandoned:

  1. The purpose of a customer service operation in general is to provide cost-effective customer service. The IVR is one piece of this, and serves multiple roles including offering self-service options and routing calls to the correct agent or queue. Focusing on containment works to the detriment of the larger purpose of the entire customer service operation.
  2. Ease of measurement isn’t a good reason to use a bad metric. At best, it’s a reason to choose one good metric over another similarly good metric, but you should not use a bad metric that’s easy to measure instead of a better metric which is harder to measure (tempting as it may be). Containment is a bad metric.
  3. Change can be difficult, but that doesn’t mean it isn’t worthwhile. I suspect that most organizations would actually have very little problem abandoning containment if there was a champion for change internally.

The problem with bad metrics like containment is that because they don’t map to an organization’s true priorities, they can cause people to do stupid things in order to make the numbers look good.

I know nothing about IVR or VUI design, but I can build a system which will achieve 100% containment guaranteed. Just send customers in an endless maze of prompts and reprompts. (The fact that this experience sounds familiar suggests that I’m not the only one to think of this strategy.) But few would argue that this design serves anybody: not the customer, not the company, and it doesn’t save any money in the end.

Worst of all, though, is that containment doesn’t even measure what it purports to measure: cost savings of an IVR. At best it measures the cost of a very narrow sliver of a customer’s overall experience (i.e. the agent cost of a single call to customer service), but it doesn’t capture extra costs elsewhere in the form of customers having to make multiple calls, customers being forced into more expensive service channels like bricks-and-mortar, and lost business.

So can we please put Call Containment to bed?

If your IVR or speech system is intended to provide self-service, you should measure self-service.

If your IVR or speech system is intended to route calls, you should measure routing accuracy.

If your IVR or speech system does both, then you should measure both.

And no matter what your IVR or speech system is supposed to do, you should be looking at it in the context of the customer’s overall experience from start to finish.

Republished with author's permission from original post.

Peter Leppik
Peter U. Leppik is president and CEO of Vocalabs. He founded Vocal Laboratories Inc. in 2001 to apply scientific principles of data collection and analysis to the problem of improving customer service. Leppik has led efforts to measure, compare and publish customer service quality through third party, independent research. At Vocalabs, Leppik has assembled a team of professionals with deep expertise in survey methodology, data communications and data visualization to provide clients with best-in-class tools for improving customer service through real-time customer feedback.


Please use comments to add value to the discussion. Maximum one link to an educational blog post or article. We will NOT PUBLISH brief comments like "good post," comments that mainly promote links, or comments with links to companies, products, or services.

Please enter your comment!
Please enter your name here