In one of my sessions at SpeechTEK a couple weeks ago we got involved in a lively discussion of metrics for IVR and speech systems.
I’ve been arguing for a long time that “containment” is the wrong metric for measuring the performance of an IVR system. Not only does containment put the emphasis on blocking customers rather than serving them, it doesn’t really measure the cost of serving a customer either. Customers who are “contained” without being served often call right back, driving up costs and frustration. My recommendation is to use a “self-service” metric instead, counting success only if the IVR actually gave the customer what she needed on the call.
A couple members of the audience took issue with this and argued in favor of containment instead. Their reasons (if I can take the liberty of summarizing someone else’s statements) are:
- The purpose of an IVR is to reduce cost by handling calls without sending them to an agent. Containment is a direct measure of this, and therefore is the right metric.
- Containment is easy to measure, and self-service is hard because it requires surveying customers to find out if they got what they wanted.
- Containment is so well-established in some organizations as the key metric of IVR performance that change is impossible.
In my view, these arguments are not persuasive. In fact, they are emblematic of all the reasons why containment is such a bad metric and needs to be abandoned:
- The purpose of a customer service operation in general is to provide cost-effective customer service. The IVR is one piece of this, and serves multiple roles including offering self-service options and routing calls to the correct agent or queue. Focusing on containment works to the detriment of the larger purpose of the entire customer service operation.
- Ease of measurement isn’t a good reason to use a bad metric. At best, it’s a reason to choose one good metric over another similarly good metric, but you should not use a bad metric that’s easy to measure instead of a better metric which is harder to measure (tempting as it may be). Containment is a bad metric.
- Change can be difficult, but that doesn’t mean it isn’t worthwhile. I suspect that most organizations would actually have very little problem abandoning containment if there was a champion for change internally.
The problem with bad metrics like containment is that because they don’t map to an organization’s true priorities, they can cause people to do stupid things in order to make the numbers look good.
I know nothing about IVR or VUI design, but I can build a system which will achieve 100% containment guaranteed. Just send customers in an endless maze of prompts and reprompts. (The fact that this experience sounds familiar suggests that I’m not the only one to think of this strategy.) But few would argue that this design serves anybody: not the customer, not the company, and it doesn’t save any money in the end.
Worst of all, though, is that containment doesn’t even measure what it purports to measure: cost savings of an IVR. At best it measures the cost of a very narrow sliver of a customer’s overall experience (i.e. the agent cost of a single call to customer service), but it doesn’t capture extra costs elsewhere in the form of customers having to make multiple calls, customers being forced into more expensive service channels like bricks-and-mortar, and lost business.
So can we please put Call Containment to bed?
If your IVR or speech system is intended to provide self-service, you should measure self-service.
If your IVR or speech system is intended to route calls, you should measure routing accuracy.
If your IVR or speech system does both, then you should measure both.
And no matter what your IVR or speech system is supposed to do, you should be looking at it in the context of the customer’s overall experience from start to finish.
