Free response questions (aka open-ended questions, aka verbatims) are a powerful tool for finding out what’s top-of-mind with customers. The traditional method for dealing with them is to have an intern perform the mind-numbing task of reading each comment and categorizing it, in order to do some statistical analysis. This is what we did for the recent NCSS report on customer complaints, though I and one of my managers played the roles of interns.
In the past I’ve played around with some text analysis software to try to reduce some of the burden of this chore, and found that it didn’t help much for the size of data set we’re working with (usually under 10,000 free responses). It took a little too much time and effort to train the software to give good results. On the other hand, if we had hundreds of thousands (or even millions) of comments I could definitely see the value.
For surveys in an interview format it makes sense to have the inteviewer provide a preliminary categorization of the customer’s comment. The interviewer won’t know as much about the company’s internal business processes as an insider, but will probably get it right 90% of the time. The interviewer can also flag ambiguous cases for further review.
There’s also a limit to what you can get from a free response question. When you ask for comments, most customers will give you just the one thing most important to them at that moment. So this is not a very good way to track the actual incidence of specific problems, since you miss customers who may have experienced the issue but didn’t happen to mention it.
So when something interesting develops in the comments, it’s a good idea to start asking about it specifically in the survey. You will get a much better read on whether there’s a problem with (for example) language skills or the IVR if you ask every customer whether they experienced that issue or not. If it turns out you don’t have a problem then the question can be removed.
