Not everyone is going to immediately spot the problems I saw with this online survey I got from Discover Card this week. But some people will, especially those who have some knowledge of User Interface design.
When designing this survey someone thought it would be cute to have the selection buttons shade from red to green. I have no idea where this idea came from, but it seems like the sort of thing that might come from a graphic designer and get put in place without bothering to consult anyone who understands user interface or survey design.
The first problem that jumped out at me when I saw this survey is that the grey in the middle for “Neutral” makes it look like the “Neutral” button is disabled (in the screen shot, “Neutral” is selected, which is why its circle is filled in rather than open). It’s become a standard part of user interface design to indicate that a control is disabled by greying it out, so at first glance some users might think that Neutral isn’t actually an allowed option on this survey.
That’s something that could affect the outcome of the survey at the margins. Does it? I have no idea–and I’m guessing that Discover Card didn’t calibrate the survey to see if their color choices make a difference. But it’s certainly plausible, which is one reason survey design is hard. So many things can affect the results of a survey that you need to be careful to either understand the design choices, or make sure that the analysis and decision-making process is robust in the face of subtle survey biases.
There was another problem I immediately spotted with this survey, one which most people won’t notice but which 7%-10% of the male population will immediately see (or, in this case, not see). The particular shades of red and green used in this survey are ones which are hard to distinguish for people with the most common form of color blindness. So for me, and a significant minority of the population, whatever Discover meant to communicate through the colors of the buttons is completely lost because we can’t easily tell the difference. Since there are color palettes out there designed to be accessible to colorblind people, and this is another important detail that good User Interface designers know to watch out for, I am again led to the conclusion that Discover didn’t really think about their color scheme very carefully before dropping it into their survey.
(Those with normal color vision will probably be surprised to read that before I wrote this article, I actually used Photoshop to verify that the colors in the image really are red and green. It would be embarrassing to get such an important detail wrong.)
I sometimes like to say that survey design isn’t hard, what’s hard is dealing with the people who think it’s easy. It’s not hard to design a reasonable survey, but there are a number of details which can change the outcome. But because designing a survey looks easy, often people will want to make changes without thinking through the implications. This survey is a great example of a seemingly-trivial design choice which might actually impact the data, and which clearly isn’t necessary.
