Are IVR Polls Cheating?


Share on LinkedIn

Predicting the outcome of an election is one of the hardest things a commercial survey company is likely to do. It’s hard to predict people’s future behavior, and with election polling there’s always a definite outcome at the end. Failure is public and obvious. In the last election, Gallup clearly lost some of its mystique by predicting Romney would win.

IVR (aka robocall) surveys are some of the most challenging surveys to get right. Customers hate them, it’s hard to get participation, self-selection bias is high, and in some states they’re illegal. Pretty much the only reason to use an IVR survey it it’s much cheaper than a live interview.

Despite these challenges, IVR surveys have had a pretty good track record predicting elections–about as good as live interview surveys, at any rate.

But some new research suggests that the companies running IVR surveys may be fudging their numbers in political polling to make them look better (via Kevin Drum). A statistical analysis of 2012 primary election survey shows that IVR surveys had about the same error rate as human interviews–but only if there was also a live-interview survey conducted on the same race before the IVR survey. If there was no live-interview survey, the IVR surveys had much larger errors in predicting the election outcome.

The obvious interpretation of this is that the people publishing the IVR surveys were fudging their numbers to better match the previously published live-interview surveys. Or, since nobody wants to stick their neck out, they might have simply chosen to not publish surveys which were too far out of line with what other surveys were showing.

Another possible interpretation is that the contests where there was no human polling were generally not interesting enough to bother fielding large surveys–in which case a larger error would be expected.

I am not a fan of IVR surveys in general, and I would expect that they would have higher error rates than human interviewers–the more surprising result to me would be if IVR surveys were truly just as accurate as live surveys. Nevertheless, someone has some explaining to do.

Republished with author's permission from original post.

Peter Leppik
Peter U. Leppik is president and CEO of Vocalabs. He founded Vocal Laboratories Inc. in 2001 to apply scientific principles of data collection and analysis to the problem of improving customer service. Leppik has led efforts to measure, compare and publish customer service quality through third party, independent research. At Vocalabs, Leppik has assembled a team of professionals with deep expertise in survey methodology, data communications and data visualization to provide clients with best-in-class tools for improving customer service through real-time customer feedback.


Please use comments to add value to the discussion. Maximum one link to an educational blog post or article. We will NOT PUBLISH brief comments like "good post," comments that mainly promote links, or comments with links to companies, products, or services.

Please enter your comment!
Please enter your name here