Predicting the outcome of an election is one of the hardest things a commercial survey company is likely to do. It’s hard to predict people’s future behavior, and with election polling there’s always a definite outcome at the end. Failure is public and obvious. In the last election, Gallup clearly lost some of its mystique by predicting Romney would win.
IVR (aka robocall) surveys are some of the most challenging surveys to get right. Customers hate them, it’s hard to get participation, self-selection bias is high, and in some states they’re illegal. Pretty much the only reason to use an IVR survey it it’s much cheaper than a live interview.
Despite these challenges, IVR surveys have had a pretty good track record predicting elections–about as good as live interview surveys, at any rate.
But some new research suggests that the companies running IVR surveys may be fudging their numbers in political polling to make them look better (via Kevin Drum). A statistical analysis of 2012 primary election survey shows that IVR surveys had about the same error rate as human interviews–but only if there was also a live-interview survey conducted on the same race before the IVR survey. If there was no live-interview survey, the IVR surveys had much larger errors in predicting the election outcome.
The obvious interpretation of this is that the people publishing the IVR surveys were fudging their numbers to better match the previously published live-interview surveys. Or, since nobody wants to stick their neck out, they might have simply chosen to not publish surveys which were too far out of line with what other surveys were showing.
Another possible interpretation is that the contests where there was no human polling were generally not interesting enough to bother fielding large surveys–in which case a larger error would be expected.
I am not a fan of IVR surveys in general, and I would expect that they would have higher error rates than human interviewers–the more surprising result to me would be if IVR surveys were truly just as accurate as live surveys. Nevertheless, someone has some explaining to do.
