How did the election forecasts get it so wrong?

The Upshot, FiveThirtyEight, Predictwise, etc: their predictions for President varied over the campaign as you'd expect as new data came in, but consistently made Clinton a solid favorite, with a probability of a win topping 70% the day before election day. So what went wrong? As in any statistical forecast, there are three possibilities:

Source: New York Times

• The models were wrong. No model is perfect, but it seemed to me at least that the various forecasts, despite their differing methodologies, all captured the essential mechanisms of being elected President: the electoral college; the similar behaviours of some states; the influence of economic and demographic statistics; the relationship between polls and votes. Clearly, something was missed, but these models have been good enough before, and it's not clear why they weren't this time.
• The models were right, and this is a fluke. Even a 95% probability isn't a guaranteed outcome. It's entirely possible that the electorate behaved in exactly the way the models described, just at the most extreme Trump-favouring end of the predicted spectrum. Looking at the "residuals" of the model should give us some clue, like the county-by-country swings from the 2012 election shown above. To me, though, that looks like something systematic was missed in the model.
• The data were bad. The US election process, because of its haphazard nature and inconsistent processes across the country, makes it unlikely that the actual election results were incorrect. That leaves the data going into the models. I see no reason why the economic and demographic data shouldn't be considered solid. That leaves the polling data: why, this 2016 election, did the responses to polls translate into votes in a fundamentally different way than in previous elections? These models corrected for "house effects" and weighted for polling methodologies, so they shouldn't have been caught out unless data from prior elections are not a useful guide for this election. One possibility that comes to my mind is that the "feedback effect" -- the influence on voters from the poll results and projections themselves -- behaved unexpectedly this year, thanks to the power of personality of the candidates, and the increased influence of the network effects of social media.

These possibilities will be considered in depth online in the weeks and months ahead and in political science theses for years to come, I'm sure. No-one knows the answers yet. 

But this is a sad day for the profession of political forecasting.

Comments

[Dalila]

Sometimes we are so blinded with our own belief that we cannot phantom something else. Hence, we follow a script blindly and miss the tectonic shifts, and miss new evidences that don't follow our script. This explains perfectly why the financial market couldn't predict the housing and financial meltdown in 2008 (see Big Short). In short, we develop complex beautiful useless models.

[Marypcbuk]

More than usual, I think we need to look at how much people disguised their intent in polling, and how good the sampling was - in an age when people being at home to answer the landline might be an outlier.

[Travis]

Was the model wrong or the data not accurate because people did not respond truthfully to the survey?

[Anne York]

Michael Moore got it right some time ago with a simple conceptual model based on electoral votes from 2012 and the focus of Trump working for votes in the US Rust Belt.

http://michaelmoore.com/trumpwillwin/

In spite of that, I think there were problems with the polling methodology. EG, did respondents answer differently to female pollsters etc etc

[Dr. Mitchell R. White]

How is this a sad day for political forecasting? If polling and predictions always work, then there would be little new to learn. With this "miss" (not a failure, if you're statistically inclined) there will be much learning. Some of it will apply to the mechanics of polling, predicting and so on. Much of it may help us in other arenas, including macroeconomics and demographics (where the data are considered "solid;" I'm not sure about that hypothesis today), political activism and more.

[PremThed]

Null hypothesis: Deviation from model predictions are unrelated to the type of voting machine (and the type of internet connectivity) at the polling location, and also with the precinct's likelihood of changing electoral outcome.

This seems a bit out there, but might be worth getting a p-value for the above.

[Tim Niiler]

As some of the post election numbers come in, we see that in some key places there were fewer blue voters than there were in 2012. For example, in Wisconsin, a tally I saw yesterday indicated that in 2012, there were 1,620,985 democratic votes vs 1,407,966 republican votes. In 2016 the numbers were 1,377,588 democratic (-243,397) and 1,404,376 republican (-3,590). While this could be the result of voter suppression, my thinking is that the models so convinced everybody of a Clinton win, that some people on the blue side didn't make it a priority to go out and vote. And yes, there's the possibility that some democratic voters stayed home because they didn't like either candidate. Perhaps there is such a thing as creating such an atmosphere of overconfidence that you change the results your models predict?

[JonReade]

I think MaryPCBUK identified the problem, and it's down to basic human psychology more than data or methodology. During the Brexit campaign here in the UK, people were reluctant to say they were voting to leave the EU, for fear of being labelled as a racist, because the immigration issue was such a highly discussed and emotive part of the entire campaign. So my guess is that the same happened in the USA, people would either not discuss how they would vote, or conceal it by saying they would vote for Clinton because they were afraid of being tarred. The same effect was seen in Germany after WWII - no one would admit to being a member of the Nazi party, even though millions were.

[William England]

Great map and visualization, David! The Rust Belt shift comes through in stark relief. I am wondering if random survey samples drawn at the national level could have watered down the enormous swing that was coming in the upper Midwest (which proved decisive in the outcome), and if that might partly explain why there was systematic error in so many forecasts?

[James]

“If you torture the data long enough, it will confess.”

― Ronald H. Coase, Essays on Economics and Economists

[David Smith]

Thanks William, but credit for the map goes to the NYT, not me. Pretty sure 538 at least does include national surveys in their predictors, but the bottom line is that with such little data it's basically impossible to predict a regional effect like that that hasn't been seen before, unless it can be explained by demographics/economics. Remember, there had only been 57 presidential elections before 2017. That's 57 data points, and only the most recent few are even relevant. Not much to make a forecast from.

[Robert]

Looking back at the UK 2015 General Election, a significant error in polling then was that voters easy to contact had different preferences when compared to voters hard to contact (found by "British Election Study" post vote). Quick, cheap polls were biased towards voters who are easy to contact, and I haven't seen anything from this US election that suggests pollsters tried to avoid this potential issue.