How Nate Silver won the election with Data Science

While I'm pleased with the outcome of yesterday's election in political terms, I'm equally thrilled about the victory for Data Science versus "gut instinct", "philosophy" and "fundamentals". This was nowhere better exemplefied than on Fox News last night, were Republican strategist (and the tactitian behind most of the SuperPAC ads) Karl Rove refused to believe the incoming returns, and broke out into argument with Fox's own election analysts when they called the election for Obama. Even before the voting began, he was sure the Republicans had it in the bag. He was wrong.

The battle between statistician Nate Silver and the pundits — especially on the right — was stark. Pundits in the media have a built-in motivation to portray the race as a close one — nothing drives pageviews like a horserace — but their rejection him forecasting a high probability of an Obama win as nothing more than a glorified spreadsheet-jockey was a classic case of tradition versus data science. The pundit class, once valued for their instinct, insider knowledge and "expert" analysis, was clearly threatened by this successful use of data and facts. 

I've been following Silver's FiveThirtyEight blog since before the 2008 election (well before it was acquired by the New York Times), and I've always admired his methods from a statistical point of view. His forecasts are much more than simple poll-averaging, and the reasons for his success are many:

• Use of many sources data. While many pundits used national polls (and often cherry-picked ones, at that), Silver also incorporated hundreds of state-level polls into his analysis. And poll data wasn't the only source either: other factors that can influence elections, like economic variables, demographics, and party registration figures were also incorporated.
• Using the past to guide the future. Like any good statistician, Silver didn't attempt to forecast the election by extrapolating from a limited range of data points. He also incorporated historical data (electoral outcomes, historical polls, and economic data) so that typical outcomes in past elections were given some weight to reoccur in the present. 
• Extracting information from every source. While some analysts might cherry-pick data sources according to whether they were qualitatively "reliable" or "unbiased", Silver incorporated them all. And with good reason: there's still information in the presence of bias. For example, Rasmussen is a reputable polling firm, but well known to have a "house effect" that favours Republican candidates. Rather than reject this data, Silver's model instead looks at trends over time: if Rasmussen's polls move from 55% Romney to 52% Romney in a week, that's still information in Obama's favour.
• Undersanding correlations. Unlike most amateur pundits, Silver understands that political data is connected. If Texas moves in a rightwards direction, it's likely that Oklahoma will too: they're similar states that have moved in similar directions in the past. That's one reason for including as much data as possible in the model: many such correlations will be captured by modeling underlying trends like party registration or minority residency. But there are still inherent correlations between states, districts and polls that Silver was able to estimate from past data and include in his model.
• Use of statistical models. To assemble all of this information requires much more than just spreadsheet-jockeying: this is more than Moneyball, folks. This is one of the areas where Silver impresses: he used the right regression models with appropriate distributional assumption to convert all that historical and contemporary data into race-by-race forecasts. You can't do this stuff in Excel, you need sophisticated statistical modeling software. 
• Monte-Carlo simulations for the Electoral College. By far the area where Silver impressed the most was in his methodology for forecasting the presidential race. The national polls have insufficient information for this race, which is decided on a state-by-state level. His secret was to combine the results of the state-level analyses to estimate the allocation of the 538 electoral votes. (This is where the name of his blog comes from, by the way.) To incorporate all this information, and include the correlations between the state outcomes, is very difficult to do with equations. So he used an elegant solution: simulation. Every day, he ran thousands of mock elections, flipping a virtual coin (weighted according to the state-level models) for each of the 50 states, and counted the electoral votes. It was this process that led him to forecast a high probability of an Obama win: even though the national polls were close, the Democratic advantage in swing states with many electoral votes like Ohio and Pennsylvania was an ultimately insurmountable edge.
• Understanding of the limitations of polls. Any analysis is only ever as good as the input data. Polls were a big part of the analysis, but they can also be unreliable: sample bias; cell phones vs land lines; language barriers. Silver understands this fact well, and included a factor modeling this variability in bias of polls in the model. That's why, even though the "fundamentals" of the state-by-state polls made an Obama victory seem almost certain, he forecast that probability at "only" 90%. The remaining 10% was the chance that the polls were wrong enough to swing the result the other way.
• A consistency in methodology. Elections are dynamic beasts, but Silver never succumbed to the temptation to tweak his model — the only thing that changed was the incoming data. That's what made his "chance of winning" charts (as shown below) which evolved as the campaign progressed, so compelling: it reflected only changes in the facts on the ground, not changes in Silver's method of analyzing them. Contrast this with pollsters who routinely changed their sampling and weighting methodologies, making it less useful to compare changes in polls on a long-term basis.
• A focus on probabilities, not predictions. If you read his blog posts carefully (and not just others reporting on them), Silver was always careful to point out that the end result was ultimately unknowable until the final polls came in. While he was forecasting a 90% chance of an Obama win, there was still the possibility that Romney could beat the odds (or the that the polls were swinging enough in Obama's direction) to pull out a win. Even if Romney had won, it wouldn't have invalidated Silver's methodology. We can only run each election once, but if we could rewind the clock and do it over many times, I'm confident that Obama would have won a similar proportion of those times as forecast by Silver's model. 
• Great communication skills. Finally, and this is something I really admire, Silver has a great skill in being able to communicate complex statistical topics to a lay audience. Probabilities, especially, are something that most people lack an intuitive understanding for. (See: casinos, success of.) I thought his Election Eve description of Romney's chances was especially elegant, put in terms most people could relate to:

All of this leaves Mr. Romney drawing to an inside straight. I hope you’ll excuse the cliché, but it’s appropriate here: in poker, making an inside straight requires you to catch one of 4 cards out of 48 remaining in the deck, the chances of which are about 8 percent. Those are now about Mr. Romney’s chances of winning the Electoral College, according to the FiveThirtyEight forecast.

All of these points are an excellent example of Data Science in practice. Silver combined his statistical analysis skills, his stastitical software and data-wrangline prowess, and a deep understanding of polling, economic influences, and above all the Electoral College to create a winning model. He deserves his success, and I hope it convinces those in the media that data is more powerful than punditry.

Comments

[EvanZ]

Why aren't we just calling this "statistics"?

[Dave]

How exactly did he "win the election"? Obama didn't win because Silver predicted he would win, Silver predicted he would win because his data said that Obama was going to win. Had he predicted a 90% chance that Romney was going to win, Obama still would have won. The irony is that even in that case it could still be said that Silver was "right". This just happened to be one of the 10% of elections where Romney won. You make the very common mistake of mixing up prediction with causation.

I agree that his work here was very, very good and shows the possibilities that using data gives for people that want to see it. I think Silver himself explains quite clearly why people like Rove act as they do even in the face of contrary evidence. The pundits don't make their living off of uncertainty, so I wouldn't hold my breath for pundits on either side to change one bit. I hope that this at least does get more people to read Silver's most recent book since it explains a lot of this very clearly.

And for the record I voted for Romney, but I knew from the data that his chances of winning were very slim. Not everyone on Romney's side rejected Silver's numbers and I definitely don't hold him responsible for the outcome. He had no influence on the actual outcome. He was just the messenger and often too many people conflate the messenger with the message.

[Joseph]

@EvanZ This sounds far more complicated than statistics. Statistics can be used to estimate basic probabilities or to express the end result, but to call what he's doing "statistics" probably is a vast oversimplification. His analysis involves running repeated complex simulations that take into account a very large number of variables. Changing any one of these variables likely influences the value, weight, and method of calculation of a large number of other variables. In the end, he expresses the results in statistical terms, but the process involves analyzing a matrix of interrelated factors and distilling each possible outcome into a probability that we can understand. Regrettably, many of his critics chose to attack him directly rather than trying to understand and criticize any possible flaws in his methodology.

[Roland Kofler]

@EvanZ is right, one shot does not explain anything in probability. But, Silver has a little series with all winns from 2008 ongoing. Darryl Holman does the same, but fully discloses his method, also predicting right since 2008.http://horsesass.org/?page_id=39659#Q19

Put together we have an argument for statistical model prediction. Not a strong yet but about 6 wins in 6 predictions if I am correct.

@Joseph: I suspect your definition of statistics is too narrow and not what is lived in the real world. Anyway debating over definitions is seldom really useful, as Karl Popper argued.

[Rick Wicklin]

David,
Thank you for this well-written post. This is an excellent summary of the statistical methods that are essential for modern data analysis. You've presented the salient points with just the right level of detail.

As you know, 2013 is the International Year of Statistics. The idea is to let non-statisticians know what statistics is (and isn't), and how it makes a difference in the world. I intend to write a few blog posts like this one in 2013 that are aimed at a general audience. I encourage you to do the same. With this post as an example, you've set the bar pretty high.