Revolutions

Facebook profile photo changes reveal trends in support for same-sex marriage

On Tuesday and Wednesday this week, the US Supreme Court heard two landmark cases related to same-sex marriage rights in this country. Simultaneously, to show solidarity for the cause of civil rights for gay members of society, thousands of Facebook users changed their profile pictures to a red equality symbol at the suggestion of the Human Rights Campaign. Most profile pictures looked like the one shown at right, but there were also many variants used as well. (By the way, here's R code use can use to generate the red equality logo, with thanks to Josef Fruehwald.)

The change in Facebook profiles provided some interesting information about the support of advances in gay rights across the USA. Some clever data science from Facebook's Eytan Bakshy reveals that there was indeed a significant increase in the rate of profile picture changes during the oral hearings:

(The chart above is the result of a time-series analysis using the R language, after extracting the normal day-night cycle of profile updates.) Moreover, by looking at the rate of changes of Facebook profiles across every county in the USA in a choropleth, we see those regions where support for gay rights is the strongest:

While it's unlikely that the Justices will consult Facebook profiles as part of their deliberations, the proceedings did seem to be favourable to the advance of gay rights. (Especially for the second case regarding DOMA, which if overturned will be a hugely positive outcome for thousands of couples already married, including me and my husband.) But for those of us for whom this cause means so much personally, it was truly heartwarming to see that sea of red from allies all across the county. What strikes me so much about the map above is that support is coming from all across the country — not one county is a totally unsupportive white. And that's something that all of America can be proud of.

Facebook (Eytan Bakshy): Showing Support for Marriage Equality on Facebook (via Sean J. Taylor)

Track the bookies' favourites for the next Pope

Tired of manually running a python script to scrape the latest bookmaker odds on the next Pope, R user AJ (an analytical research manager at a large healthcare company) instead created an R script to track the odds on the Papal successor, and automated it with the Shiny package for R. The screenshot below shows the odds of each candidate being elected Pope since Benedict XVI's adbication was announced: as of today Cardinal Turkson of Ghana is the bookies' favourite. Click on the chart to see the latest odds, where you can interactively display up to the top 15 candidates:

AJ published the R code behind the chart on GitHub, and provides more details in the blog post below.

Data, data, data: pope tracker

Because it's Friday: Surprise Russian meteor

While astronomers were waiting for one asteroid to fly by, something much more exciting happened: a different, much smaller rock plunged through the atmosphere over Russia to become a spectacular meteor. Many Russians have dashboard cameras running at all times, so the event was caught hundreds of times on video. Here's one of my favourites: wait a few seconds to see the meteor suddenly appear in the sky and create an impressive train as it streaks by:

 

Talking Points Memo has a great roundup of other videos, including several of curious Russians filming the train in a serene blue sky, only to be jolted by the sound of the sonic boom created by the supersonic object. (Many people were injured by shattered glass from the boom, not as had been reported, by falling rock.) Phil Plait has some great reporting on the event, including this photo (by Itar-Tass Photos/Newscom) of where the meteorite apparently plunged through the ice of a frozen lake:

While astronomers knew about the flyby of 2012 DA14 ahead of time, this Russian meteor was not being tracked and came as a complete surprise. Just imagine if something like this had happened in a big city like New York! The damage and danger from thousands of skyscraper windows exploding from the sonic boom would catastrophic enough, not to mention the explosion that could occur if a small object like this actually hit a building. That's why it's critical to fund NASA's asteroid-tracking program, not just so we'll know if a dinosaur-killer is on the way, but so we can be forewarned of smaller, but still dangerous, objects like this one in Russia.

Something to think about over the weekend! Have a good one. 

Did an Excel error bring down the London Whale?

When JP Morgan Chase announced it had lost more than 2 billion dollars on the capital markets back in May 2012, many pointed to the actions of rogue trader Bruno Iksil as the cause. But was the "London Whale" — the nickname he was given by other traders for his outsized positions — the victim not of hubris, but a simple spreadsheet error?

James Kwak, associate professor at the University of Connecticut School of Law and co-founder of the Baseline Scenario blog, noted some interesting facts in JP Morgan Chase's post-mortem investigation of the losses. Specifically, that the Value at Risk (VaR) model that underpinned the hedging strategy

“operated through a series of Excel spreadsheets, which had to be completed manually, by a process of copying and pasting data from one spreadsheet to another", and "that it should be automated" but never was.

This is a surprisingly common practice: through accretion and incremental advancements, an important statistical calculation somehow ends up being implemented as a convoluted series of Excel worksheets, connected by hundreds (or even thousands) of cell-reference formulas, all driven by a series of input parameters that need to be entered manually. Not only does this impose the risk of introducing errors when cutting-and-pasting the inputs, it also makes the workbook extremely fragile. As anyone who's build a budget in Excel knows, it's very easy when editing the spreadsheet to find that formulas no longer extend to their expected ranges (ever missed the bottom row from a formula when adding new data?), or point to the wrong data entirely. And then there's the possibility of errors in the formulas themselves, which seemed to have been an issue here as well:

“After subtracting the old rate from the new rate, the spreadsheet divided by their sum instead of their average, as the modeler had intended. This error likely had the effect of muting volatility by a factor of two and of lowering the VaR . . .”

Excel is an excellent tool for many applications, but the intertwined cross-references of formulas make errors like this hard to detect, and hard to correct even if discovered. That's why a programming language designed for data analysis such as the R language is a better platform for building the computational engines behind VaR models and other financial systems. Not only can it automate the process of importing data and inputs from other systems (and thus eliminate cut-and-paste errors), it also provides a structured, maintainable environment for the computational logic, within a framework that promotes code review and unit testing to detect errors. Excel may still be the preferred vehicle for delivering the results, but use R to generate the analytics computations (or embed real-time R computations in Excel) instead of risking an implementation in Excel formulas.

Read also: How Validus Re uses Revolution R Enteprise for risk management

The Baseline Scenario: The Importance of Excel (via Ben Lorica)

Tutorial: How to make NYT-style bar charts with R

New York Times columnist Charles Blow needed a chart to accompany his op-ed piece Lincoln, Liberty and Two Americas (about one-party control in state legislatures). So he turned to resident graphic editor Kevin Quealy, who found the source data and used R to create the chart below:  

If you'd like to create similar charts yourself, Kevin provides a useful tutorial on his behind-the-scenes blog, chartsnthings. It's just a few lines of base graphics in R, beginning with line plots and then moving on to bar plots. The abline function is used to create the gridlines. You can also use the same technique to create "waffle charts" (the display technique often used by the NYT, where small blocks represent data elements). You can find the full tutorial at the link below.

chartsnthings: R tutorial: Simple charts

How Nate Silver won the election with Data Science

While I'm pleased with the outcome of yesterday's election in political terms, I'm equally thrilled about the victory for Data Science versus "gut instinct", "philosophy" and "fundamentals". This was nowhere better exemplefied than on Fox News last night, were Republican strategist (and the tactitian behind most of the SuperPAC ads) Karl Rove refused to believe the incoming returns, and broke out into argument with Fox's own election analysts when they called the election for Obama. Even before the voting began, he was sure the Republicans had it in the bag. He was wrong.

The battle between statistician Nate Silver and the pundits — especially on the right — was stark. Pundits in the media have a built-in motivation to portray the race as a close one — nothing drives pageviews like a horserace — but their rejection him forecasting a high probability of an Obama win as nothing more than a glorified spreadsheet-jockey was a classic case of tradition versus data science. The pundit class, once valued for their instinct, insider knowledge and "expert" analysis, was clearly threatened by this successful use of data and facts. 

I've been following Silver's FiveThirtyEight blog since before the 2008 election (well before it was acquired by the New York Times), and I've always admired his methods from a statistical point of view. His forecasts are much more than simple poll-averaging, and the reasons for his success are many:

• Use of many sources data. While many pundits used national polls (and often cherry-picked ones, at that), Silver also incorporated hundreds of state-level polls into his analysis. And poll data wasn't the only source either: other factors that can influence elections, like economic variables, demographics, and party registration figures were also incorporated.
• Using the past to guide the future. Like any good statistician, Silver didn't attempt to forecast the election by extrapolating from a limited range of data points. He also incorporated historical data (electoral outcomes, historical polls, and economic data) so that typical outcomes in past elections were given some weight to reoccur in the present. 
• Extracting information from every source. While some analysts might cherry-pick data sources according to whether they were qualitatively "reliable" or "unbiased", Silver incorporated them all. And with good reason: there's still information in the presence of bias. For example, Rasmussen is a reputable polling firm, but well known to have a "house effect" that favours Republican candidates. Rather than reject this data, Silver's model instead looks at trends over time: if Rasmussen's polls move from 55% Romney to 52% Romney in a week, that's still information in Obama's favour.
• Undersanding correlations. Unlike most amateur pundits, Silver understands that political data is connected. If Texas moves in a rightwards direction, it's likely that Oklahoma will too: they're similar states that have moved in similar directions in the past. That's one reason for including as much data as possible in the model: many such correlations will be captured by modeling underlying trends like party registration or minority residency. But there are still inherent correlations between states, districts and polls that Silver was able to estimate from past data and include in his model.
• Use of statistical models. To assemble all of this information requires much more than just spreadsheet-jockeying: this is more than Moneyball, folks. This is one of the areas where Silver impresses: he used the right regression models with appropriate distributional assumption to convert all that historical and contemporary data into race-by-race forecasts. You can't do this stuff in Excel, you need sophisticated statistical modeling software. 
• Monte-Carlo simulations for the Electoral College. By far the area where Silver impressed the most was in his methodology for forecasting the presidential race. The national polls have insufficient information for this race, which is decided on a state-by-state level. His secret was to combine the results of the state-level analyses to estimate the allocation of the 538 electoral votes. (This is where the name of his blog comes from, by the way.) To incorporate all this information, and include the correlations between the state outcomes, is very difficult to do with equations. So he used an elegant solution: simulation. Every day, he ran thousands of mock elections, flipping a virtual coin (weighted according to the state-level models) for each of the 50 states, and counted the electoral votes. It was this process that led him to forecast a high probability of an Obama win: even though the national polls were close, the Democratic advantage in swing states with many electoral votes like Ohio and Pennsylvania was an ultimately insurmountable edge.
• Understanding of the limitations of polls. Any analysis is only ever as good as the input data. Polls were a big part of the analysis, but they can also be unreliable: sample bias; cell phones vs land lines; language barriers. Silver understands this fact well, and included a factor modeling this variability in bias of polls in the model. That's why, even though the "fundamentals" of the state-by-state polls made an Obama victory seem almost certain, he forecast that probability at "only" 90%. The remaining 10% was the chance that the polls were wrong enough to swing the result the other way.
• A consistency in methodology. Elections are dynamic beasts, but Silver never succumbed to the temptation to tweak his model — the only thing that changed was the incoming data. That's what made his "chance of winning" charts (as shown below) which evolved as the campaign progressed, so compelling: it reflected only changes in the facts on the ground, not changes in Silver's method of analyzing them. Contrast this with pollsters who routinely changed their sampling and weighting methodologies, making it less useful to compare changes in polls on a long-term basis.
• A focus on probabilities, not predictions. If you read his blog posts carefully (and not just others reporting on them), Silver was always careful to point out that the end result was ultimately unknowable until the final polls came in. While he was forecasting a 90% chance of an Obama win, there was still the possibility that Romney could beat the odds (or the that the polls were swinging enough in Obama's direction) to pull out a win. Even if Romney had won, it wouldn't have invalidated Silver's methodology. We can only run each election once, but if we could rewind the clock and do it over many times, I'm confident that Obama would have won a similar proportion of those times as forecast by Silver's model. 
• Great communication skills. Finally, and this is something I really admire, Silver has a great skill in being able to communicate complex statistical topics to a lay audience. Probabilities, especially, are something that most people lack an intuitive understanding for. (See: casinos, success of.) I thought his Election Eve description of Romney's chances was especially elegant, put in terms most people could relate to:

All of this leaves Mr. Romney drawing to an inside straight. I hope you’ll excuse the cliché, but it’s appropriate here: in poker, making an inside straight requires you to catch one of 4 cards out of 48 remaining in the deck, the chances of which are about 8 percent. Those are now about Mr. Romney’s chances of winning the Electoral College, according to the FiveThirtyEight forecast.

All of these points are an excellent example of Data Science in practice. Silver combined his statistical analysis skills, his stastitical software and data-wrangline prowess, and a deep understanding of polling, economic influences, and above all the Electoral College to create a winning model. He deserves his success, and I hope it convinces those in the media that data is more powerful than punditry.

Another look at ideology of the US congress

In response to last week's post on the rapidly increasing ideology of the US Republican Party, Mike Lawrence suggested another way of looking at the DW-NOMINATE ideology data. Rather than simply looking at boxplots of the congress scores by party over time, we could fit a smooth curve to get a better sense of the trends over time. Mike provided R code to do exactly that, using a generalized additive model to fit the smooth curve to the data since the two-party system began, and using his ez package to visualize the result. I modified Mike's code slightly to match the traditional party colours and to put year (rather than congress number) on the horizontal axis:

In this chart, higher means more rightward-leaning. While the Democratic party has drifted leftwards over the past 50 years, you can see a sudden turn to the right beginning in 2005. And in this version of the chart you can clearly see the march of the Republican party to the far right, beginning in 1975.

Watch Obama and Romney criss-cross the US

The Washington Post has an interactive graphic showing the rate at which the US presidential candidates Barack Obama and Mitt Romney have visited the various states for campaign rallies and fundraisers. Here's how it looks today:

You can clearly see the focus on key swing states like Florida and Ohio, as well as non-competitive (but donor-rich) states like California and Texas. But this aggregate view doesn't give a sense of the sheer logistics of the candidates (plus their entourages and press pool) travelling across the country from event to event. 

So statistician Jerzy Wieczorek used data on the invididual campaign stops and dates to create this animated visualization of Obama's and Romney's itineraries over the past few months:

Now that's a hectic travel schedule! And that's not even all the travel: this animation shows only the campaign stops, not other trips unrelated to the campaign (such as returning to Washington DC for the President, or Romney returning to his Boston HQ). It's a bit of a shame they're not collecting frequent flyer miles in the process.

Jerzy used the R language to create the animation. This involved extracting the campaign stops in JSON format, geocoding the campaign stop locations into latitudes and longitudes (for mapping), and breaking down each day into 24 one-hour blocks so the animation runs smoothly in accelerated time. He then used the code behind this historic shipping animation to create the video above. R programmers can find all the technical details at Jerzy's blog post, linked below.

Civil Statistician: Animated map of 2012 US election campaigning, with R and ffmpeg 

 

 

More data apps spawned by Sandy

As the clean-up continues on the eastern seaboard, I wanted to follow up on Monday's post on tracking Hurricane Sandy with Open Data with a couple of other R-based data applications spawned by the storm.

Josef Fruehwald created an R script to tap into local weather sensors to keep track of air pressure, wind speed and rainfall near his home and for other affected locations. Check out the plunge in barometric pressure as the centre of the storm passed near Atlantic City on Monday night, and then over Philadelphia a couple of hours later:

The greater scale of destruction in Atlantic City compared to Philadelphia is reflected in this comparison of wind speeds over the same period:

Joseph used R to download the data from Weather Underground station records (like this one) in XML format, and then used the ggplot2 package to create the charts. You can find the complete R script at RPubs: Analysis of a Philadelphia Weather Station Data during Hurricane Sandy.

Twitter's Ed Chen took a different look at the effects of Sandy: how the storm knocked out power around the NY region, and the race by the utulities to restore power. Where did he find such data? Tweets from Twitter users, of course! Geolocated tweets (from smartphones, presumably) along the lines of "I've just lost power!" , "Still no power here!" and "The power is back on!" gave Ed information he needed to use R to create an animation over time of the power going out and coming back on across NYC.

The above is just a snapshot; visit Ed Chen's blog for an animated version, with a slider to see the changes during the night of the storm.

Tracking Hurricane Sandy with Open Data and R

Hurricane Sandy is shaping up to be a major, and very dangerous, meteorological event for the US's East coast. Naturally, everyone is looking for the latest information and forecasts. Fortunately, the wealth of public meteorological data available on the open web, combined with real-time on-the-ground updates via social media, means that an ecosystem of on-line apps is now available providing all the up-to-date information you need. O'Reilly's Strata site has a good overview of the open-data Sandy-tracking apps available.

For example, R user Bob Rudis has written an R script to grab up-to-date hurricane tracking data from Unisys Weather, and combine it with the forecast cone from Google's crisis map, to create the following combined picture of where Sandy's been, and where it's going:

The forecast track shows Sandy heading north into Canada. It differers from the version on Bob's blog post because I ran the R script just now (at 16:38 PST) — that's the power of using live data!

If you're anywhere near Sandy, stay informed and above all, stay safe. You can find information on how to keep safe at Ready.gov.

Update Oct 30: See also this Analysis of a Philadelphia Weather Station Data during Hurricane Sandy by Josef Fruehwald, featuring charts of barametric pressure, wind speed and rose charts of wind direction created from live data using R.

rud.is: Watch Sandy in “R” (Including Forecast Cone)