Revolutions

NYT uses R to investigate NFL draft picks

Last week, the New York Times published online an interactive tool to explore NFL draft picks, revealing the fact that there's not much relationship between an early pick and the star performers in the season:

Kevin Quealy, graphics editor at the NYT, detailed the process behind creating this graphic on his chartsnthings blog. He and others on the graphics team used the R language to rapidly sketch many different options before this final version was settled upon, and it's fascinating to see the creative process in action. The resolution and space afforded by the print edition allowed for an even more information-dense graphic; compare the detail of the final print version (below, top) with the R version (below, bottom):

The Times regularly uses R to develop its online and print data visualizations. Other recent examples include investigations of the Facebook IPO, the wealthiest 1% of Americans, and Mariano Rivera’s pitching prowess. You can find the details of how these and other visualizations were created with R at the chartsnthings blog, linked below.

chartsnthings: Charting Skill and Chance in the N.F.L. Draft (via Sharon Machlis)

How UpStream uses R for Attribution Analysis

Major retailers like Williams Sonoma use UpStream Software for marketing analytics, including revenue attribution, targeting, and optimization. In the video below Tess Nesbitt (senior statistician at UpStream) describes how she uses Revolution R Enterprise and Hadoop to figure out the impact on various marketing channels (for example direct mail, email offers, and catalogs) on consumer retail sales.

 

(The slides for Tess's presentation, The Impact of Big Data on Marketing Analytics, are available for download.) Because Tess needs to massive amounts of consumer behaviour data to tease out the effects of the different marketing channels, she switched from SAS/WPS and now uses the big-data capabilities of Revolution R Enterprise to fit a survival model to more than 36 million records in less than 4 minutes. (You can find some of the details of the model and the R code used to fit it in Tess's recent presentation to the Bay Area R User Group, Statistical Marketing Analytics with Big Data (PPT).) By reducing the time it takes to fit the model from an overnight wait to the time it takes to make a cup of coffee, Tess has more opportunity to refine and improve the model, as she describes in a recent Channel Reseller News article:

"Revolution R Enterprise 6.2 enables UpStream to build highly efficient statistical models on extremely large data. Because their parallelized algorithms are so efficient, it enables us to take multiple passes at the data, build iterative models, and it provides everything we need to glean as much information and build the best models we can for our customers."

These more powerful models in turn mean that retailers can finally understand what marketing activities are most likely to lead consumers to make purchases, as Mohan Namboodiri (VP Customer Analytics, Williams Sonoma) describes in this GigaOM Structure 2013 panel discussion.

To learn more about how UpStream uses Revolution R Enterprise, take a look at this Revolution Analytics case study.

Facebook profile photo changes reveal trends in support for same-sex marriage

On Tuesday and Wednesday this week, the US Supreme Court heard two landmark cases related to same-sex marriage rights in this country. Simultaneously, to show solidarity for the cause of civil rights for gay members of society, thousands of Facebook users changed their profile pictures to a red equality symbol at the suggestion of the Human Rights Campaign. Most profile pictures looked like the one shown at right, but there were also many variants used as well. (By the way, here's R code use can use to generate the red equality logo, with thanks to Josef Fruehwald.)

The change in Facebook profiles provided some interesting information about the support of advances in gay rights across the USA. Some clever data science from Facebook's Eytan Bakshy reveals that there was indeed a significant increase in the rate of profile picture changes during the oral hearings:

(The chart above is the result of a time-series analysis using the R language, after extracting the normal day-night cycle of profile updates.) Moreover, by looking at the rate of changes of Facebook profiles across every county in the USA in a choropleth, we see those regions where support for gay rights is the strongest:

While it's unlikely that the Justices will consult Facebook profiles as part of their deliberations, the proceedings did seem to be favourable to the advance of gay rights. (Especially for the second case regarding DOMA, which if overturned will be a hugely positive outcome for thousands of couples already married, including me and my husband.) But for those of us for whom this cause means so much personally, it was truly heartwarming to see that sea of red from allies all across the county. What strikes me so much about the map above is that support is coming from all across the country — not one county is a totally unsupportive white. And that's something that all of America can be proud of.

Facebook (Eytan Bakshy): Showing Support for Marriage Equality on Facebook (via Sean J. Taylor)

Explore March Madness face-offs with this NCAA data visualizer

If you're laying down a friendly bet on the March Madness games or just tweaking your fantasy roster, this NCAA Data Visualizer by Rodrigo Zamith will be a boon. Just choose two teams to compare head-to-head, choose an attribute to compare them on. You can look at more than a dozen invividual player attributes (e.g. points scored, assists, 3-point shots made) or over 20 team attributes (e.g. fouls, wins, and turnovers), and compare against other top players or NCAA team averages.

For example, here's how the top players match up for individual points scored for tomorrow's Oregon/Saint Louis game:

All of the data used was scraped from the NCAA website for the 2012-2103 season. Rodrigo created the visualizations using the R programming language, which means he can used some advanced techniques like the boxplots above. Rather than just showing the median or highest/lowest scores for each player, he can show all of each player's games, and highlight the bulk (the middle 50%) of their games using the box. But you can still see, for example, that Saint Louis's Kwamain Mitchell had a couple of hot streaks last season and was in fact their highest scorer last season — a fact that you might miss just looking at averages.

You can play with the NCAA data visualizer and choose your own teams and stats to compare at the link below.

Rodrigo Zamith: Visualizing Season Performance by NCAA Tournament Teams (via Myles Harrison)

Find the fairest place to meet on the Paris Métro

When I lived in Paris years ago, I worked near Gare du Nord, but my friend Jenny lived near République. If we wanted to meet up after work, we'd just meet halfway along the Orange Métro line, around Gare de l'Est. Easy. Since that's within walking distance we wouldn't actually take the Métro, but Métro stations are useful waypoints in Paris even if you're travelling by foot.

But when we wanted to meet with our other friend Eric who would travel from Porte de Versailles, it was tricky to find a mutually-agreeable place to meet, so that we'd all be taking an approximately equally long Métro ride. If only we'd had Jean-Robert's Metro Meeting Point app, it would have made the decision an easy one.

The app works by representing the Métro network as a connected graph, and finding the station that minimises the maximum number of stops from each of 2 or 3 or more starting points. This is possible thanks to the fact that the RATP (the Paris transit agency) has published open data on Métro stops and their geographic location. All it took was for Jean-Robert to write a script in the R language to import the data and create the network as an R object, and then use the Djikstra algorithm and minimax optimization to find the best meeting point. The web interface for the app was done using RStudio's Shiny.

For complete details on how Jean-Robert created this cool and useful app from open data, check out his blog post linked below. As for me, I'll be daydreaming about Spring evenings in a Paris café with a kir royale in hand.

Jean-Robert blog: Optimal Meeting Point on the Paris Metro

A map of worldwide email traffic, created with R

The Washington Post reports that by analyzing more than 10 million emails sent through the Yahoo! Mail service in 2012, a team of researchers used the R language to create a map of countries whose citizens email each other most frequently:

The chart above shows the top 1000 country-country pairs by email frequency, arranged in a clustered network using the R igraph package. While it's not surprising that countries that share proximity, language and culture might tend to email each other more often, it's interesting that the clusters tend to resemble those of Huntington's Clash of Civilizations, whose "major civilizations" are used to color-code the map. (A formal test of similarity was done using R's SNA package, with country-specific factors corrected for using a linear mixed-effects model.) On this point, the authors write:

In this respect we cautiously assign a level of validity to Huntington’s contentions, with a few caveats. The first issue was already mentioned – overlap between civilizations and other factors contributing to countries’ level of association. Huntington’s thesis is clearly reflected in the graph [above], but some of these civilizational clusters are found to be explained by other factors ... The second limitation concerns the fact that we investigated a communication network. There is no necessary “clash” between countries that do not communicate, and Huntington’s thesis was concerned primarily with ethnic conflict.

Indeed, the validity of Huntington’s ideas with respect to ethnic conflict has come into controversy, and we limit ourselves to showing the validity — at least partial — of this division for communication networks.

The Washington Post article (linked below) has more discussion about the research, and you can find details of the methodology in the arXiv paper.

Washington Post: An incredible map of which countries e-mail each other, and why (via reader GF, with thanks)

Track the bookies' favourites for the next Pope

Tired of manually running a python script to scrape the latest bookmaker odds on the next Pope, R user AJ (an analytical research manager at a large healthcare company) instead created an R script to track the odds on the Papal successor, and automated it with the Shiny package for R. The screenshot below shows the odds of each candidate being elected Pope since Benedict XVI's adbication was announced: as of today Cardinal Turkson of Ghana is the bookies' favourite. Click on the chart to see the latest odds, where you can interactively display up to the top 15 candidates:

AJ published the R code behind the chart on GitHub, and provides more details in the blog post below.

Data, data, data: pope tracker

Analyze web traffic data with Google Analytics and R

If you run an e-commerce site, blog or other web property there's a good chance you use Google Analytics to monitor traffic, look at visitor sources, and measure conversions. And while Google Analytics is quite powerful at looking at historic activity on your site, it lacks much in the way of predictive analytics. That's where R shines of course, and it's now quite easy to connect R to your Google Analytics account to make use of R's superior data visualization, forecasting, and other advanced analytics capabilities.

Digital analytics expert Randy Zwitch explains how to use the rga package for R to access your Google Analytics data and import it into R. There, you can use all of R's capabilities to analyze the data. For example, you can use boxplots to look at the proportion of visits to a site via a Google search where the search terms are actually provided:

To interface with Google Analytics you'll need an access token for the Google Analytics API console (the Tatvic Blog provides some tips for obtaining one). Once you're set up, you can run the code at the link below to create your own version of the chart above, or simply start using R to analyze your web traffic data.

randyzwitch.com: (not provided): Using R and the Google Analytics API

Political revolutions on Twitter, visualized with R

Twitter has become a powerful medium for organizing and communicating with factions during popular uprisings: the crisis in Egypt, the uprising in Syria, the revolution in Iran, and other conflicts all around the world. Twitter's effectiveness relies on its ability for the various factions to self-organize and to fight the information battle in social media.

Esteban Moro Egido, a mathematics professor at Universidad Carlos III in Madrid, puts this battle into stark relief with a video depicting Twitter activity around Spain's general strike in March this year. Esteban has used the R language for years to understand complex networks with applications in areas such as telecommunications and social media, and has put those skills to great use analyzing all of the tweets, retweets and mentions related to the strike. Here's the video:

 

Each point in the animation represents a twitter user, each colour-coded according to their faction in the debate (pro-strike, anti-strike, or somewhere in between). He used community-finiding algorithms to automatically assign Twitter users to factions, and Esteban described the factions in an email:

The communities where identified using one of the community-finding algorithms in R. Specifically the walk trap algorithm run over the static graph of (weighted) RT graph between twitter accounts. So each Twitter account is assigned to a particular community right from the beginning. 

What we did afterwards was to check what these communities were talking about (tweets and RTs). We found that the tweets and RTs in the orange community were in favor of the general strike and the reasons behind it, while the tweets and RTs in the dark blue community were against the unions and the reasons behind the general strike. It is interesting to see that the communities found in the structural analysis of the RT graph also correspond to opinion communities, the reason being that there are not many RTs between groups which have such different opinions.  

There are other two communities in the video (light blue and green) which correspond to news media and, a bot network automatically tweeting about the general strike.

The animation itself was created entirely with R using the igraph package, and encoded to video using ffmpeg. You can create similar videos yourself for other dynamic political discussions on Twitter: Esteban has kindly provided a tutorial on how to create such animations, with R code. 

Implicit None: Temporal network of information diffusion in Twitter

 

R analysis shows how UK health system could save £200m

According to an analysis by Prescribing Analytics (a joint venture of technologists and doctors in the UK), Britain's cash-strapped National Health Service (NHS) is overspending on prescription drugs. While cheaper (but equally effective) generic drugs are widely available for many treatments, some doctors continue to prescribe patented drugs which can cost 10 times as much — and often much more. The problem varies regionally: for example, in the administrative region including the city of York nearly 30% of prescriptions for statins (drugs used to lower cholesterol levels) are for expensive non-generics:

Looked at another way, a nearly 70% generic prescribing rate seems pretty good. But the cost differential in generics vs patented drugs is the key here: generic Simvastatin (in blue below) still represents a small fraction of the costs for prescribed statins, compared to non-generic drugs containing Atorvastatin and Rosuvastatin.  According to the study:

Guidelines for that period said that the generic Simvastatin should be a default option for patients. Wherever the percentage of branded items is high, it represents potential to make big savings by switching. For example, if two thirds of the proprietary drugs had switched to the generic forms of the same drugs in the year to June 2012, this would have saved £200 million pounds. Any savings could be used to help other patients.

The analysis was done using R statistics software (and the R code is available for download at github). The prescription data is available thanks to the open data practices of the NHS, but unfortunately a precise estimate of cost savings isn't currently possible because prescription drug price data isn't public.

Prescription Analytics: NHS efficiency savings: the role of prescribing analytics (via Ben Goldacre)

Update: Corrected the description of statins in the first paragraph.