Revolutions

R Tops Data Mining Software Poll

For the past 12 years, KDNuggets has conducted an annual poll asking "What analytics/data mining software you used in the past 12 months for a real project (not just evaluation)". In this year's poll, R was the top-ranked data mining solution, selected by 30.7% of poll respondents. Microsoft Excel was second, at 29.8%. Rapidminer, which took the #1 spot over R in 2011 and 2010, ranked third. And as Bob Muenchen notes, four of the top five ranked data mining solutions in this year's poll are open-source. R was also ranked in this poll as the most popular language for implementing data mining application, beating out SQL and Java.

See the link below for the complete list of tools ranked in the KDNuggets poll.

KDNuggets: Poll Results: Top Analytics,Data Mining, Big Data software used 

R 2.15.1 scheduled for June 22

The next release of open-source R, codenamed "Roasted Marshmallows", is scheduled to be released on June 22, according to this announcement on the r-announce mailing list. Don't expect too many changes in this update: despite the fact that "there have been very few issues with 2.15.0 ... some people may be waiting superstitiously for a .1 release". This will probably be the final patch before 2.16, due sometime next year now that R is in an annual release cycle. (Correction: R Core member Duncan Murdoch notes in the comments that there will, in fact, be at least one more patch before 2.16.)

Revolution R Enterprise, meanwhile, will be updated to the final patchlevel of the 2.14 series (R version 2.14.2) when Revolution R Enterprise 6 is released in June.

r-announce mailing list: Release plans: R-2.15.1 on June 22

Value of Open Source: Lower costs, freedom from lock-in

The results of the 2012 Future of Open Source Survey were presented at last week's OSBC conference in San Francisco. (Revolution Analytics is a collaborator in the Survey.) You can read a summary of the results in this press release, but one interesting trend is evident from Slide 13 of the presentation:

With the same question, "What makes Open Source Software Attractive" having been asked in four successive surveys, a trend emerges. Among the top three reasons each year are:

• Lower costs
• Freedom from vendor lock-in

The rank changes from year to year, but these two reasons are among the top concerns. Some other interesting conclusions from the survey:

• Open source is leading, not following, in Big Data;
• 44% of respondents identified data management as the segment most likely to be affected by open-source software in 2012.

For other highlights of the survey results, follow the link below.

BlackDuck Software: Sixth Annual Future Of Open Source Survey Results Point to Era of "Open Innovation On Demand"

Because it's a long weekend: Who's at your party?

If you're planning a house party this long weekend, you might want to take a cue from Everett Hiller to liven up your party snaps with a bit of Bill Murray:

And what party is complete without Tom Cruise on a piñata?

Yes, these are photoshops -- but awesome photoshops. (Everett clearly has some skills with lighting and composition.) You can see more celebrity substitutions at Twisted Sifter.

We'll be taking a break at the blog for the Memorial Day holiday; we'll be back will on Tuesday. Enjoy the weekend, and for those of you in the US have a great long weekend.

Twisted Sifter: This is What Happens When You Photoshop Celebrities Into Your Holiday Party

Facebook-class social network analysis with R and Hadoop

In computing, social networks are traditionally represented as graphs: a connection of nodes (people), pairs of which may be connected by edges (friend relationships). Visually, the social networks can then be represented like this:

Social network analysis often amounts to calculating the statistics on a graph like this: the number of edges (friends) connected to a particular node (person), and the distribution of the number of edges connected to nodes across the entire graph. When the graph consists of up to 10 billion elements (nodes and edges), such computations can be done on a single server with dedicated graph software like Neo4j. But bigger networks — like Facebook's social network, which is a graph with more than 60 billion elements — require a distributed solution.

Marko A. Rodriguez, a graph consultant with Aurelius, shows in a blog post how to use R and Hadoop (integrated with Revolution Analytics' RHadoop packages) to analyze Facebook-scale social networks. He first simulates a social network (shown at the top of this post) using R's igraph package, and then distributed the network in the Hadoop cluster with to.dfs function (from the rhdfs package). He then used the mapreduce function (from the rmr package) to write a simple map-reduce algorithm in R to count the number of edges associated with each node:

degree.V <- mapreduce(edge.list, 
    map=function(k,v) keyval(v[2],1), 
    reduce=function(k,v) keyval(k,length(v)))
from.dfs(degree.V)[[1]]

From there, it's another simple map-reduce job to calculate the connectivity statistics for the entire network. For more details on how Marko used RHadoop to perform this analysis, see the entire blog post linked below.

Aurelius blog: Graph Degree Distributions using R over Hadoop

Data Science shows maturity at 2012 Summit

As a discipline, Data Science is growing up fast. That's my key takeaway from the 2012 Data Science Summit.

At the inaugural 2011 Data Science Summit (you can see some highlights in this recap video), the focus was on the Big Data part of Data Science: issues with streaming data, how to store big data, technology platforms, that kind of thing. This year's summit was much more focused on the "Science" part of Data Science: applications of Big Data, and statistical issues related to the analysis of Big Data. A few examples:

• Nate Silver (political forecaster for the NYT) talked not just about building models and making predictions, but also the importance of, in his words, "embracing uncertaintly". A prediction often isn't useful without an assessment of its uncertaintly (or risk). He gave this real-life example: a flood-level prediction of 49 feet doesn't mean a city can rest easy because the levees are 51 feet high. The weather service failed to mention that there was a plus-or-minus 9 feet margin of error to that prediction, or about a 50-50 chance the city would be flooded. (It was.)
• Michael Chui (author of the McKinsey Big Data report) said that schools should be teaching more Statistics, and less Calculus, so that graduates have a better grasp of issues like sampling and selection bias.
• Michael Brown (CTO of ComScore) talked about the need to understand the impact of recall bias and outliers.
• Jeremy Howard (Chief Data Scientist of Kaggle) warned of the dangers of observation bias inherent in "data exhaust", and extolled the benefits of statistical experiments to distringuish between causality and correlation.
• Tony Jebara (co-founder of Sense Networks) expressed the need for the focus of predictive analytics to graduate from mere accuracy to making models interpretable, and predictions actionable.
• Hadley Wickham (R package author and educator) described the variety of application areas for Data Science, from cheesemakers to airport designers, and from sports teams to cruise lines. 

These are all important statistical issues, which until recently have had a back-seat to the technological and operational issues of data science. It's great to see the practice maturing, and this new focus will lead to data applications which are not just more powerful, but more reliable and more impactful as well. Data Science has come of Statistical age.

NYT charts the Facebook IPO with R

In conjunction with Facebook's record-setting IPO last Thursday, the New York Times created an infographic to put the size of the offer in context with other recent IPOs. A detail of the graphic as it appeared in the print edition appears below:

ChartsNThings gives a fascinating peek into the weeklong process that went into creating this chart, where about a dozen "sketches" of charts were tried and considered until the final chart was selected when Thursday's deadline arrived. It's a great look at the developmental process that goes into creating the quality infographics that are the hallmark of the NYT team led by graphics editor Amanda Cox. It's also a testament to the value of working in the R language (used to create all of the prototype charts), which is expressly designed to encourage experimentation and creativity in data analysis and visualization. Plus, the highly expressive nature of the language allows such prototypes to be created quickly, allowing the NYT team to iterate through many alternatives as the deadline loomed.

The post also highlights an interesting angle related to the differing requirements of the web version and print version of NYT infographics: 

If you’ve seen the web version, though, you know it doesn’t look like this. [Amanda thinks print graphics can be smarter than web graphics.] For one, the browser window doesn’t give us this kind of space. But the medium itself plays a part too. Online, if you’re not engaged in 10 seconds, you’re not going to stay on the page, so they needed to keep it fun.

The prototype below was considered for the print version, but was instead ultimately adapted for the web:

The fun aspect here is that as you switch between the different view on the web, it's not just a slideshow; each IPO "bubble" moves into it's new position as the view changes. Like a motion chart, it's a great way to track comparisons of individual data points between data views. (Try it out here.)

There are lots more details about the process of creating these charts at the post linked below; the whole thing is a great read.

ChartsNThings: Amanda Cox and countrymen chart the Facebook I.P.O.

The grade level of Congress speeches, analyzed with R

As widely reported by CNN, the Huffington Post, Talking Points Memo, the sophistication of speeches by US politicians has declined in recent years, dropping from an 11th-grade level in 2005 to a 10th-grade level today. The reports are based on an analysis by the Sunlight Foundation, based on textual analysis of congressional speeches given since 1996 provided by the Capitol Words API. You can see where your favourite legislator ranks in this table. Picking out a couple of famous names, Sen. Rand Paul (R-KY) ranks near the bottom with an 8th-grade speaking level; near the top of the ranks is retiring senator Olympia-Snowe (R-ME) who speaks at a Grade 14 level. 

A regression analysis detailed in the article teases out some of the contributing factors. While the speech levels of members of both parties have declined over the past couple of years, the chart below shows that for Republicans, the more ideological extreme the congressperson, the less sophisticated the speech.

It's clear that the data visualizations were created using R's ggplot2 package; presumably the statistical analyses were peformed with the R language as well. For the infographic circulated to the media, the ggplot2 charts have been cleaned up using an editing tool like Illustrator (easy if you export the chart to PDF from R).

Read the details of the analysis in the full report from the Sunlight Foundation linked below.

Sunlight Foundation: The changing complexity of congressional speech

A visual data summary for data frames

If you want to get a quick numerical summary of a data set, the summary function gives a nice overview for data frames:

> require(ggplot2)
Loading required package: ggplot2
> data(diamonds)
> summary(diamonds)
     carat               cut        color        clarity          depth           table      
 Min.   :0.2000   Fair     : 1610   D: 6775   SI1    :13065   Min.   :43.00   Min.   :43.00  
 1st Qu.:0.4000   Good     : 4906   E: 9797   VS2    :12258   1st Qu.:61.00   1st Qu.:56.00  
 Median :0.7000   Very Good:12082   F: 9542   SI2    : 9194   Median :61.80   Median :57.00  
 Mean   :0.7979   Premium  :13791   G:11292   VS1    : 8171   Mean   :61.75   Mean   :57.46  
 3rd Qu.:1.0400   Ideal    :21551   H: 8304   VVS2   : 5066   3rd Qu.:62.50   3rd Qu.:59.00  
 Max.   :5.0100                     I: 5422   VVS1   : 3655   Max.   :79.00   Max.   :95.00  
                                    J: 2808   (Other): 2531                                  
     price             x                y                z         
 Min.   :  326   Min.   : 0.000   Min.   : 0.000   Min.   : 0.000  
 1st Qu.:  950   1st Qu.: 4.710   1st Qu.: 4.720   1st Qu.: 2.910  
 Median : 2401   Median : 5.700   Median : 5.710   Median : 3.530  
 Mean   : 3933   Mean   : 5.731   Mean   : 5.735   Mean   : 3.539  
 3rd Qu.: 5324   3rd Qu.: 6.540   3rd Qu.: 6.540   3rd Qu.: 4.040  
 Max.   :18823   Max.   :10.740   Max.   :58.900   Max.   :31.800  

But if you'd prefer a visual overview of your data, Andrew Barr suggests the tableplot function (included in the tabplot package) for a graphical version:

tableplot(diamonds, cex = 1.8)

Andrew explains how to use the tabplot function in the post linked below.

W. Andrew Barr's Paleoecology Blog: Quickly Visualize Your Whole Dataset (via @JacquelynGill)

Because it's Friday: Game theory

Game Theory is the mathematical study of how agents in a system make choices for their actions, in light of the fact that other agents are also making competitive choices of their actions. As the name suggests, the "system" is often some kind of game and the "agents" are players, but game theory is also used to explain the behaviour of crowd motion, business dealings, foreign relations, and even the evolution of altruism. (There's a excellent chapter involving game theory in The Selfish Gene.)

The textbook example of game theory is the Prisoner's Dilemma. The UK game show "Golden Balls" includes a form of the Prisoner's Dilemma where the stake is prizemoney instead of freedom. Each player may independently choose to "split" or "steal" the prize, but if both steal each goes away empty-handed. You might think there aren't many strategic options to guarantee a win in a game like this, but one player found a way:

 

It's an elegant strategy. Nick is apparently altruistic (or at least risk-averse), and wants to share the prizemoney. But altruistic actions are open to subversion by a selfish opponent. Ibraham, unable to discern Nick's apparent "mutually assured destruction" strategy, has no option but to behave altruistically. I wonder if this strategy has ever been attempted again on the show; but now that this strategy is in the "gene pool" of strategic options, it would likely be less effective the next time around. And that's one of the subtle beauties of Game Theory.

via Business Insider: British Gameshow Contestant Puts On Badass Display Of Game Theory