Revolutions

What’s Next for Revolution R and Hadoop?

It's been a busy fall season for the team at Revolution Analytics. Over the past few months, we've announced major product enhancements for Revolution R --  RevoScaleR, for tackling big data sets, and RevoDeployR, for embedding Revolution R into wider applications. We've continued to add to our growing customer base at an aggressive rate and we've been busy crisscrossing the country to participate in a number of data-driven conferences and meet-ups. While it seems like we've been working around the clock, it's nice to know that our efforts are paying off and we're continuing to grow at an increasingly rapid rate.

But, it’s not just our customer base that’s growing. Revolution is expanding our team, bringing in new talent that will allow us to further develop our R-based analytics engine with an eye towards Big Data. On that note, we're proud to officially welcome Dr. Saptarshi Guha to the Revolution Analytics team. Saptarshi, whom we featured on the R-Files last month, joins Revolution as a Solutions Architect. We're thrilled to bring in someone with Saptarshi's background as we continue to build out ideal solutions that allow enterprises to tackle their biggest data challenges. Saptarshi authored the popular open source RHIPE package for integrating R and Hadoop, and at Revolution, he will be leveraging his expertise in this area to develop an advanced tool for integrating Hadoop and Revolution R.

Saptarshi's efforts will focus on developing a package that allows users to pull data out of Hadoop-based databases, including HBase and Cassandra. As he did with RHIPE, Saptarshi will be building a package that lets users perform analysis on massive data sets within the R environment, rather than having to jump across different computing platforms.

For more background on Saptarshi’s previous work integrating R and Hadoop, I’d recommend this informative Q&A that was published on the Cloudera blog prior to his talk at Hadoop World in October. We look forward to sharing more details on our efforts integrating Hadoop and Revolution R in the not-too-distant future, but in the meantime interested users can download the RHIPE package here.

The Joy of Visualizations

This is a clip from the forthcoming BBC4 program, The Joy of Stats:

 

The clip shows Hans Rosling (who we've profiled here before), plotting life expectancy versus income for various countries, and animating over time. The clip amply demonstrates that with the right presentation and story, even the simplest of scatterplots can be a joy to behold. I'm looking forward to seeing the entire program; hopefully it will be broadcast here in the States sometime soon.

John Chambers on R and Multilingualism

John Chambers, one of the creators of R's predecessor S and a current member of the R Core Group, gave a seminar at Stanford last week titled "R, other languages and object-oriented programming". Unfortunately, I was away for the Thanksgiving break and couldn't make it myself, but John has kindly made his slides (PDF) available for download.

The talk covers the benefits of multilingualism: in this case, being able to call other languages from R. It covers in particular the Rcpp interface between R and C++, and the new ability in R 2.12 to create C++-like classes that pass data by reference instead of by value. The slides give a concise introduction to reference classes in R, and note that Rcpp is the first implementation of using R reference classes from another language.

Also, be sure to check out slides 13 and 14 for an important piece of history: the first whiteboard concepts of a new language in 1976, that would lead to the development of S and, ultimately, R.

John Chambers: R, Other Languages and Object-Oriented Programming (PDF)

Happy Thanksgiving!

It's holiday time here in the US, so we're taking a break at Revolutions. We'll be back with more R goodness on Monday, but in the meantime, think of the turkeys. 

R-chart: Don't be a Turkey

R preferred by Kaggle competitors

Kaggle, the predictive-analytics competition site, has analyzed the preferences of the 2,500 data scientists who participate in its competitions, and R was the most-preferred software of the competitors at 22.5%. The next-nearest alternative was Matlab, at 16%.

On a related note, the premier of the Australian state of New South Wales has just launched a competition on Kaggle to predict the traffic on Sydney's M4 motorway. It's great to see government promoting the use of data analysis to solve (or at least better understand) civic problems, and this competition comes with some serious prizemoney: AUD$10,000 (about the same in $USD). Might be worth your time spending the Thanksgiving break doing a little modeling in R...

No Free Hunch: Profiling Kaggle’s user base

How to make beautiful bubble charts with R

Nathan Yau has just published at FlowingData a step-by-step guide on making bubble charts in R. It's actually pretty simple: read in data, sqrt-transform the "bubble" variable (to scale the bubbles by area, not radius), and use the symbols function to plot. It's the last step, though, that really ups the presentation quality: read R's PDF file into Illustrator and clean up for publication:

You can mess around with this in R, if you like, but I've found it's way easier to save my file as a PDF and do what I want with Illustrator. I uncluttered the state labels to make them more readable, rotated the y-axis labels, so that they're horizontal, added a legend for population, and removed the outside border. I also brought Georgia to the front, because most of it was hidden by Texas.

The final chart is, indeed, beautiful:

FlowingData: How to Make Bubble Charts

R and AOL in NYC

R and the NYC R User Group get brief mentions in this article about AOL's offices in New York City. The NYC UseRs meet at AOL and (ironically) the next meeting on Dec 9 is on the topic of R at Google.

New York Observer: Bringing Some Sizzle to the Dial-Up King (via)

Access the InfoChimps API from R

InfoChimps.com is mainly known as a clearinghouse for finding large data sets, for free or for sale. But they have also released (in beta, at least) an API that lets you find some pretty useful information on-demand. Normally, you'd have you use RESTful calls to access the API, but now Drew Conway has created an R package (and released gist sources) that lets you query the API using simple R commands. With the infochimps package, you can:

• Find information about a Twitter user, such as their "Word Bag" and strong links
• Create a data frame of conversations between two Twitter users
• Find the geographical location of an IP address and demographic data from the US census
• ...and more.

To use the infochimps package in R, you'll need to sign up for an API key. (The free subscription is limited in the number of API calls allowed, and requires attribution.)

inside-R Package Reference: infochimps

Because it's Friday: A t-shirt for stats geeks

Saw this in the office yesterday and had to share. Revolutionary Saar Golde models a clever T-shirt:

Yes, it says "Stanford Economics", but as a stats geek I loved it (although who calls their dependent variable L?). Saar is wearing a cotton T-shirt, but I believe it also comes in scratchy-yet-durable hessian.

Airport security: science vs backlash

The United States has recently introduced millimeter wave and backscatter x-ray scanners to the security screening process in many airports, prompting a backlash in some quarters. Much of the opposition is centered around the invasion of privacy: the scanners generate an image of the traveller's naked body. There are also health concerns, at least for the backscatter x-ray variants of the scanners. I had initially dismissed these concerns as part of the usual overblown fears of anything related to radiation, but some scientists have recently raised concerns that while the radiation dose is low, unlike a medical x-ray or cosmic radiation, the entire dose is concentrated on the skin, intensifying the local dose by orders of magnitude. Personally, I agree with the that more study is warranted.

But are the scanners worth it, overall? Whether they are successful in deterring terrorists is one question that's been endlessly debated. But what about the effect of these security procedures on the economy? Nate Silver of the New York Times likens these security procedures as a "tax upon air travel":

Teleconferences are often a poor substitute for person-to-person interaction, and when people are reluctant to travel, some business deals don’t get done that otherwise would have. Recreational travelers, meanwhile, may skip out on vacations that otherwise would have brought them pleasure and stress-relief (while improving revenues for tourism-dependent economies). The tenuous profits of the airline industry are also affected, of course. Revenue losses from the new bag-checking procedures may have measured in the billions.

It certainly seems plausible that there is a quantifiable drain on the economy. But could we measure it? Nate suggests we could, via a statistical experiment.

If, for instance, Chicago O’Hare installs new machines and Chicago Midway does not, and passenger traffic at O’Hare drops by 9 percent while traffic at Midway holds steady, that would provide considerably more tangible evidence of how travelers are reacting to the new protocols than polls ever could.

It's a great idea. It's just a shame that political considerations would never allow it to be implemented.

New York Times: The Hidden Costs of Extra Airport Security