Revolutions

New open-source IDE for R in beta test

Rstudio, a new open-source IDE for R, has just been announced and is now available for beta test on Windows, MacOS, and Ubuntu. It works with an existing installation of R (2.11.1 or above) on your desktop, and is also available as a server version for Linux, where the IDE runs in your browser. It looks like an interesting project, and I look forward to seeing more to come from the Rstudio team.

Rstudio Blog: RStudio, new open-source IDE for R

 

Because it's Friday: The Average Australian

Data Scientists love averages the way tigers love pepper. We all know that averages of numbers (by definition!) regress towards the mean. But the weird thing is that when you average faces, instead of numbers, the result is a face which is more attractive than any of its constituents. For example, here's the average of a number of male and female Sydneysider faces (it's not clear how many), with some of the constituent faces in between:  

This image is a detail of a poster from The Face of Tomorow, and vitual modern art exhibition at "The Contemporary Art Museum Saint Gery" in virtual world Second Life. You can find there averages of citizens of other cities too, including London, Paris and Toronto.

I found this exhibition while trying to find the source of this image, which was doing the rounds a couple of weeks ago. As far as I can tell, the source is this blog from which I guess someone created a composite of all the female figures. (If someone wants to do the same with the male figures, please do!) I'm guessing the averages themselves were made with the face averaging tool from faceresearch.org.

The Face of Tomorrow: Posters

R 2.12.2 is available

As previously announced, R 2.12.2 is available for download today. Browsing through the various mirrors (using the Download R tool on inside-R.org), it looks like the Windows version is already available on many mirrors; the Mac and Linux versions will follow soon (and of course, sources are available now). The complete list of changes is in the announcement on the r-announce mailing list.

r-announce mailing list: R 2.12.2 is released

 

Setting up a parallel computing cluster for R with OpenSSH and doSNOW

Responding to yesterday's post which included an aside on using parallel processing for by-group computations in R, reader Christian Gunning mused about the possibility of using doSNOW on his network, with OpenSSH to manage the authentication:

I sit on a fast campus network and have at least 10 remote cores available that I could farm out for big jobs. the SSHcluster method would require minimal invasion on those machines -- just ability to execute ssh and Rscript on the remote machines -- not even login privileges are required!

Well, Christian developed that idea into a step-by-step guide for setting up a multi-node cluster for parallel programming using foreach and doSNOW. Check out the full guide (including both R and shell configuration code) at the link below.

Life In Code: snow and ssh -- secure inter-machine parallelism with R

 

Packages for By-Group Processing in R

Analyst and BI expert Steve Miller takes a look at the facilities in R for doing "by-group" processing of data. The task consisted of:

... read several text files, merge the results, reshape the intermediate data, calculate some new variables, take care of missing values, attend to meta data, execute a few predictive models and graph the results.

Then repeat the models and graphs for groups or sub-populations marked by distinct values of one or more dimension variables of interest.

The latter step is commonly referred to as “by-group processing.” SAS programmers will recognize by group processing with syntax that invokes a procedure on a sorted data set that looks something like:

proc reg data = dblahblah; by vblahblah;

Check out Steve's post for how he addressed this in R using the high-performance data.table package by Matthew Dowle (and as Steve suggests, a good place to get started is the example vignettes). 

I'd also add a recommendation for the plyr package which also offers tools to split up data sets by various criteria, and then do by-processing. Here, the plyr: divide and conquer guide is a good place to start. As an added bonus, you can also divide and conquer the computations by exploiting multiple nodes in parallel by engaging a parallel backend for the foreach function. (Note for Windows users: the doSMP backend from Revolution R is also available now on R-Forge and will be on CRAN soon, too.)

Information Management: By-Group Processing, the R data.table and the Power of Open Source

 

What's the best platform for a high score on Canabalt?

The Web-based Flash game Canabalt, whose scores have been analyzed by R before, is now available as an iOS App. Because the app is configured to work on three different platforms: the iPad, iPhone and iPod Touch; and because players are invited to tweet their best scores at the end of the game, like this:

the Twitter stream again becomes a great data source to analyze the distribution of Canabalt high scores, and now also to compare performance on the various platforms, and even to analyze individual death types. 

R user Neil Kodner has done just that, by writing Python code to scrape data from such tweets stored in a MongoDB instance, and then using the plyr package in R to aggregate the data to create some interesting visualizations of the scores, such as this one by platform:

Surprisingly the iPhone, despite its smaller screen, is the platform upon which the higher scores tend to be reported. (It's also the most-played platform, which surely skews the results somewhat -- experts on extreme value theory are welcome to chime in!) The deeper analysis by type of death is also interesting: my own demise in my high-score game "somehow hitting the edge of a crane" is one of the rarer ones. Check out the full analysis and code in Neil's blog post linked below.

neilkodner.com: Visualizations of Canabalt scores scraped from Twitter

Course: Machine Learning with R

Starting on March 5 at the Hacker Dojo in Mountain View (CA), Mike Bowles and Patricia Hoffmann will present a course on Machine Learning where R will be the "lingua franca" for looking at homework problems, discussing them and comparing different solution approaches. The class will begin at the level of elementary probability and statistics and from that background survey a broad array of machine learning techniques including: Unsupervised Learning, Clustering Techniques, and Fault Detection.  

More details and registration info at the link below.

EventBrite: Machine Learning 102

Who did HBGary contact the most?

Following on from Friday's post about the travails of internet security firm HBGary, R user Michael Bommarito has done an analysis of the leaked emails to find the top 20 most contacted email addresses and the top 20 most referenced internet domains. There are some interesting names on those lists, to be sure. Check them out at the link below.

Michael Bommarito: Most Contacted HBGary Emails and Domains

New R User Groups in Canada, India

Three new local R user groups have just been added to the directory:

In Québec, the group Plein-R is affilliated with the department of Forestry, Geography and Geomatics at Laval University. Although the group's website is in French, group organizer Etienne Racine says, "Our group is bilingual. Our meetings are in a mix of French and English: we call it 'Frenglish'."

Staying in Canada, there's alse a new R user group in Vancouver, BC. The Vancouver R Users Group will host their first meeting on March 22, with an Introduction to R.

There's also a new R user group in Bangalore, India: The Bangalore R Users (BRU).  

Finally, while not a new group per se, in Dallas, TX the Dallas R User Group has a brand-new group website at meetup.com.

All four of these groups have received assistance though Revolution Analytics' R User Group Sponsorship Program. If you are a leader of a local R user group and haven't yet filled out the application form, the deadline for established  is March 31, 2011. (Brand new groups can apply for kickstart funding through the end of October.)

Revolutions: Local R User Group Directory

Because it's Friday: The darker side of internet security

At the RSA Conference this week here in San Francisco, Internet security firm HBGary was scheduled to be present. But this is what greeted vistors to the booth:

The story of how this came to pass is a fascinating one, and involves the attempts by the internet group Anonymous to conduct denial-of-service attacks on opponents of Wikileaks, an attempt by HB Gary to infiltrate the ranks of Anonymous, and the inevitable digital counterattack. Ars Technica has the full story, and it's a must-read for anyone who want to learn about practices in the darker side of internet security.

Ars Technica: How one man tracked down Anonymous—and paid a heavy price (with followups here and here)