Revolutions

Wanted: Big-data beta testers

We're nearing completion of the package of statistical tools for very large data sets that I gave an early preview of at R/Finance 2010. It will be released for Revolution R Enterprise later this year, but we're looking for some R users with big data sets to put the 1.0 version through its paces in the beta program and offer suggestions for subsequent versions. 

For now, it's a limited beta program -- we're looking for testers with some fairly specific attributes:

• You'll need to be running a recent version of Windows (other platforms will be open for testing later) on a machine with at least a 2-core and ideally 4-core or more processors. 
• You'll need to have a large data set (larger than 1 GB, up to around 200 GB for now) for analysis with linear regression, logistic regression, or crosstabulations. The data should be in text (ideally CSV) format or can be exported as such.

If you think you might fit the bill, send me an email at david@revolutionanalytics.com with a brief description of your setup and we'll see if we can get you in to the beta program. Thanks!

How to peg 7 cores with doSMP

Statistics PhD student Nathan VanHoudnos has an 8-core laptop, and by his own admission, takes "an almost unhealthy pleasure in pushing [his] computer to its limits". It seems like he's found an outlet for this passion with the new doSMP library included with Revolution R, that allows him to use all his processors for some gnarly simulations in R:

 

Actually, he's written his code to use 7 processors instead of 8: it's good practice to leave one CPU unpegged if you want to continue to use your computer interactively during the computations. See Nathan's R code at the link below.

Nathan VanHoudnos: Pegging your multicore CPU in Revolution R, Good and Bad

Making Data Work online conference

O'Reilly is hosting a conference on June 9 on the topic of the analysis of large data sets. The title of the conference is Making Data Work:

Ever since Hal Varian proclaimed that data analysis is the sexy career for the coming decade, people have been talking about data. And big data. And even bigger data.

This online conference, Making Data Work, brings together four experts on data and data analysis to show you what data is all about. How is data used in successful enterprises? What kinds of tools do you use to work wth the data? How do you use use visual techniques to give you insight to what the data is saying, and present those conclusions to others? Along the way, we'll talk about topics such as statistics, machine learning, really large databases, and baseball, and we'll look at industries as diverse as banking and biology.

Amongst the speakers are Mike Driscoll of Dataspora, who we've featured several times on this blog as an leading innovator of large-scale data analysis using the R language. Mike was also recently quoted in an article about Revolution and R in Forbes Magazine, and he recently published a most thought-provoking article about new tools for big-data analysis, and the processes needed to make the analysis itself help instead of hinder the decision-making process. Also speaking is Joe Adler from LinkedIn, author of the excellent R reference manual, R in a Nutshell. I'm not familiar with the work of the other two speakers, Hilary Mason (Bit.ly) and Ben Fry (Ben Fry LLC), but the abstracts for their talks on (respectively) machine learning and visualization look very interesting. 

Best of all, it's an on-line conference, so you don't even need to leave your desk to take part! Registration is just US$149, but as a sponsor Revolution Analytics has passes to give away -- all you need to do is follow RevolutionR on Twitter for a chance to win one of three passes on Monday. Update: Winners will be selected at random from RevolutionR's followers at 6PM PDT on Monday and notified via DM.

O'Reilly: Making Data Work Online Conference

Map-reduce in R with Amazon EMR

There's new video at the Video Rchive[*] of a JD Long presentation: Amazon Elastic Map-Reduce (So Easy an Economist Can Do It). I'm on a plane right now and the in-flight wi-fi isn't quite up to playing the 30-minute video, but judging from the slides it looks like a nice, practical demonstration of getting a map-reduce computation programmed in R running on the Amazon cloud system. The code also appears quite complete: in a clever innovation, the commands to launch the Amazon instances run directly from the R code, which should eliminate a lot of the tedious command-line monkeying that typically accompanies an Amazon EC2 session. I look forward to trying it out when I'm landside.

[* Revolution Analytics sponsors the Video Rchive, which is provided as a service for the community by Drew Conway. Thanks, Drew! You can donate to the Video Rchive too, see the link in the right column of Drew's blog.]

VCASMO/JD Long: Map-reduce in R with Amazon EMR

Prediction in the cloud: turbulent

While Microsoft rolled out its Technical Computing Initiative -- promising new tools for distributed parallel computing on large data sets in the cloud -- with much fanfare earlier this week, Google made a rather more understated response. In a post to the developer-focused Google Code Blog, they quietly announced two new, but potentially disruptive, products. Google BigQuery promises super-fast SQL-like queries on massive data sets (provided the data has been uploaded to a Google host). And the Google Prediction API apparently offers a machine learning black-box: pump some (again, Google-hosted) data in, and "advanced machine learning algorithms" will then give a service that predicts outcomes from new data. Presumably, it does something like using a subset of the data to train a number of machine learning algorithms (or an ensemble of them) and then chooses the best one for prediction. A little like the (now-discontinued) machine-learning service based on R, but obviously at Google scale. The Google products, for now, are only available to invitees as part of Google Labs, but it's certainly an interesting development: essentially making machine learning a commodity. If it works (and if the intended business users don't mind having their private data hosted by Google), it could certainly add some turbulence to the nascent market of machine learning in the cloud. I'll be following tomorrow's keynote on the topic (via Google Wave) with much interest.

Google Code blog: BigQuery and Prediction API: Get more from your data with Google

Parallel Computing with R for Life Sciences

I hadn't heard of the CloudAsia 2010 conference before, but from the programme the workshop Master Class on HPC Application For Life Sciences looked like it was interesting. One workshop session in particular caught my eye: Practical Parallel Computing in R by Xie Chao and Tan Tin Wee (from the National University of Singapore). The workshop notes (PDF) provide a practical and concise overview of the parallel programming options in R, including several examples using our own foreach package under the heading "One Ring to Rule Them All":

The foreach and iterators packages created by REvolution Computing [now Revolution Analytics -- ed.] provide us a convenient framework for parallel computing in R. With these two packages, you only need to write one version of your code for all parallel backends.

The document gives examples of using several parallel backends with foreach, including doMC, doSNOW and doMPI. 

By the way, If you're planning to try out foreach yourself, don't forget that a new parallel backend for Windows is now available: doSMP, included with all editions of Revolution R. doSMP is an open-source R package, we're haven't yet submitted it to CRAN. But if you want to get in early and try it with R 2.11, Tal Galili has compiled it and made a Windows binary compatible with R 2.11 at his R Statistics blog (thanks, Tal!).

CloudAsia 2010: Master Class on HPC Applications For Life Sciences

Slides from High-Performance Analytics webinar now available

Thanks to everyone who attended the webinar I presented this morning, High-Performance Analytics with REvolution R and Windows HPC Server. My slides are now available for download at the link below; even if you're not using Windows, I hope the slides are a useful introduction to the foreach parallel programming construct in general. If you do use R on Windows, there's some new material on using the new doSMP package (included in current versions of REvolution R Community and REvolution R Enterprise) for local parallel processing on multi-core workstations, and a guide for distributing those computation to a Windows HPC Server cluster using the doNWS package (included with REvolution R Enterprise).

In addition to the slides, a replay of the webinar with audio (in Windows WMV format[*]) will appear in the next couple of days at the link below.

REvolution Events: High-Performance Analytics with REvolution R and Windows HPC Server

[*] Apologies to folks not on Windows for whom that format is a pain - it's an artifact of the NetMeeting software we used for the webinar, which also caused audio problems for some viewers. We'll return to using GoToWebinar (with cross-platform support) for future webinars.

Webinar: High-Performance Analytics with R and Microsoft HPC Server

On April 14 I'll be giving a new webinar in partnership with Microsoft on High-Performance Computing with R. I'll be focusing on the new parallel programming capabilities of REvolution R Enterprise 3.1 for Windows, and how to use the features of Microsoft HPC Server to enable computing on clusters. Here's the complete agenda, and you can register at the link below.

Statistical data analysis is a key part of the operations of just about every business today. But as data sets get larger, analyzing trends or generating predictions becomes more and more of a challenge.

If you're doing predictive modeling today and find that you can no longer use all of your data because of size limitations, or the computations are taking too long for you to take action on the results, then the parallel-processing capabilities of Windows HPC Server can help. In this webinar, we'll introduce the R language for statistical computing, and show how the easy-to-use parallel programming capabilities of REvolution R Enterprise work with a HPC cluster to cut processing times by an order of magnitude or more.

We'll give practical examples of how to speed up many kinds of analytic computations, from simple summary statistics to cutting-edge tools like ensemble predictive models.

Audience: Programmers, researchers and analysts who need to process large volumes of data for data mining, statistical analysis, or predictive analytics

REvolution events: High-Performance Analytics with REvolution R and Windows HPC Server

Speeding up simulations with Amazon EC2

Over at Cerebral Mastication, JD Long tells a characteristically entertaining and informative story about how he uses R to run stochastic simulations of insurance portfolios and reinsurance treaties. A typical job involves 10,000 simulations, and when each estimate takes over 20 seconds you're talking some serious time to get the job done. 

Fortunately, this is the kind of problem that lends itself to parallelization: with a multiprocessor machine running R, you can run as many simulations as you have processors at the same time, and cut the total computation time by the same factor. JD describes how he boots up an 8-core instance running R on Amazon EC2, and then uses the multicore package on Linux to reduce the computation time by a factor of 8.

Sadly for Windows users, multicore doesn't run on Windows. But as Boris Shor points out in a comment to JD's post, REvolution R Enterprise allows you to do parallel programming like this (using the foreach operator) on a multiprocessor Windows machine (or even a cluster of many machines).

Cerebral Mastication: Using the R multicore package in Linux with wild an passionate abandon

According to Microsoft, the fourth paradigm of science is data

In scientific discovery, the first three paradigms were experimental, theoretical and (more recently) computational science. A new book of essays published by Microsoft (and available for free download -- kudos, MS!) argues that a fourth paradigm of scientific discovery is at hand: the analysis of massive data sets. The book is dedicated to the late Microsoft researcher Dr Jim Gray, who pioneered the idea with the catchphrase: "It's the data, stupid". The basic idea is that our capacity for collecting scientific data has far outstripped our present capacity to analyze it, and so our focus should be on developing technologies that will make sense of this "Deluge of Data" (as this New York Times review of the book -- well worth a read -- calls it). 

Dr Gray's call-to-arms was not to develop isolated super-powerful super-computers but “to have a world in which all of the science literature is online, all of the science data is online, and they interoperate with each other.” This dream is already close to a reality in some scientific domains like astronomy, where advanced instruments routinely generate petabytes of data available for public analysis. And with further developments in distributed and high-performance computing, with freely-available high-scale data management tools like Hadoop, and with advanced open-source data-analysis tools like R rapidly adapting to the scales of these data sets, the fourth paradigm is certain to become a mainstream reality in other scientific domains as well. 

Microsoft Research: The Fourth Paradigm: Data-Intensive Scientific Discovery