Revolutions

Accelerating analytics at MSU with Revolution R Enterprise

Erik Sigur, Information Technologist for the Department of Statistics and Probability at Michigan State University, writes at ReadWriteWeb about using Revolution R Enterprise to provide high-performance computation in R to the researchers in his department:

Our search for a more effective version of R ultimately brought us to a product called Revolution R Enterprise by Revolution Analytics, which provides commercial support and software for open source R. It takes advantage of multiple processor cores by using optimized assembly code and efficient multi-threaded algorithms that use all of the processor cores simultaneously.

The department at MSU provides high-performance computing facilities via their Statistical Computing Cluster: a network of high-performance PCs running under the Microsoft HPC Server environment. Revolution R Enterprise 5 provides features to distribute R jobs amongst the nodes of a cluster, and to use the power of distributed computing to reduce the time required to process big-data statistical analyses. Says Erik:

Once the department could schedule R jobs in an HPC environment, the demand began to drastically increase. The HPC cluster is now scheduling more than four times the amount of jobs that were scheduled in previous semesters, from 200 jobs over a year ago to over 800 jobs this past semester. Jobs that were taking over three months to complete on open source R were completed in less than a few days with Revolution R. Computational jobs are now run multiple times with significantly higher levels of accuracy than ever before.

Learn more about Erik's experiences with Revolution R Enterprise at the link below. You can also read the case study on the Revolution Analytics website, or learn more about the HPC features of Revolution R Enterprise in our archived webinar.

ReadWriteEnterprise: [Case Study] Lessons in High Performance Computing with Open Source

R Chart featured in Facebook IPO

Page 7 of Facebook's 213-page S-1 filing for their record-breaking IPO includes the following chart, under the headline: "Our Mission: To make the world more open and connected".

This chart was created using the R language and Hadoop by Facebook intern Paul Butler. (Thanks to the blog IOER Tools for first noticing the inclusion of the chart.)

And speaking of R charts appearing in widely-read places, the latest issue of the prestigious science journal Nature includes several charts created in R in an article on the evolution of the nematode C. elegans. Hadley Wickham, author of the ggplot2 package, noted that his package was used to create one of the charts.

Analytic applications are built by data scientists

Ventana Research analyst David Menninger was on the judging panel for the Applications of R in Business contest. In a post on the Ventana research blog, he offers his perspectives on the contest, noting that

R, as a statistical package, includes many algorithms for predictive analytics, including regression, clustering, classification, text mining and other techniques. The contest submissions supported a variety of business cases, including, among others, predicting order amounts to optimize manufacturing processes,  predicting marketing campaign effectiveness to optimize marketing spending, predicting liquid steel temperatures to optimize steel plant processes and performing sentiment analysis of Twitter data.

(Incidentally, David also has a great riff on the terminology of "predictive analytics" and "big data" out today.) He also notes that these applications are compelling precisely because of the close relationship between the contest entrants and the business problems they demonstrated how to solve:

The entries also demonstrated a best practice: close alignment between the analyst and the underlying business objectives. Predictive analytics is not magic. It requires an understanding of business processes and an understanding of statistical techniques. The judging criteria reflected this requirement as well. One of the three categories we were asked to score was applicability of the submission to business. I think it’s clear how the analyses in the winning entries could provide significant business value.

As David notes, however, the counterpoint to this is that the analyst must combine *both* the . "How many people in your organization could perform those types of analyses", he rightly asks. A combination of statistical tools along with domain expertise (plus the technical skills to implement the solution) is the hallmark of a good data scientist, which exactly why many organizations are looking to build effective data science teams.

By the way, while the concept of "data scientist" is relatively new, this idea of combining statistical analysts with domain expertise is not. Bill Cleveland (yes, that Bill Cleveland) made similar suggestions in a prescient paper back in 2001: "Data Science: An Action Plan for Expanding the Technical Areas of the Field of Statistics". (ISI Review, 69)

David Menninger: Revolution Analytics Hosts Contest on Business Predicting the Future

Announcing the winners of the Applications of R in Business contest

It's with great pleasure that the team at Revolution Analytics announces the winners of the inaugural "Applications of R in Business" contest:

Grand Prize ($10,000): Shannon Terry and Ben Ogorek, Nationwide Insurance for 'A Direct Marketing In-flight Forecasting System'

Runner Up Prize ($5,000): Jeffrey Breen, Atmosphere Research Group for 'Mining Twitter for Airline Consumer Sentiment'

Congratulations to both winners, who were selected by a judging panel of industry experts, analysts, and R users. (The competition was also very close: there was only a coupe of points in the judges' aggregate scores separating the top two entries.) 

In addition, the following applications were selected for $1,000 Honorable Mention prizes by Revolution Analytics:

• Hakan Koç and Bengt Maas, Salzgitter Mannesmann Forschung GmbH: Towards an ideal steel plant - Online liquid steel temperature prediction using R
• Imre Kocsis, Kristóf Gáti, András Pataricza, and Gábor Horváth, Budapest University of Technology and Economics: Time Series Analysis and Order Prediction with R
• Stephen D. Simon, P.Mean Consulting / University of Missouri-Kansas City: Predict the duration of your clinical trial
• Shannon Terry and Ben Ogorek, Nationwide Insurance: Quantifying Uncertainty in IT Estimates
• Yihui Xie, PhD Candidate at Department of Statistics, Iowa State University: knitr: Elegant, flexible and fast dynamic report generation with R

You can learn more about all of the winning applications in the slideshow on the Applications of R in Business Contest Winners page, in the press release and in the reports in Read Write Web, Enterprise Apps Today and Information Management.

Thanks to everyone who participated in the contest. A special mention is due to Shannon Terry and Ben Ogorek, both from Customer Analytics within Nationwide Insurance. They have been gracious enough to donate both their grand prize and honorable mention prizes to charity Mid-Ohio Foodbank, a 501(c)(3) organization based in Columbus, OH and a close community partner with Nationwide Insurance, will receive $11,000 to support their efforts in hunger relief. What a wonderful contribution!

Revolution Analytics: Applications of R in Business Contest: 2012 Winners 

Visualize your Facebook friends network with R

A few days ago, Romain François explained how to interface with the Facebook Graph API explorer with R. This was a low-level interface, giving the R programmer the ability to the raw data that Facebook can provide about your connections. Now, just four days later, the first application in R (that I know of) based on the Facebook Graph API has been published.

Petr Simecek describes how to use R to visualize your Facebook friends as an interconnected graph. Petr uses Romain's facebook function to download the friends list via the API, and then constructs the friendship graph using the Rgraphviz package. To enhance the presentation, Petr goes one step further and downloads his friend's profile pictures (again, using the Facebook API and the download.file function), and uses the addlogo function (from the pixmap package) to decorate the nodes of the graph with faces. Here's the final result:

To find the R code to create a chart like this for your own Facebook friend network, follow the link to Petr's blog below.

Apply R: Mining Facebook Data: Most "Liked" Status and Friendship Network (via)

NYT uses R to map the 1%

Last Saturday, the New York Times published a feature article on the wealthiest 1% of Americans. The on-line version of the article included interactive features like this interactive map showing where your household ranks in the country and in local regions. The print edition, however, included some different (and necessarily static) representations of US wealth data, such as this map of where the wealthiest 1% live:

(Image credit: New York Times, January 14 2002.)

Kevin Quealy, graphics editor at the New York Times, used R to help create this chart. The process began with an initial proof of concept, which used R's maptools package and required five lines of R code. 

The original plan was to use ArcMap to generate the chart, but ArcView lacks the flexibility of styling the output that R has. Later iterations added color and algorithmically moved the labels around for legibility, and some final tweaking outside of R led to the chart in the print edition.

Interestingly, the chart doesn't appear in the on-line edition of the NYT. I knew that some features like interactive charts, were online only, but I wasn't aware that the reverse was also true, and that special content was made only for the print edition. Quealy points out that this is a necessity:

This is a good example, I think, of using each medium to its best potential, meeting the design constraints of each. More and more, this means making totally separate versions of things – admittedly, it frequently takes twice the time and energy – but the mediums are just so different that works well in one just doesn’t work well in another. 

For the full story of how this lovely chart was created, check out Kevin Quealy's tubmlr linked below.

chartsnthings: Before, During and After: The Richest 1 Percent

Vote Compass: visualizing Canadian poll results with R

Vote Compass is an online "electoral literacy application, whose goal is to encourage engagement with and stimulate discussion around the policy platforms of Canada's political parties. In the lead-up to the 2011 Canadian election, Vote Compass collected the results of an on-line 10-minute survey from more than 2 million participants, and used the results to align voters with the political party that best matched their stated political beliefs. (You can see an overview of the project in this YouTube video.)

Now, the Vote Compass team has used the R programming language to visualize the results of the survey in an attractive poster format. For each question, Gregory Eady and colleagues used the spplot function to draw a Canada-wide (and also city-by-city) choropleth of the results:

Also, the breakdown of the question responses by sub-group are also presented using the ggplot2 package:

It's certainly a visually pleasing and interesting way of exploring the results of this survey. One design point: at first, I wasn't sure what to think about using continuous lines for the sub-group charts above, since there were only five possible survey responses. (It seemed a bit odd to interpolate and smooth between "somewhat disagree" and "neither agree not disagree", for example.) But having explored the site a bit, I found these charts surprisingly useful to interpret the results: it's much easier to compare the subgroups with these overlaid "density" charts, than it is using the traditional stacked barchart or similar. And I guess preferences (from "strongly disagree" to "strongly agree") do lie along a continuum, even if only discrete survey responses are allowed.

Anyway, it's nice to see such clarity of vision and attention to detail in the presentation of data like this.

Vote Compass: 2011 Canadian Federal Election

Applications of R in Business Contest: Final Entries

The revision period for the Applications of R in Business Contest is now at a close, and the competitors have finalized their entries for a chance at $20,000 in prizes from Revolution Analytics. We're now in the judging phase, where the finalists will be rated on applicability to business, innovation and persuasiveness by an independent panel of judges from the analyst, industry and open source R communities.

The formal announcement will come in January, but in the meantime you can take a look at all the competition entries at: inside-r.org/category/tags/contest. Also, in the run-up to the announcement we'll be highlighting the finalists via Twitter: follow @inside_R for daily links to the submitted applications with practical examples of how R is used in finance, life science, manufacturing and other industries.

inside-R.org: Applications of R in Business Contest- Final Submissions

"Applications of R" contest submissions online

Thanks to everyone for participating in the "Applications of R in Business" contest. R users submitted more than 25 entries, describing how R is used in industries including life sciences, finance, manufacturing, sentiment analysis, and even sports. Some entries are just outlines for now (competitors have until November 30 to finalize their entries), but already there are some quite comprehensive entries such as these:

• Mining Twitter for Airline Consumer Sentiment
• Use R to perfect quarterly ministerial briefings on export
• Towards an ideal steel plant - Online liquid steel temperature prediction using R
• Quantifying Uncertainty in IT Estimates
• Automated Emailing of Sweave Reports for supporting Quality Improvement in Healthcare
• A Direct Marketing In-flight Forecasting System
• Time Series Analysis and Order Prediction with R
• Predict the duration of your clinical trial

We encourage feedback from everyone on all the competition entries. Not only will your comments help the competitors improve their entries, but comments will be taken into account by the judges when awarding the $20,000 prize pool provided by Revolution Analytics. If you think the entries you see represent a particularly useful application of R, and especially if it's something you could put to use in your own work, please do leave a comment in the space provided at the bottom of each entry.

You can see all of the competition entries (including the stubs to be fleshed out by the competitors) at the link below.

inside-R.org: Applications of R in Business Competition - Submissions

Building diversified portfolios with R

A common approach to reducing risk associated with financial portfolios is diversification. A portfolio made of components that are all highly correlated with each other -- a portfolio composed solely of financial stocks, for example -- is risky, because if there's a wide-spread crisis that affects the banking sector, all components of the portfolio will tank at once, together. This is bad. A way to avoid risks like this is to try and choose components that are as uncorrelated (or even anti-correlated) as possible: that way, if one sector tanks, the entire portfolio isn't brought down. 

The classical way to deal with this is problem is Markowitz mean-variance portfolio optimization: for a given level of risk (say, 12%), find the portfolio that maximizes the expected return, given the historic correlations between the different potential components (treasury bonds, equities, index funds, commodities, etc). Choosing a higher or lower level of risk will result in a different mix of components: generally more on the equities side for the higher risk levels, more in treasuries for the lower risk levels.

Portfolio managers often set constraints on the amount of stocks to be allocated specific sectors (say, 20% in finance equities and 10% in municipal bonds). Given those constraints, the classical mean-variance optimization process can still be used, but the set of solutions is constrained to those portfolios that meet the sector allocations. Nonetheless, the individual assets in those sectors are still considered independently in the optimization process.

A recent paper suggests a better approach might be to minimize not overall risk, but instead the average correlation of the components within each sector. The Systematic Investor blog shows that it's easy to implement a criterion like this in the R language:

portfolio.sigma = sqrt( t(weight) %*% assets.cov %*% weight )
mean( ( weight %*% assets.cov ) / ( assets.sigma * portfolio.sigma ) )

You can then use one of R's nonlinear solvers -- they use Rdonlp2 -- to maximize the equations and return the optimal portfolios for different levels of risk. (The R code to do this is available at github.) Here are their results for standard mean-variance portfolios (at the top), and minimum average correlation portfolios at the bottom:

In each case, read the vertical line above a given level of risk to see how the optimal portfolio is allocated. At the lower risk levels, the average-correlation portfolio includes gold (GLD) and 20-year treasuries (TLT); at higher risk levels emerging markets securities (EEM) get mixed in as well.

For the full details of average-correlation portfolios and their implementation in R, see the blog post at Systematic Investor linked below.

Systematic Investor: The Most Diversified or The Least Correlated Efficient Frontier