Revolutions

Information Age: graduates driving industry adoption of R

Information Age recently published a feature article devoted to the R language, "Putting the R in analytics". Says author Pete Swabey:

Already popular in universities, there are signs that R is finding increasing adoption in the enterprise. This promises to lower the barriers of entry for advanced analytics, and may accelerate the mathemitisation of business management.

The article includes an overview of the history of R: its predecessor, the S language; the transition to open-source R; the pervasiveness of R in academia; and how this is driving an increasing rate of adoption in industry.

This popularity in academia means that R is being taught to statistics students, says Matthew Aldridge, co-founder of UK- based data analysis consultancy Mango Solutions. “We're seeing a lot of academic departments using R, versus SPSS which was what they always used to teach at university,” he says. “That means a lot of students are coming out with R skills.”

Finance and accounting advisory Deloitte, which uses R for various statistical analyses and to visualise data for presentations, has found this to be the case. “Many of the analytical hires coming out of school now have more experience with R than with SAS and SPSS, which was not the case years ago,” says Michael Petrillo, a senior project lead at Deloitte's New York branch.

Like many companies today, Deloitte needs to develop analytics with varied and large quantities of data, and is using Revolution R Enterprise for big-data analytics and API integration of R:

Deloitte is currently preparing a big data pilot using Revolution Analytics’ enhanced R product. “We are using the server-based version of Revolution R to investigate big data analysis capabilities,” says Petrillo.

“We are looking at integration options to [big data programming platform] Hadoop, as well as ability to integrate R code into other applications via a web services framework.”

I'm also quoted in the article discussing the big-data extensions of Revolution R Enterprise:

Smith argues that these enhancements are necessary if R is to be applied to ‘big data’, i.e. data whose volume, velocity and variability outstrip the capabilities of conventional relational databases.

For more on the history of R and its applications in business, read the complete article in Information Age at the link below.

Information Age: Putting the R in Analytics

Because it's Friday: Five musicians, one guitar

Check out this cover of Goyte's "Somebody That I Used to Know", played on a single guitar by all five members of Walk Off The Earth:

  

Incidentally, the Gotye original version is the first song by an Australian artist to reach #1 on the US charts since Savage Garden in 2000. There's so much good Australian music it's a shame more of it doesn't make it to the charts over here in the US. But this is still a great song, so enjoy it as a kick-off to your weekend!

R equivalents to SAS and SPSS procedures

With more than 5,000 R packages now available (from the CRAN and BioConductor repositories), for any statistical or data analysis procedure you can confidently say, "there's a package for that". To make it easier for SAS and SPSS users to find what they need in R, Bob Muenchen has updated his useful table of equivalent R packages for SAS and SPSS add-ons. For example, if you're doing conjoint analysis, the equivalent packages are:

Conjoint Analysis

• SAS/STAT: PROC TRANSREG
• IBM SPSS Conjoint
• R: homals, psychoR, bayesm

Other categories of analysis include Operations Research, Power Analysis, Correspondence Analysis,& Structural Equation Modeling, Text Mining, Association Analysis , and many more. Note this only covers methods included in add-ons to SAS and SPSS, so some base-level functionality like survival analysis and mixed-effects modeling (also available for R), aren't listed here. 

For the full list of R equivalents to SAS and SPSS add-ons, follow the link below.

r4stats.com: Add-ons

A concise history of Data Science

The term "Data Science" has been around longer than you might think. Gil Press has researched the evolution of the use of the term, from the first definition by Peter Naur in 1974, to Bill Cleveland's 2001 paper, and Drew Conway's "Data Science Venn Diagram", introduced in 2010. As history demonstrates, use the term has really picked up in the last couple of years, but I was surprised to learn that it's been around as a term for more than thirty years. You can read all of the major events that have shaped the concept of "Data Science" at the post below. 

What's The Big Data?: A Very Short History of Data Science

 

Big Data statistics in the search for a cure for MS

Multiple Sclerosis (MS) is a debilitating and complex disease with an unknown cause — and for which there is currently no cure. The SUNY Buffalo is home to one of the leading multiple sclerosis (MS) research centers in the world, and as reported in Healthcare IT News, the research team is using IBM Netezza and Revolution R Enterprise to accelerate its study of more than 2,000 genetic and environmental factors that may contribute to MS symptoms:

Shawn Dolley, vice president and GM of global healthcare and life science at IBM Big Data (he worked for Marlborough, Mass.-based Netezza before IBM's acquisition of that company in 2010), remembers when Harvard Medical School researchers told him not long ago that, when it came to solving problems with big data analytics, "We have four questions we can ask per month." Why four? "Well, there's four weekends per month," he recalls them saying. "That's how long it takes us to run our jobs."

Now, using an IBM Netezza analytics appliance, in conjunction with software from Revolution Analytics, researchers can analyze disparate data in a matter of minutes instead of days, regardless of what type or size it is, say IBM officials. The technology automatically consumes and analyzes the data, and makes the results available for further analysis, leaving researchers more time to analyze trends. 

As described in this case study, SUNY Buffalo researchers were able to accelerate their research by reducing computation times more than 100-fold in the search for significant interactions between thousands of genetic and environmental factors. The R language itself also accelerates the research process, by making it easier to include new kinds of variables such as discrete dependent, Poisson dependent or continuous normally-distributed variables by simply adding a few lines of code. (Using another language would have required a re-writte of the entire algorithm.)

From an analytics perspective, the researchers were able to write a version of their software tools in in Revolution R Enterprise for IBM Netezza, so that all their reporting as well as our analysis is all consolidated in one place. This prevents the need to move very large amounts of data in and out of Netezza, which would cause further delays. They were also able to use a wider variety of data sets. 

The SUNY researchers were able to: 

• Use the new algorithms and add multiple variables that before, were nearly impossible to achieve. 
• Reduce the time required to conduct analysis from 27.2 hours without Netezza to 11.7 minutes with it. 
• Carry out their research with little to no database administration (Unlike other HPC platforms or databases available, Netezza was designed to require a minimum amount of maintenance)
• Publish multiple articles in scientific journals, with more in process.
• Proceed with studies based on ‘vector phenotypes’—a more complex variable that will further push the Netezza platform.

You can learn more about the research by the SUNY Buffalo team in this IBM Netezza blog post, and in the IBM press release at the link below.

IBM Press Releases: Leading Global Multiple Sclerosis Research Center Taps IBM Analytics to Improve Patient Care

20 free R tutorials (and one reference card)

If you're just getting started with the R language, R user Pairach Piboonrungroj has published a handly list of 20 free R tutorials published by university departments. Included on the list: Getting Started with the R Data Analysis Package (by Norm Matloff), Getting Started with R (from York University), and An Introduction to R (by Phil Spector). Paraich also points to a concise one-page reference card of common R commands, which may be useful to print out and keep by your keyboard as you're learning R.

See the complete list of 20 free R tutorials at the link below.

Pairach Piboonrungroj: R Tutorials from Universities Around the World

Rmetrics financial engineering workshop

For those looking for an in-depth workshop on financial engineering with R, look no further than the R/Rmetrics Workshop and Summer School held annually in beautiful Meielisalp, Switzerland. This is an intimate workshop limited to around 50 participants, and features tutorials from leading practitioners in finance with R. This year's workshop takes plase June 24-28. You can find the announcement from the Rmetrics Foundation below, and you can register for the workshop here:

Rmetrics Foundation: Registration: Meielisalp 2012

Puzzle: A path through pairs making squares

Ted Harding posed an interesting puzzle challenge on the r-help mailing list recently. Here's the puzzle:

Take the numbers 1, 2, 3, etc. up to 17.

Can you write out all seventeen numbers in a line so that every pair of numbers that are next to each other, adds up to give a square number?

You can figure out the solution from first principles fairly easily (hint: what neighbours must 17 have?), but Ted's challenge was to write a "neat" R function to solve this puzzle programmatically. The r-help thread generated several solutions (including an elegant one based on recursion, and a generalization to larger sets of numbers), but I wanted to highlight this solution from Vincent Zoonekynd on StackOverflow:

# Allowable pairs form a graph
p <- outer(
  1:17, 1:17, 
  function(u,v) round(sqrt(u + v),6) == floor(sqrt(u+v)) ) 
)
rownames(p) <- colnames(p) <- 1:17
image(p, col=c(0,1)) 
 
# Read the solution on the plot
library(igraph)
g <- graph.adjacency(p, "undirected")
V(g)$label <- V(g)$name 
plot(g, layout=layout.fruchterman.reingold)

It's a clever use of the graph theory. Consider the numbers 1-17 as vertices in a graph, with connections between pairs defined by whether they sum to a square. The call to outer defines the T/F adjacency matrix (which the code displays as an image), and then the plot.igraph function displays the graph as an R chart:

All you need to do is to trace a path that passes through each number once, and you have your solution to the puzzle. A neat and unexpected use of the igraph package for handling graphs in R.

StackOverflow: Ordering 1:17 by perfect square pairs

Because it's Friday: Pharoah's Serpent

When Mercury thiocynate is heated, it undergoes an exothermic reaction with surprising results:

 

This is actually a centuries-old fireworks effect, the Pharaoh's Serpent. But because of the toxic gases produced during the reaction, it is rarely used today. Still, it's cool to see.

Revolution Newsletter: April 2012

The most recent edition of the Revolution Newsletter is out. The news section is below, and you can read the full April edition (with highlights from this blog and community events) online. You can subscribe to the Revolution Newsletter to get it monthly via email.

Spring Webinar Series. Our Spring Webinar Series features presentations from Revolution Analytics staff and thought leaders in predictive analytics and Big Data. Don't miss webinars coming soon on Big Data with R and Hadoop;integrating R with MS Office, spatial statistics with R, data mining with R, retail marketing analytics, a graduate certificate program in R, and much more. See our webinar schedule for full details. Registration is free, but attendance is limited -- sign up soon to claim your seat for the live presentation.

Using Revolution R Enterprise with the IBM Netezza appliance, you can build powerful predictive models for Big Data using using Naive Bayes and Classification Trees, as shown in this demonstration video.

Revolution Analytics' Senior Advisor for Products and Strategy Norman Nie shares Five Benefits of Big Analyticsin the 2012 SAP Sybase Capital Markets Guide.

If you missed any of our recent webinars, catch our on-demand replays to see: how R was used to analyze when banks should modify distressed home loans; how R is used for actuarial analysis at Deloitte Consulting; and how to make model results more impactful.

Revolution Analytics is proud to sponsor R/Finance 2012, the premier conference devoted to applied financial analysis using R. Register now for two-days of in-depth presentations May 11-12 in Chicago, including David Humke (Northern Trust) who will speak on Operational Risk Measurement with R and RevoScaleR.

The annual Future of Open Source Survey (sponsored by the 451 Group, Revolution Analytics and other open-source firms) tracks opinions and trends on the use of open-source software in commercial organizations. Please take 5 minutes to provide your input. The conclusions from the 2011 survey are available here.

Revolution Analytics: Newsletter Archive