Revolutions

Orbitz: R has become the data-mining tool of choice

Sameer Chopra, vice president of Advanced Analytics at Orbitz Worldwide, wrote recently in Analytics magazine about the changing landscape of processes, software and systems for statistical modelers. In a section on "Big Data and Open Source Analytics", Chopra lays out the reasons why the R language "has become the data-mining tool of choice for machine learners":

• R has very good integration with Hadoop, an area where established commercial statistical tools have frankly been playing catch-up over the past year. (Note: At the time of this writing, some established statistical solution providers were announcing an access interface to Hadoop.)
• Many startups and smaller firms do not have deep pockets and are embracing open source tools such as the R programming language and NoSQL database systems such as MongoDB.
• R is a leading language for developing new statistical methods, and it is a platform for statistical innovation and collaboration across both the corporate world and academia. In my opinion, for the first time in years, the stronghold of established commercial players seems to be potentially threatened; open source tools are better suited for Big Data and will slowly but surely continue to take share away from commercialized statistical packages. In fact, traditional statistical vendors have recognized that R is a force to be reckoned with. In response, many of these vendors have developed hooks into R so users can interface with the R language.
• Based on the resumes I’ve been reading, the next generation of data miners is flocking to R as their go-to tool. Professors in general are comfortable with R; they tend to use R and Excel as part of their curriculum.
• In short, open-source analytics tools and platforms have arrived.

Chopra says that the usage of R in the commercial sector is growing "as firms such as Revolution Analytics focus on the enterprise capabilities for R" (for example, Revolution R Enterprise's Hadoop support and enterprise deployment).

Chopra also has some interesting perspectives on statistical modeling vs machine learning which you can find in the full article linked below.

Analytics magazine: The times they are a changin’ for advanced analytics

Thursday: Tweet-chat on Multiple Sclerosis research

The story about the great work that SUNY Buffalo has been doing to find a cure for Multiple Sclerosis with Revolution R Enterprise and IBM Netezza has generated a lot of attention, with stories in Forbes, InformationWeek and eWeek (amongst others). To continue the discussion, IBM has put together a panel for a "Tweet Chat" on Thursday (May 10) at noon Eastern time. I'll be there with the lead researcher Murali Ramanathan, Shawn Dolley from IBM and Tim Coetzee from the National MS Society. We'll be chatting about the search for a cure for MS, Big Data and analytics, in snippets of 140 characters or less. Join the conversation by following the #IBMDataChat hashtag at twitter.com or in your favourite Twitter client.

When: Thursday, May 10, 2012 from 12:00-1:00 p.m. ET (find your local time)
Where: Twitter hashtag #IBMDataChat
Who: 

• Shawn Dolley, IBM VP of Big Data Healthcare & Life Sciences (Moderator), @shawndolley
• Dr. Murali Ramanathan, SUNY Buffalo Professor Pharmaceutical Sciences and Neurology, Director of Graduate Studies & Co-Director, Data Intensive Discovery Initiative,@M_Ramanathan
• David Smith, Revolution Analytics R Evangelist & VP of Marketing, @revodavid
• Tim Coetzee, National MS Society Chief Research Officer, @tim_coetzee

Yes, you need more than just R for Big Data Analytics

Douglas Merrill, former CIO/VP of Engineering at Google, writes in Forbes about using the R language for data analysis:

Most folks with math-oriented graduate degrees will have written something in R, a non-commercial option for your big data analysis.  So, great graduates from great graduate schools know great tools.

His post is titled 'R Is Not Enough For "Big Data"', and you might be surprised to learn that I agree that title, although for a different reason. Douglas's point — and it's a valid one — is that simply pumping data through any software tool, without an understanding of the problem you're trying to solve and how statistical models apply to it, can lead to getting the wrong answers to the wrong questions:

If you ask the wrong question, you will be able to find statistics that give answers that are simply wrong (or, at best, misleading).

On net, having a degree in math, economics, AI, etc., isn’t enough. Tool expertise isn’t enough.  You need experience in solving real world problems, because there are a lot of importat limitations to the statistics that you learned in school.  Big data isn’t about bits, it’s about talent.

This is a great illustration of why the data science process is a valuable one for extracting information from Big Data, because it combines tool expertise with statistical expertise and the domain expertise required to understand the problem and the data applicable to it. He's right that you need data science talent and software to solve problems with Big Data ... and having software like R that supports the exploratory nature of the data science process is also critical.

But I also agree with the title for a different, technical reason: the R software is just one piece of software ecosystem — an analytics stack, if you will — of tools used to analyze Big Data. For one thing R isn't a data store in its own right: you also need a data layer where R can access structured and unstructured data for analysis. (For example, see how you can use R to extract data from Hadoop in the slides from today's webinar by Antonio Piccolboni.) At the analytics layer, you need statistical algorithms that work with Big Data, like those in Revolution R Enterprise. And at the presentation layer, you need the ability to embed the results of the analysis in reports, BI tools, or data apps.

So yes, Douglas is right: you need more than just R for Big Data. You also need a data layer, an analytics layer, and a presentation layer (all of which supports Big Data) ... and you need Data Science skills to make sure you're asking the right questions and getting appropriate answers.

Forbes: R Is Not Enough For "Big Data"

Information Age: graduates driving industry adoption of R

Information Age recently published a feature article devoted to the R language, "Putting the R in analytics". Says author Pete Swabey:

Already popular in universities, there are signs that R is finding increasing adoption in the enterprise. This promises to lower the barriers of entry for advanced analytics, and may accelerate the mathemitisation of business management.

The article includes an overview of the history of R: its predecessor, the S language; the transition to open-source R; the pervasiveness of R in academia; and how this is driving an increasing rate of adoption in industry.

This popularity in academia means that R is being taught to statistics students, says Matthew Aldridge, co-founder of UK- based data analysis consultancy Mango Solutions. “We're seeing a lot of academic departments using R, versus SPSS which was what they always used to teach at university,” he says. “That means a lot of students are coming out with R skills.”

Finance and accounting advisory Deloitte, which uses R for various statistical analyses and to visualise data for presentations, has found this to be the case. “Many of the analytical hires coming out of school now have more experience with R than with SAS and SPSS, which was not the case years ago,” says Michael Petrillo, a senior project lead at Deloitte's New York branch.

Like many companies today, Deloitte needs to develop analytics with varied and large quantities of data, and is using Revolution R Enterprise for big-data analytics and API integration of R:

Deloitte is currently preparing a big data pilot using Revolution Analytics’ enhanced R product. “We are using the server-based version of Revolution R to investigate big data analysis capabilities,” says Petrillo.

“We are looking at integration options to [big data programming platform] Hadoop, as well as ability to integrate R code into other applications via a web services framework.”

I'm also quoted in the article discussing the big-data extensions of Revolution R Enterprise:

Smith argues that these enhancements are necessary if R is to be applied to ‘big data’, i.e. data whose volume, velocity and variability outstrip the capabilities of conventional relational databases.

For more on the history of R and its applications in business, read the complete article in Information Age at the link below.

Information Age: Putting the R in Analytics

Big Data statistics in the search for a cure for MS

Multiple Sclerosis (MS) is a debilitating and complex disease with an unknown cause — and for which there is currently no cure. The SUNY Buffalo is home to one of the leading multiple sclerosis (MS) research centers in the world, and as reported in Healthcare IT News, the research team is using IBM Netezza and Revolution R Enterprise to accelerate its study of more than 2,000 genetic and environmental factors that may contribute to MS symptoms:

Shawn Dolley, vice president and GM of global healthcare and life science at IBM Big Data (he worked for Marlborough, Mass.-based Netezza before IBM's acquisition of that company in 2010), remembers when Harvard Medical School researchers told him not long ago that, when it came to solving problems with big data analytics, "We have four questions we can ask per month." Why four? "Well, there's four weekends per month," he recalls them saying. "That's how long it takes us to run our jobs."

Now, using an IBM Netezza analytics appliance, in conjunction with software from Revolution Analytics, researchers can analyze disparate data in a matter of minutes instead of days, regardless of what type or size it is, say IBM officials. The technology automatically consumes and analyzes the data, and makes the results available for further analysis, leaving researchers more time to analyze trends. 

As described in this case study, SUNY Buffalo researchers were able to accelerate their research by reducing computation times more than 100-fold in the search for significant interactions between thousands of genetic and environmental factors. The R language itself also accelerates the research process, by making it easier to include new kinds of variables such as discrete dependent, Poisson dependent or continuous normally-distributed variables by simply adding a few lines of code. (Using another language would have required a re-writte of the entire algorithm.)

From an analytics perspective, the researchers were able to write a version of their software tools in in Revolution R Enterprise for IBM Netezza, so that all their reporting as well as our analysis is all consolidated in one place. This prevents the need to move very large amounts of data in and out of Netezza, which would cause further delays. They were also able to use a wider variety of data sets. 

The SUNY researchers were able to: 

• Use the new algorithms and add multiple variables that before, were nearly impossible to achieve. 
• Reduce the time required to conduct analysis from 27.2 hours without Netezza to 11.7 minutes with it. 
• Carry out their research with little to no database administration (Unlike other HPC platforms or databases available, Netezza was designed to require a minimum amount of maintenance)
• Publish multiple articles in scientific journals, with more in process.
• Proceed with studies based on ‘vector phenotypes’—a more complex variable that will further push the Netezza platform.

You can learn more about the research by the SUNY Buffalo team in this IBM Netezza blog post, and in the IBM press release at the link below.

IBM Press Releases: Leading Global Multiple Sclerosis Research Center Taps IBM Analytics to Improve Patient Care

R at the Consumer Financial Protection Bureau

The O'Reilly Radar blog has a lengthy and very interesting interview with the lead and deputy CIOs of the Consumer Financial Protection Bureau, the new US government agency devoted to consumer protections in the financial markets. In that interview, they talk about the many open-source tools used in the agency (and the parent Treasury Department): Linux, WordPress, Splunk, Django, Git and  yes -- R for data analysis with Big Data. (The CFPB has also made public its policy on use of open-source software.) They reveal an interesting cultural shift related to the use of R in this question at the end of the interview:

How do you see open source contributing to your ability to get insights from large amounts of data? If you're recruiting developers, can they actually make a difference in helping their fellow citizens?

[...] R is an interesting example. What we're finding is that as more people are coming out of academia into the professional world, they're actually used to using R in school. And then they have to come out and learn a different tool and they're actually working in the marketplace.

It's similar with the Mac versus the PC. You get people using the Mac in college — and suddenly they have to go to a Windows interface. Why impose that on them? If they're going to be extremely productive with a tool like R, why not allow that to be used?

Here at Revolution Analytics, we're hearing much the same story from commercial organizations who have data analysis platforms build around legacy platforms like SAS. When all of the new recruits coming out of school are already trained in R, the choice is between spending a year training the up in SAS (and incurring both the expense and the opportunity cost of a year's work in the process), or setting up a platform based around a commercially-supported R distribution. And more and more often, that second choice is the more compelling one.

O'Reilly Radar: Open source is interoperable with smarter government at the CFPB

R, Twitter and McDonald's

Ed Chen is a data scientist at Twitter, so he's accustomed to working with big data and complex models. In an interview with MIT Technology Review, he describes his data science toolbox:

A common pattern for me is that I'll code a MapReduce job in Scala, do some simple command-line munging on the results, pass the data into Python or R for further analysis, pull from a database to grab some extra fields, and so on, often integrating what I find into some machine learning models in the end.

He put this toolbox to great use in a recent blog post, Infinite Mixture Models with Nonparametric Bayes and the Dirichlet Process. After using simulation in Ruby and Python to generate some test data, he used the R language to create a novel classification model which can group "similar" members from a data set, without needing to specify the number of groups in advance. He used this model to categorize the McDonald's member into a number of "food groups" containing products with similar nutritional content.

One cluster contained all the desserts: Baked Hot Apple Pie, Snack Size McFlurry, and others including the three below: 

As you can see, the foods in the "dessert" group cluster together because of high trans fat content, low fiber, and other similar nutritional attributes. Other groups identified by the model include salads, burgers and other fried food, three categories of sauces, and (in a cluster all on its own) Fruit and Maple Oatmeal: the only high-fibre item on the menu.

For an astounding amount of detail about the analysis, visit Ed Chen's blog at the link below.

Edwin Chen's Blog: Infinite Mixture Models with Nonparametric Bayes and the Dirichlet Process

SAP integrates R with HANA

We can add SAP to the list of vendors offering R integration with their products. InformationWeek reports that the new SAP BusinessObjects Predictive Analysis model provides a graphical user interface to R. Created in reaction to "competitive and market forces, including the momentum of open source R", the new module provides in-database processing (presumably by embedding R within HANA in-memory database appliance, as described in this SAP presentation), and by offering "mainstream analytics" (presumably in-built analytic methods, like those included with the IBM Netezza appliance).

SAP BusinessObjects Predictive Analysis is currently available only to a limited set of clients and expected to be generally available later this year. 

InfoWorld: SAP Steps Up Commitment To Predictive Analytics

Big Data Analytics to Revolutionize Services

Revolution Analytics' CEO Dave Rich was interviewed by Wikibon's David Vellante and SiliconAngle's John Furrier at the Strata 2012 conference last week. Given Dave's background at Accenture Analytics, the conversation naturally turned to impact of predictive analytics and R on business services. (See the video after the jump, below.)

Bret Latmore of SiliconANGLE provides highlights of the interview, including Dave's assessment of the opportinity for R in the enterprise:  

[Big Data revolutionizing business processes] is the reason Rich chose to become CEO of Revolution Analytics, which he calls "the Red Hat of R", which is the next-generation analytical programming language that has replaced SAS and SPSS in academia. Like Linux, R was developed by an open source community of mostly academics as an advanced analytics language.

Now, he says, the legacy language owners are milking their markets. IBM, for instance, hiked the price of SPSS licenses after buying it. As a result, R is growing in commercial use – for instance Oracle recently announced that it was standardizing on R. "We thought that was great. The same thing happened with Linux. Legacy customers were tired of the high license fees [for the commercial operating systems], so they put Red Hat in business. We're at that same inflection point."

See the video after the jump, and read more of the interview at the link below.

ServicesAngle: Big Data Analytics to Revolutionize Services, Says Industry CEO

R integrated throughout the enterprise analytics stack

The past couple of years have seen a dramatic growth in the use of the R language in the enterprise. R has always been pervasive in academia for research and teaching in statistics and data science, and as new graduates trained in R have migrated to the workplace the demand for R in corporations has become more and more intense. 

Database vendor Oracle estimates that "R has attracted over two million users since its introduction". James Kobielus, noted Forrester analyst and predictive analytics expert, recently said in PCWorld that "R has become a real ubiquitous force in advanced analytics. It's everywhere. Enterprise adoption of it has been growing steadily. When we ask our customers what they're using for statistical modeling they'll say SAS or [IBM's] SPSS, but they increasingly say R in the same breath." 

With rapid growth of the use of R in the enterprise comes a corresponding increase in demand for enterprise support for R from its users, and demand for integration of R into corporate systems from IT (both areas in which Revolution Analytics provides software and expertise). Most large organizations have a sophisticated infrastructure devoted to data analysis, with an "analytics stack" of software to provide data warehousing and query, predictive analytics, reporting, presentation, and Business Intelligence (BI). As a result software vendors at every layer of this stack have added functionality to integrate R, accommodating demand from both users and IT, and to serve the needs of data-driven business decision makers. Let's take a look at some of the applications within the analytics stack which now provide integration with R. 

The Data Layer

The data layer is where the lifeblood of the analysis — the data — is stored and prepared. Especially for high-performance and big-data applications, analytics based in R can benefit from the infrastructure that the data layer provides. The IBM blog has a great post with an in-depth discussion about integrating R with the data layer.

IBM Netezza, the high-performance data-warehousing appliance, is integrated with R (in partnership with Revolution Analytics). R users can use Revolution R Enterprise to run massively-parallel computations in R within the IBM Netezza appliance and implement high-performance, big-data analytics (with high-frequency financial data, for example). [Update: a free webinar on February 29 will describe the integration between IBM Netezza and Revolution R Enterprise in detail.] 

Oracle announced last year a forthcoming connection between R and Oracle, which was made available in February 2012. Oracle R Enterprise is aimed at statisticians who are "don't know SQL" and are "not familiar with DBA tasks". It is available as part of the Oracle Advanced Analytics option (priced at around $23,000 per core), and provides a "transparency layer" for with functions to connect to Oracle  and use R functionality in the Oracle database. Oracle also maintains the open-source ROracle package which provides similar functionality for open-source R. 

Cloudera's Distribution Including Apache Hadoop provides support for R in partnership with Revolution Analytics. This connection makes it possible to manipulate Hadoop data stores in R directly from HDFS and HBASE, and give R programmers the ability to write MapReduce jobs in R using Hadoop Streaming.

IBM BigInsights, the Hadoop platform from IBM, is also integrated with R and Revolution R Enterprise. BigInsight queries can make use of the Map-Reduce construct while running R computations in parallel.

Teradata's Enterprise Data Warehousing platform provides in-database analytics using R via the free teradataR package. This package allows R users to connect to Teradata, create data frames linked to Teradata and to call in-database analytic functions. The Teradata Aster MapReduce Platform also provides integration with R. 

Sybase RAP, the edition of the Sybase database for financial data, provides integration with R. Providing the R language alongside Sybase RAP allows for faster algorithm development and extensive backward testing on historical data. Sybase also regularly highlights R integration in its financial newsletters and webinars.

SAP HANA provides in-database analytics based on R with its SAP BusinessObjects Predictive Analysis module.

The Analytics Layer

The Analytics layer is where the magic happens: statistical modeling, predictive analytics and custom data visualization. Fed with (usually) structured data sourced from the Data Layer, R is widely used here to categorize, predict, and generally provide insight into corporate data stores. In many organizations, older data analysis tools remain in use, and so interfaces to R have been added provide support for analysts and data scientists who prefer to use R and to fill in the gaps of these legacy tools with modern, high-performance analytics.

Revolution Analytics is the leading commercial organization focused on software and support for R. Its Revolution R Enterprise software extends open-source R with productivity interfaces, high-performance statistical computing, big-data analytics, and enterprise integration of R. 

SAS has been a statistical analysis workhorse since the early 70's. Now, with so many graduates in statistics trained in R instead of SAS, SAS has introduced the ability to call R from SAS/IML. (It's also possible to call R directly from base SAS thanks to a free package developed at Roche Pharmaceuticals.) SAS JMP, the point-and-click data analysis package, now also provides support for R.

IBM SPSS Statistics, the popular desktop data analysis software known simply as SPSS before being acquired by IBM in 2010, provides integration to R via the Statistics Programmability Extension module.

RStudio, an open-source software company, provides an integrated development environment for developing code in the R language. 

Matlab, a numerical computing language used by engineers, also offers the ability to call R from Matlab on Windows.

Zementis software allows models created with R to be scored on massive data sets using the ADAPA Decision Engine and Revolution R Enterprise.

The Presentation Layer

Data analysis makes the most impact in the enterprise when it can be readily acted upon by decision makers: often, business executives not steeped in the arcana of data warehousing or statistical analysis. As a result, many reporting and business intelligence tools now make it possible to make it possible to incorporate the resuts of analyses generated in R in the presentation layer, in a format tuned to the needs of a business audiecne.  

Jaspersoft's business intelligence software makes it possible to incorporate the results of R-based analytics into BI dashboards and reports, via integration with Revolution R Enterprise.

TIBCO Spotfire's interactive business intelligence dashboards make it possible to share results and models from R. 

You might not think of Microsoft Excel as more of a spreadsheet than a presentation tool, but it is very widely used on the desktop as a "container" for static and interactive reports based on statistical analysis. While Excel does not have out-of-the-box integration with R, is is possible to integrate R-based computation into Excel spreadsheets via RExcel and Revolution Analytics' RevoDeployR web services API.

R: Integrated throughout the analytics stack

As you can see, for organizations who need to create advanced analytics applications, R is integrated throughout the analytics stack: for data access, for presentation of results, and of course for the statistical analysis process itself. This degree of integration by so many companies is indicative of the level of demand for R throughout the enterprise. As the leading provider of commercial software and support for R, Revolution Analytics supports R users througout the organization, helps IT departments integrate Revolution R Enterprise throughout the analytics stack, for high-performance and big data applications based on the R language.

[Update Mar 15 2012: Added SAP HANA to the list.]

Revolution R Enterprise: production-grade analytics software built upon the powerful open source R statistics language.