Revolutions

R with Hadoop and NoSQL on ComputerWorld Cover

R, Hadoop and SQL are the focus of the cover story in the May 2012 issue of ComputerWorld Canada:

Brian Bloom's article The Open-Source Answer to Big Data (also published in PCWorld) looks at the impact of open-source on enterprise IT architectures:

Cost, flexibility and the availability of trained personnel are major reasons for the open-source boom. Hadoop, R and NoSQL are now the supporting pillars of many enterprises' big data strategies, whether they involve managing unstructured data or performing complex statistical analyses on it.

...

[Former Forrester analyst] Kobielus wrote, platforms like Hadoop, R and NoSQL have enjoyed an advantage over proprietary software because they were able to evolve faster. They're also being continuously developed and refined by many different parties. 

The article also looks at the influence of data scientists in large companies (and includes a quote from me):

Not surprisingly, the most brilliant minds with backgrounds in statistical modeling are also in the highest demand, especially since organizations in other sectors, like financial institutions, are scooping them up.

"They've hired a bunch of people out of school to a data science department or an R&D department and a modeling department," says Smith, "and they've found that all of them have been trained in R, and not in, say SAS."

The entire article is a great read, and you can find it at the link below.

ITWorld Canada: The open-source answer to big data

 

R Tops Data Mining Software Poll

For the past 12 years, KDNuggets has conducted an annual poll asking "What analytics/data mining software you used in the past 12 months for a real project (not just evaluation)". In this year's poll, R was the top-ranked data mining solution, selected by 30.7% of poll respondents. Microsoft Excel was second, at 29.8%. Rapidminer, which took the #1 spot over R in 2011 and 2010, ranked third. And as Bob Muenchen notes, four of the top five ranked data mining solutions in this year's poll are open-source. R was also ranked in this poll as the most popular language for implementing data mining application, beating out SQL and Java.

See the link below for the complete list of tools ranked in the KDNuggets poll.

KDNuggets: Poll Results: Top Analytics,Data Mining, Big Data software used 

Orbitz: R has become the data-mining tool of choice

Sameer Chopra, vice president of Advanced Analytics at Orbitz Worldwide, wrote recently in Analytics magazine about the changing landscape of processes, software and systems for statistical modelers. In a section on "Big Data and Open Source Analytics", Chopra lays out the reasons why the R language "has become the data-mining tool of choice for machine learners":

• R has very good integration with Hadoop, an area where established commercial statistical tools have frankly been playing catch-up over the past year. (Note: At the time of this writing, some established statistical solution providers were announcing an access interface to Hadoop.)
• Many startups and smaller firms do not have deep pockets and are embracing open source tools such as the R programming language and NoSQL database systems such as MongoDB.
• R is a leading language for developing new statistical methods, and it is a platform for statistical innovation and collaboration across both the corporate world and academia. In my opinion, for the first time in years, the stronghold of established commercial players seems to be potentially threatened; open source tools are better suited for Big Data and will slowly but surely continue to take share away from commercialized statistical packages. In fact, traditional statistical vendors have recognized that R is a force to be reckoned with. In response, many of these vendors have developed hooks into R so users can interface with the R language.
• Based on the resumes I’ve been reading, the next generation of data miners is flocking to R as their go-to tool. Professors in general are comfortable with R; they tend to use R and Excel as part of their curriculum.
• In short, open-source analytics tools and platforms have arrived.

Chopra says that the usage of R in the commercial sector is growing "as firms such as Revolution Analytics focus on the enterprise capabilities for R" (for example, Revolution R Enterprise's Hadoop support and enterprise deployment).

Chopra also has some interesting perspectives on statistical modeling vs machine learning which you can find in the full article linked below.

Analytics magazine: The times they are a changin’ for advanced analytics

Thursday: Tweet-chat on Multiple Sclerosis research

The story about the great work that SUNY Buffalo has been doing to find a cure for Multiple Sclerosis with Revolution R Enterprise and IBM Netezza has generated a lot of attention, with stories in Forbes, InformationWeek and eWeek (amongst others). To continue the discussion, IBM has put together a panel for a "Tweet Chat" on Thursday (May 10) at noon Eastern time. I'll be there with the lead researcher Murali Ramanathan, Shawn Dolley from IBM and Tim Coetzee from the National MS Society. We'll be chatting about the search for a cure for MS, Big Data and analytics, in snippets of 140 characters or less. Join the conversation by following the #IBMDataChat hashtag at twitter.com or in your favourite Twitter client.

When: Thursday, May 10, 2012 from 12:00-1:00 p.m. ET (find your local time)
Where: Twitter hashtag #IBMDataChat
Who: 

• Shawn Dolley, IBM VP of Big Data Healthcare & Life Sciences (Moderator), @shawndolley
• Dr. Murali Ramanathan, SUNY Buffalo Professor Pharmaceutical Sciences and Neurology, Director of Graduate Studies & Co-Director, Data Intensive Discovery Initiative,@M_Ramanathan
• David Smith, Revolution Analytics R Evangelist & VP of Marketing, @revodavid
• Tim Coetzee, National MS Society Chief Research Officer, @tim_coetzee

Yes, you need more than just R for Big Data Analytics

Douglas Merrill, former CIO/VP of Engineering at Google, writes in Forbes about using the R language for data analysis:

Most folks with math-oriented graduate degrees will have written something in R, a non-commercial option for your big data analysis.  So, great graduates from great graduate schools know great tools.

His post is titled 'R Is Not Enough For "Big Data"', and you might be surprised to learn that I agree that title, although for a different reason. Douglas's point — and it's a valid one — is that simply pumping data through any software tool, without an understanding of the problem you're trying to solve and how statistical models apply to it, can lead to getting the wrong answers to the wrong questions:

If you ask the wrong question, you will be able to find statistics that give answers that are simply wrong (or, at best, misleading).

On net, having a degree in math, economics, AI, etc., isn’t enough. Tool expertise isn’t enough.  You need experience in solving real world problems, because there are a lot of importat limitations to the statistics that you learned in school.  Big data isn’t about bits, it’s about talent.

This is a great illustration of why the data science process is a valuable one for extracting information from Big Data, because it combines tool expertise with statistical expertise and the domain expertise required to understand the problem and the data applicable to it. He's right that you need data science talent and software to solve problems with Big Data ... and having software like R that supports the exploratory nature of the data science process is also critical.

But I also agree with the title for a different, technical reason: the R software is just one piece of software ecosystem — an analytics stack, if you will — of tools used to analyze Big Data. For one thing R isn't a data store in its own right: you also need a data layer where R can access structured and unstructured data for analysis. (For example, see how you can use R to extract data from Hadoop in the slides from today's webinar by Antonio Piccolboni.) At the analytics layer, you need statistical algorithms that work with Big Data, like those in Revolution R Enterprise. And at the presentation layer, you need the ability to embed the results of the analysis in reports, BI tools, or data apps.

So yes, Douglas is right: you need more than just R for Big Data. You also need a data layer, an analytics layer, and a presentation layer (all of which supports Big Data) ... and you need Data Science skills to make sure you're asking the right questions and getting appropriate answers.

Forbes: R Is Not Enough For "Big Data"

Information Age: graduates driving industry adoption of R

Information Age recently published a feature article devoted to the R language, "Putting the R in analytics". Says author Pete Swabey:

Already popular in universities, there are signs that R is finding increasing adoption in the enterprise. This promises to lower the barriers of entry for advanced analytics, and may accelerate the mathemitisation of business management.

The article includes an overview of the history of R: its predecessor, the S language; the transition to open-source R; the pervasiveness of R in academia; and how this is driving an increasing rate of adoption in industry.

This popularity in academia means that R is being taught to statistics students, says Matthew Aldridge, co-founder of UK- based data analysis consultancy Mango Solutions. “We're seeing a lot of academic departments using R, versus SPSS which was what they always used to teach at university,” he says. “That means a lot of students are coming out with R skills.”

Finance and accounting advisory Deloitte, which uses R for various statistical analyses and to visualise data for presentations, has found this to be the case. “Many of the analytical hires coming out of school now have more experience with R than with SAS and SPSS, which was not the case years ago,” says Michael Petrillo, a senior project lead at Deloitte's New York branch.

Like many companies today, Deloitte needs to develop analytics with varied and large quantities of data, and is using Revolution R Enterprise for big-data analytics and API integration of R:

Deloitte is currently preparing a big data pilot using Revolution Analytics’ enhanced R product. “We are using the server-based version of Revolution R to investigate big data analysis capabilities,” says Petrillo.

“We are looking at integration options to [big data programming platform] Hadoop, as well as ability to integrate R code into other applications via a web services framework.”

I'm also quoted in the article discussing the big-data extensions of Revolution R Enterprise:

Smith argues that these enhancements are necessary if R is to be applied to ‘big data’, i.e. data whose volume, velocity and variability outstrip the capabilities of conventional relational databases.

For more on the history of R and its applications in business, read the complete article in Information Age at the link below.

Information Age: Putting the R in Analytics

Big Data statistics in the search for a cure for MS

Multiple Sclerosis (MS) is a debilitating and complex disease with an unknown cause — and for which there is currently no cure. The SUNY Buffalo is home to one of the leading multiple sclerosis (MS) research centers in the world, and as reported in Healthcare IT News, the research team is using IBM Netezza and Revolution R Enterprise to accelerate its study of more than 2,000 genetic and environmental factors that may contribute to MS symptoms:

Shawn Dolley, vice president and GM of global healthcare and life science at IBM Big Data (he worked for Marlborough, Mass.-based Netezza before IBM's acquisition of that company in 2010), remembers when Harvard Medical School researchers told him not long ago that, when it came to solving problems with big data analytics, "We have four questions we can ask per month." Why four? "Well, there's four weekends per month," he recalls them saying. "That's how long it takes us to run our jobs."

Now, using an IBM Netezza analytics appliance, in conjunction with software from Revolution Analytics, researchers can analyze disparate data in a matter of minutes instead of days, regardless of what type or size it is, say IBM officials. The technology automatically consumes and analyzes the data, and makes the results available for further analysis, leaving researchers more time to analyze trends. 

As described in this case study, SUNY Buffalo researchers were able to accelerate their research by reducing computation times more than 100-fold in the search for significant interactions between thousands of genetic and environmental factors. The R language itself also accelerates the research process, by making it easier to include new kinds of variables such as discrete dependent, Poisson dependent or continuous normally-distributed variables by simply adding a few lines of code. (Using another language would have required a re-writte of the entire algorithm.)

From an analytics perspective, the researchers were able to write a version of their software tools in in Revolution R Enterprise for IBM Netezza, so that all their reporting as well as our analysis is all consolidated in one place. This prevents the need to move very large amounts of data in and out of Netezza, which would cause further delays. They were also able to use a wider variety of data sets. 

The SUNY researchers were able to: 

• Use the new algorithms and add multiple variables that before, were nearly impossible to achieve. 
• Reduce the time required to conduct analysis from 27.2 hours without Netezza to 11.7 minutes with it. 
• Carry out their research with little to no database administration (Unlike other HPC platforms or databases available, Netezza was designed to require a minimum amount of maintenance)
• Publish multiple articles in scientific journals, with more in process.
• Proceed with studies based on ‘vector phenotypes’—a more complex variable that will further push the Netezza platform.

You can learn more about the research by the SUNY Buffalo team in this IBM Netezza blog post, and in the IBM press release at the link below.

IBM Press Releases: Leading Global Multiple Sclerosis Research Center Taps IBM Analytics to Improve Patient Care

R at the Consumer Financial Protection Bureau

The O'Reilly Radar blog has a lengthy and very interesting interview with the lead and deputy CIOs of the Consumer Financial Protection Bureau, the new US government agency devoted to consumer protections in the financial markets. In that interview, they talk about the many open-source tools used in the agency (and the parent Treasury Department): Linux, WordPress, Splunk, Django, Git and  yes -- R for data analysis with Big Data. (The CFPB has also made public its policy on use of open-source software.) They reveal an interesting cultural shift related to the use of R in this question at the end of the interview:

How do you see open source contributing to your ability to get insights from large amounts of data? If you're recruiting developers, can they actually make a difference in helping their fellow citizens?

[...] R is an interesting example. What we're finding is that as more people are coming out of academia into the professional world, they're actually used to using R in school. And then they have to come out and learn a different tool and they're actually working in the marketplace.

It's similar with the Mac versus the PC. You get people using the Mac in college — and suddenly they have to go to a Windows interface. Why impose that on them? If they're going to be extremely productive with a tool like R, why not allow that to be used?

Here at Revolution Analytics, we're hearing much the same story from commercial organizations who have data analysis platforms build around legacy platforms like SAS. When all of the new recruits coming out of school are already trained in R, the choice is between spending a year training the up in SAS (and incurring both the expense and the opportunity cost of a year's work in the process), or setting up a platform based around a commercially-supported R distribution. And more and more often, that second choice is the more compelling one.

O'Reilly Radar: Open source is interoperable with smarter government at the CFPB

R, Twitter and McDonald's

Ed Chen is a data scientist at Twitter, so he's accustomed to working with big data and complex models. In an interview with MIT Technology Review, he describes his data science toolbox:

A common pattern for me is that I'll code a MapReduce job in Scala, do some simple command-line munging on the results, pass the data into Python or R for further analysis, pull from a database to grab some extra fields, and so on, often integrating what I find into some machine learning models in the end.

He put this toolbox to great use in a recent blog post, Infinite Mixture Models with Nonparametric Bayes and the Dirichlet Process. After using simulation in Ruby and Python to generate some test data, he used the R language to create a novel classification model which can group "similar" members from a data set, without needing to specify the number of groups in advance. He used this model to categorize the McDonald's member into a number of "food groups" containing products with similar nutritional content.

One cluster contained all the desserts: Baked Hot Apple Pie, Snack Size McFlurry, and others including the three below: 

As you can see, the foods in the "dessert" group cluster together because of high trans fat content, low fiber, and other similar nutritional attributes. Other groups identified by the model include salads, burgers and other fried food, three categories of sauces, and (in a cluster all on its own) Fruit and Maple Oatmeal: the only high-fibre item on the menu.

For an astounding amount of detail about the analysis, visit Ed Chen's blog at the link below.

Edwin Chen's Blog: Infinite Mixture Models with Nonparametric Bayes and the Dirichlet Process

SAP integrates R with HANA

We can add SAP to the list of vendors offering R integration with their products. InformationWeek reports that the new SAP BusinessObjects Predictive Analysis model provides a graphical user interface to R. Created in reaction to "competitive and market forces, including the momentum of open source R", the new module provides in-database processing (presumably by embedding R within HANA in-memory database appliance, as described in this SAP presentation), and by offering "mainstream analytics" (presumably in-built analytic methods, like those included with the IBM Netezza appliance).

SAP BusinessObjects Predictive Analysis is currently available only to a limited set of clients and expected to be generally available later this year. 

InfoWorld: SAP Steps Up Commitment To Predictive Analytics