Douglas Merrill, former CIO/VP of Engineering at Google, writes in Forbes about using the R language for data analysis:
Most folks with math-oriented graduate degrees will have written something in R, a non-commercial option for your big data analysis. So, great graduates from great graduate schools know great tools.
His post is titled 'R Is Not Enough For "Big Data"', and you might be surprised to learn that I agree that title, although for a different reason. Douglas's point — and it's a valid one — is that simply pumping data through any software tool, without an understanding of the problem you're trying to solve and how statistical models apply to it, can lead to getting the wrong answers to the wrong questions:
If you ask the wrong question, you will be able to find statistics that give answers that are simply wrong (or, at best, misleading).
On net, having a degree in math, economics, AI, etc., isn’t enough. Tool expertise isn’t enough. You need experience in solving real world problems, because there are a lot of importat limitations to the statistics that you learned in school. Big data isn’t about bits, it’s about talent.
This is a great illustration of why the data science process is a valuable one for extracting information from Big Data, because it combines tool expertise with statistical expertise and the domain expertise required to understand the problem and the data applicable to it. He's right that you need data science talent and software to solve problems with Big Data ... and having software like R that supports the exploratory nature of the data science process is also critical.
But I also agree with the title for a different, technical reason: the R software is just one piece of software ecosystem — an analytics stack, if you will — of tools used to analyze Big Data. For one thing R isn't a data store in its own right: you also need a data layer where R can access structured and unstructured data for analysis. (For example, see how you can use R to extract data from Hadoop in the slides from today's webinar by Antonio Piccolboni.) At the analytics layer, you need statistical algorithms that work with Big Data, like those in Revolution R Enterprise. And at the presentation layer, you need the ability to embed the results of the analysis in reports, BI tools, or data apps.
So yes, Douglas is right: you need more than just R for Big Data. You also need a data layer, an analytics layer, and a presentation layer (all of which supports Big Data) ... and you need Data Science skills to make sure you're asking the right questions and getting appropriate answers.
Forbes: R Is Not Enough For "Big Data"
I am a fortran IV programmer which happen to program in python. R or the like is just decoration.
Most probably I have the right age: I started to program
in the early 80s.
You do not need R at all.
But maybe once you start to use it, you might like it.
So why would you ask yourself about a different code?
Data handling is an issue in every program, and specific to some (any) programming languages.
BigData is a problem also in fortran IV.
Why should R be any special?
Posted by: mario | May 03, 2012 at 12:19