Revolutions

R is to SAS as Java is to COBOL

An interview with Revolution Analytics CEO Dave Rich was published this week by BeyeNetwork. During the interview, Dace was asked about how the statistical modeling platforms have changed over the decades:

People have been doing statistical modeling and predictive analytics for 50 years now, SAS and SPSS have been around since the early ‘70s. What’s different now -- what’s making this move toward other statistical and “big data” areas?

David Rich: Well, I think obviously SAS and SPSS have been around, as you pointed out, for decades. We call that sort of the first generation of analytics and insight-driven solutions. In my perspective, having been in the business for more than three decades, it reminds me a bit of what COBOL was back in the day relative to business software. I see R as the more modern language. In this analogy, R would represent Java or C++. What happened in the middle of the nineties when the shift occurred is very similar to where we are now with R. Open source is a worldwide collaboration innovation. It’s a way to tap into that channel for research, and I think the role that Revolution Analytics can play – very similar to what Red Hat did back in the Linux days – is to be the conduit between the community and enterprise deployment.

The conversation also touched on the future of big data analytics, impact of advanced analytics on business, and the benefits of R and Revolution R Enteprise to reduce costs and expand the scope of possibility with big data analytics. For the complete interview, follow the link below.

BeyeNetwork: Advanced Analytics, Big Data and the Power of R: A Q&A Spotlight with David Rich, CEO of Revolution Analytics

Orbitz: R has become the data-mining tool of choice

Sameer Chopra, vice president of Advanced Analytics at Orbitz Worldwide, wrote recently in Analytics magazine about the changing landscape of processes, software and systems for statistical modelers. In a section on "Big Data and Open Source Analytics", Chopra lays out the reasons why the R language "has become the data-mining tool of choice for machine learners":

• R has very good integration with Hadoop, an area where established commercial statistical tools have frankly been playing catch-up over the past year. (Note: At the time of this writing, some established statistical solution providers were announcing an access interface to Hadoop.)
• Many startups and smaller firms do not have deep pockets and are embracing open source tools such as the R programming language and NoSQL database systems such as MongoDB.
• R is a leading language for developing new statistical methods, and it is a platform for statistical innovation and collaboration across both the corporate world and academia. In my opinion, for the first time in years, the stronghold of established commercial players seems to be potentially threatened; open source tools are better suited for Big Data and will slowly but surely continue to take share away from commercialized statistical packages. In fact, traditional statistical vendors have recognized that R is a force to be reckoned with. In response, many of these vendors have developed hooks into R so users can interface with the R language.
• Based on the resumes I’ve been reading, the next generation of data miners is flocking to R as their go-to tool. Professors in general are comfortable with R; they tend to use R and Excel as part of their curriculum.
• In short, open-source analytics tools and platforms have arrived.

Chopra says that the usage of R in the commercial sector is growing "as firms such as Revolution Analytics focus on the enterprise capabilities for R" (for example, Revolution R Enterprise's Hadoop support and enterprise deployment).

Chopra also has some interesting perspectives on statistical modeling vs machine learning which you can find in the full article linked below.

Analytics magazine: The times they are a changin’ for advanced analytics

Revolution Newsletter: May 2012

The most recent edition of the Revolution Newsletter is out. The news section is below, and you can read the full May edition (with highlights from this blog and community events) online. You can subscribe to the Revolution Newsletter to get it monthly via email.

New R Training Courses Announced. Three new R courses from leading R experts are now available for registration:

• An Introduction to R for SAS, SPSS, and Stata Users will be presented by Bob Muenchen (author of R for SAS and SPSS users) June 26-29. This is an on-line workshop with live instruction that you can attend from your desk - no travel necessary!
• Visualization in R with ggplot2 is also a new web-based course with live instruction from Garrett Grolemund & Dr. Winston Chang of Rice University, presented online June 19-20.
• R Development Master Class will be presented in-person by R package author and professor Hadley Wickham, June 21-22 in NYC and June 28-29 in Redwood City, CA.

Sign up to these courses now to secure your seat -- attendance is limited!

On the hunt for a cure for MS. Researchers at SUNY Buffalo are using big-data analytics and generic data toresearch a cure for Multiple Sclerosis. Read in Forbes how Revolution R and IBM Netezza speeds up their research, or join the #IBMDataChat discussion on Twitter at noon ET on Thursday, May 10 to learn more.

More free R webinars. Revolution Analytics' Spring webinar series continues, with upcoming presentations on Getting Up to Speed with R,  spatial statistics with R, data mining with R, Achieving High-Performing, Simulation-Based Operational Risk Measurement with RevoScaleR, and a look at the next version of Revolution R Enterprise.

Catch our archived webinar replays. Missed any of the recent presentations on Big Data with R and Hadoop, integrating R with BI tools and MS Office, or How Big Data is Changing Retail Marketing Analytics? Check our webinar archives for streaming replays and presentation downloads. 

Putting the R in Analytics. A feature article in Information Age reveals how new graduates are driving adoption of R in industry.

Upcoming Conferences for R Users. Revolution Analytics is proud to sponsor: R/Finance 2012 (the premier conference for R users in the finance industry) on May 11-12 in Chicago; and the Interface 2012 conference on the future of Statistical Computing, May 16-18 in Houston.

R user conference deadlines approaching. If you're planning to attend useR! 2012 in Nashville but want to avoid late registration fees, the deadline for regular registration is May 12. 

Revolution Analytics: Newsletter Archive

Multiple Sclerosis Tweet-Chat: Review

We had a great Twitter conversation last Thursday on the use of big-data analytics, Revolution R Enterprise, and IBM Netezza in the search for a cure for MS. Many thanks to the other panelists: Murali Ramanathan (SUNY Buffalo), Tim Coetzee (National MS Society) and moderator Shawn Dolley (IBM) for fielding and answering questions from interested parties following #IBMDataChat. As you can see from this twitteR analysis, it was a lively discussion, with more than 300 tweets during the designated hour:

IBM's James Kobielus has a summary of the chat, highlighting some of the key nuggets of information. For example, Dr Ramanathan revealed that this research is helping to understand interactions between genetic factors and environment that could help identify lifestyle and diet changes to manage MS. 

Also, in this interview with IBM's Mike Kearney I explain why the R language is uniquely suited for research like this, and how Revolution R works with the massive data volumes required for this study.

Thinking Inside the Box: Are you ready for big data? R is.

See R integrated with QlikView, Jaspersoft, Excel, and mobile apps

In yesterday's webinar, Revolution Analytics CTO David Champagne demonstrated how to integrate statistical graphics and analytic computations created using R software with a variety of third-party applications. In each case Revolution R Enterprise Server is running as a compute server to the client application, with R scripts launched on each user interaction via the RevoDeployR Web Services API. David demonstrated five examples of such integration (watch the demo by clicking on the links below, and switch to full-screen for easier viewing).

David showed three different ways of delivering interactive sales forecasts in client applications:

• An HTML 5 application that offers sales forecasts based on a user-selected history from product sales data. This type of interface would be suited to mobile devices like an Android smartphone or iPad. 

• An enhanced QlikView report, including a sales forecast from patio furniture sales data, with options for the type of forecast model and with data visualization from both R and QlikView.

• An enhanced Jaspersoft BI Dashboard, with an interactive sales forecast based on T-shirt sales.

David also showed two examples where the end-user has even more control over the parameters of the analytic computation:

• A custom interactive web-application written in JavaScript, to perform market basket analysis on retail transaction data. The distribution of purchases is rendered on an interactive map, and products commonly purchased together (within a selected sub-region) are displayed as a tree chart and as association rules. 

• A Microsoft Excel spreadsheet, with a custom toolbar button that displays a regression dialog, with the regression results (from R) embedded directly in the same spreadsheet the source data was taken from.

The great thing about integrating R into client applications in this way is that:

• The R computation is run on-demand, and with the results presented in context to the end user;
• The end user doesn't need to know R (in fact, they're probably don't even know R is involved), while still providing some control over the computation done in R.
• R doesn't need to be installed on the client device. In fact, Revolution R Enterprise will most likely be running on a dedicated remote server, in a data center or even in the cloud.

You can see David's entire presentation (including details about the client/server architecture and the Web Services API that makes all of this possible) in the webinar replay below, or by downloading the webinar slides.

 

Revolution Analytics webinars: Calling All Data Scientists and Web Developers! Integrate Your Advanced Analytics into BI Apps and MS Office and Multiply Their Value

Thursday: Tweet-chat on Multiple Sclerosis research

The story about the great work that SUNY Buffalo has been doing to find a cure for Multiple Sclerosis with Revolution R Enterprise and IBM Netezza has generated a lot of attention, with stories in Forbes, InformationWeek and eWeek (amongst others). To continue the discussion, IBM has put together a panel for a "Tweet Chat" on Thursday (May 10) at noon Eastern time. I'll be there with the lead researcher Murali Ramanathan, Shawn Dolley from IBM and Tim Coetzee from the National MS Society. We'll be chatting about the search for a cure for MS, Big Data and analytics, in snippets of 140 characters or less. Join the conversation by following the #IBMDataChat hashtag at twitter.com or in your favourite Twitter client.

When: Thursday, May 10, 2012 from 12:00-1:00 p.m. ET (find your local time)
Where: Twitter hashtag #IBMDataChat
Who: 

• Shawn Dolley, IBM VP of Big Data Healthcare & Life Sciences (Moderator), @shawndolley
• Dr. Murali Ramanathan, SUNY Buffalo Professor Pharmaceutical Sciences and Neurology, Director of Graduate Studies & Co-Director, Data Intensive Discovery Initiative,@M_Ramanathan
• David Smith, Revolution Analytics R Evangelist & VP of Marketing, @revodavid
• Tim Coetzee, National MS Society Chief Research Officer, @tim_coetzee

Yes, you need more than just R for Big Data Analytics

Douglas Merrill, former CIO/VP of Engineering at Google, writes in Forbes about using the R language for data analysis:

Most folks with math-oriented graduate degrees will have written something in R, a non-commercial option for your big data analysis.  So, great graduates from great graduate schools know great tools.

His post is titled 'R Is Not Enough For "Big Data"', and you might be surprised to learn that I agree that title, although for a different reason. Douglas's point — and it's a valid one — is that simply pumping data through any software tool, without an understanding of the problem you're trying to solve and how statistical models apply to it, can lead to getting the wrong answers to the wrong questions:

If you ask the wrong question, you will be able to find statistics that give answers that are simply wrong (or, at best, misleading).

On net, having a degree in math, economics, AI, etc., isn’t enough. Tool expertise isn’t enough.  You need experience in solving real world problems, because there are a lot of importat limitations to the statistics that you learned in school.  Big data isn’t about bits, it’s about talent.

This is a great illustration of why the data science process is a valuable one for extracting information from Big Data, because it combines tool expertise with statistical expertise and the domain expertise required to understand the problem and the data applicable to it. He's right that you need data science talent and software to solve problems with Big Data ... and having software like R that supports the exploratory nature of the data science process is also critical.

But I also agree with the title for a different, technical reason: the R software is just one piece of software ecosystem — an analytics stack, if you will — of tools used to analyze Big Data. For one thing R isn't a data store in its own right: you also need a data layer where R can access structured and unstructured data for analysis. (For example, see how you can use R to extract data from Hadoop in the slides from today's webinar by Antonio Piccolboni.) At the analytics layer, you need statistical algorithms that work with Big Data, like those in Revolution R Enterprise. And at the presentation layer, you need the ability to embed the results of the analysis in reports, BI tools, or data apps.

So yes, Douglas is right: you need more than just R for Big Data. You also need a data layer, an analytics layer, and a presentation layer (all of which supports Big Data) ... and you need Data Science skills to make sure you're asking the right questions and getting appropriate answers.

Forbes: R Is Not Enough For "Big Data"

Google BigQuery and the Github Data Challenge

Github has made data on its code repositories, developer updates, forks etc. from the public GitHub timeline available for analysis, and is offering prizes for the most interesting visualization of the data. Sounds like a great challenge for R programmers! The R language is currently the 26th most popular on GitHub (up from #29 in December), and it would be interesting to visualize the usage of R compared to other languages, for example. The deadline for submissions to the contest is May 21.

Interestingly, GitHub has made this data available on the Google BigQuery service, which is available to the public today. BigQuery was free to use while it was in beta test, but Google is now charging for storage of the data: $0.12 per gigabyte per month, up to $240/month (the service is limited to 2TB of storage - although there a Premier offering that supports larger data sizes ... at a price to be negotiated). While members of the public can run SQL-like queries on the GitHub data for free, Google is charging subscribers to the service 3.5 cents per Gb processed in the query: this is measured by the source data accessed (although columns of data not referenced aren't counted); the size of the result set doesn't matter.

With analysis limited to simple queries — "select" statements and by-row aggregations and the like — it's hard to see how this will have a big impact on companies needing to do even moderately advanced analytics on their data. It may prove useful as a store for relatively static data (like the GitHub example above), but given it takes 20 minutes to transfer 200Gb of data into BigQuery it doesn't seem well suited for frequently-changing data. Unlike Amazon S3 though, Google doesn't charge for data transfer in and out of the cloud — but at least with Amazon you can transfer data for free to Amazon AWS. And with Amazon AWS, you have a limitless range of AMIs (machine images) at your disposal, and can use advanced analytic tools like Revolution R Enterprise for predictive modeling and the like. If you're already storing your data in the cloud (as more companies are doing — Netflix is great example), this makes Amazon a more compelling cloud platform for Big Data Analytics. But if Google opens up more flexible data analysis options (or even the Google Prediction API) to BigQuery, this might change.

Big Data statistics in the search for a cure for MS

Multiple Sclerosis (MS) is a debilitating and complex disease with an unknown cause — and for which there is currently no cure. The SUNY Buffalo is home to one of the leading multiple sclerosis (MS) research centers in the world, and as reported in Healthcare IT News, the research team is using IBM Netezza and Revolution R Enterprise to accelerate its study of more than 2,000 genetic and environmental factors that may contribute to MS symptoms:

Shawn Dolley, vice president and GM of global healthcare and life science at IBM Big Data (he worked for Marlborough, Mass.-based Netezza before IBM's acquisition of that company in 2010), remembers when Harvard Medical School researchers told him not long ago that, when it came to solving problems with big data analytics, "We have four questions we can ask per month." Why four? "Well, there's four weekends per month," he recalls them saying. "That's how long it takes us to run our jobs."

Now, using an IBM Netezza analytics appliance, in conjunction with software from Revolution Analytics, researchers can analyze disparate data in a matter of minutes instead of days, regardless of what type or size it is, say IBM officials. The technology automatically consumes and analyzes the data, and makes the results available for further analysis, leaving researchers more time to analyze trends. 

As described in this case study, SUNY Buffalo researchers were able to accelerate their research by reducing computation times more than 100-fold in the search for significant interactions between thousands of genetic and environmental factors. The R language itself also accelerates the research process, by making it easier to include new kinds of variables such as discrete dependent, Poisson dependent or continuous normally-distributed variables by simply adding a few lines of code. (Using another language would have required a re-writte of the entire algorithm.)

From an analytics perspective, the researchers were able to write a version of their software tools in in Revolution R Enterprise for IBM Netezza, so that all their reporting as well as our analysis is all consolidated in one place. This prevents the need to move very large amounts of data in and out of Netezza, which would cause further delays. They were also able to use a wider variety of data sets. 

The SUNY researchers were able to: 

• Use the new algorithms and add multiple variables that before, were nearly impossible to achieve. 
• Reduce the time required to conduct analysis from 27.2 hours without Netezza to 11.7 minutes with it. 
• Carry out their research with little to no database administration (Unlike other HPC platforms or databases available, Netezza was designed to require a minimum amount of maintenance)
• Publish multiple articles in scientific journals, with more in process.
• Proceed with studies based on ‘vector phenotypes’—a more complex variable that will further push the Netezza platform.

You can learn more about the research by the SUNY Buffalo team in this IBM Netezza blog post, and in the IBM press release at the link below.

IBM Press Releases: Leading Global Multiple Sclerosis Research Center Taps IBM Analytics to Improve Patient Care

Revolution Newsletter: April 2012

The most recent edition of the Revolution Newsletter is out. The news section is below, and you can read the full April edition (with highlights from this blog and community events) online. You can subscribe to the Revolution Newsletter to get it monthly via email.

Spring Webinar Series. Our Spring Webinar Series features presentations from Revolution Analytics staff and thought leaders in predictive analytics and Big Data. Don't miss webinars coming soon on Big Data with R and Hadoop;integrating R with MS Office, spatial statistics with R, data mining with R, retail marketing analytics, a graduate certificate program in R, and much more. See our webinar schedule for full details. Registration is free, but attendance is limited -- sign up soon to claim your seat for the live presentation.

Using Revolution R Enterprise with the IBM Netezza appliance, you can build powerful predictive models for Big Data using using Naive Bayes and Classification Trees, as shown in this demonstration video.

Revolution Analytics' Senior Advisor for Products and Strategy Norman Nie shares Five Benefits of Big Analyticsin the 2012 SAP Sybase Capital Markets Guide.

If you missed any of our recent webinars, catch our on-demand replays to see: how R was used to analyze when banks should modify distressed home loans; how R is used for actuarial analysis at Deloitte Consulting; and how to make model results more impactful.

Revolution Analytics is proud to sponsor R/Finance 2012, the premier conference devoted to applied financial analysis using R. Register now for two-days of in-depth presentations May 11-12 in Chicago, including David Humke (Northern Trust) who will speak on Operational Risk Measurement with R and RevoScaleR.

The annual Future of Open Source Survey (sponsored by the 451 Group, Revolution Analytics and other open-source firms) tracks opinions and trends on the use of open-source software in commercial organizations. Please take 5 minutes to provide your input. The conclusions from the 2011 survey are available here.

Revolution Analytics: Newsletter Archive