Revolutions

SAP integrates R with HANA

We can add SAP to the list of vendors offering R integration with their products. InformationWeek reports that the new SAP BusinessObjects Predictive Analysis model provides a graphical user interface to R. Created in reaction to "competitive and market forces, including the momentum of open source R", the new module provides in-database processing (presumably by embedding R within HANA in-memory database appliance, as described in this SAP presentation), and by offering "mainstream analytics" (presumably in-built analytic methods, like those included with the IBM Netezza appliance).

SAP BusinessObjects Predictive Analysis is currently available only to a limited set of clients and expected to be generally available later this year. 

InfoWorld: SAP Steps Up Commitment To Predictive Analytics

R integrated throughout the enterprise analytics stack

The past couple of years have seen a dramatic growth in the use of the R language in the enterprise. R has always been pervasive in academia for research and teaching in statistics and data science, and as new graduates trained in R have migrated to the workplace the demand for R in corporations has become more and more intense. 

Database vendor Oracle estimates that "R has attracted over two million users since its introduction". James Kobielus, noted Forrester analyst and predictive analytics expert, recently said in PCWorld that "R has become a real ubiquitous force in advanced analytics. It's everywhere. Enterprise adoption of it has been growing steadily. When we ask our customers what they're using for statistical modeling they'll say SAS or [IBM's] SPSS, but they increasingly say R in the same breath." 

With rapid growth of the use of R in the enterprise comes a corresponding increase in demand for enterprise support for R from its users, and demand for integration of R into corporate systems from IT (both areas in which Revolution Analytics provides software and expertise). Most large organizations have a sophisticated infrastructure devoted to data analysis, with an "analytics stack" of software to provide data warehousing and query, predictive analytics, reporting, presentation, and Business Intelligence (BI). As a result software vendors at every layer of this stack have added functionality to integrate R, accommodating demand from both users and IT, and to serve the needs of data-driven business decision makers. Let's take a look at some of the applications within the analytics stack which now provide integration with R. 

The Data Layer

The data layer is where the lifeblood of the analysis — the data — is stored and prepared. Especially for high-performance and big-data applications, analytics based in R can benefit from the infrastructure that the data layer provides. The IBM blog has a great post with an in-depth discussion about integrating R with the data layer.

IBM Netezza, the high-performance data-warehousing appliance, is integrated with R (in partnership with Revolution Analytics). R users can use Revolution R Enterprise to run massively-parallel computations in R within the IBM Netezza appliance and implement high-performance, big-data analytics (with high-frequency financial data, for example). [Update: a free webinar on February 29 will describe the integration between IBM Netezza and Revolution R Enterprise in detail.] 

Oracle announced last year a forthcoming connection between R and Oracle, which was made available in February 2012. Oracle R Enterprise is aimed at statisticians who are "don't know SQL" and are "not familiar with DBA tasks". It is available as part of the Oracle Advanced Analytics option (priced at around $23,000 per core), and provides a "transparency layer" for with functions to connect to Oracle  and use R functionality in the Oracle database. Oracle also maintains the open-source ROracle package which provides similar functionality for open-source R. 

Cloudera's Distribution Including Apache Hadoop provides support for R in partnership with Revolution Analytics. This connection makes it possible to manipulate Hadoop data stores in R directly from HDFS and HBASE, and give R programmers the ability to write MapReduce jobs in R using Hadoop Streaming.

IBM BigInsights, the Hadoop platform from IBM, is also integrated with R and Revolution R Enterprise. BigInsight queries can make use of the Map-Reduce construct while running R computations in parallel.

Teradata's Enterprise Data Warehousing platform provides in-database analytics using R via the free teradataR package. This package allows R users to connect to Teradata, create data frames linked to Teradata and to call in-database analytic functions. The Teradata Aster MapReduce Platform also provides integration with R. 

Sybase RAP, the edition of the Sybase database for financial data, provides integration with R. Providing the R language alongside Sybase RAP allows for faster algorithm development and extensive backward testing on historical data. Sybase also regularly highlights R integration in its financial newsletters and webinars.

SAP HANA provides in-database analytics based on R with its SAP BusinessObjects Predictive Analysis module.

The Analytics Layer

The Analytics layer is where the magic happens: statistical modeling, predictive analytics and custom data visualization. Fed with (usually) structured data sourced from the Data Layer, R is widely used here to categorize, predict, and generally provide insight into corporate data stores. In many organizations, older data analysis tools remain in use, and so interfaces to R have been added provide support for analysts and data scientists who prefer to use R and to fill in the gaps of these legacy tools with modern, high-performance analytics.

Revolution Analytics is the leading commercial organization focused on software and support for R. Its Revolution R Enterprise software extends open-source R with productivity interfaces, high-performance statistical computing, big-data analytics, and enterprise integration of R. 

SAS has been a statistical analysis workhorse since the early 70's. Now, with so many graduates in statistics trained in R instead of SAS, SAS has introduced the ability to call R from SAS/IML. (It's also possible to call R directly from base SAS thanks to a free package developed at Roche Pharmaceuticals.) SAS JMP, the point-and-click data analysis package, now also provides support for R.

IBM SPSS Statistics, the popular desktop data analysis software known simply as SPSS before being acquired by IBM in 2010, provides integration to R via the Statistics Programmability Extension module.

RStudio, an open-source software company, provides an integrated development environment for developing code in the R language. 

Matlab, a numerical computing language used by engineers, also offers the ability to call R from Matlab on Windows.

Zementis software allows models created with R to be scored on massive data sets using the ADAPA Decision Engine and Revolution R Enterprise.

The Presentation Layer

Data analysis makes the most impact in the enterprise when it can be readily acted upon by decision makers: often, business executives not steeped in the arcana of data warehousing or statistical analysis. As a result, many reporting and business intelligence tools now make it possible to make it possible to incorporate the resuts of analyses generated in R in the presentation layer, in a format tuned to the needs of a business audiecne.  

Jaspersoft's business intelligence software makes it possible to incorporate the results of R-based analytics into BI dashboards and reports, via integration with Revolution R Enterprise.

TIBCO Spotfire's interactive business intelligence dashboards make it possible to share results and models from R. 

You might not think of Microsoft Excel as more of a spreadsheet than a presentation tool, but it is very widely used on the desktop as a "container" for static and interactive reports based on statistical analysis. While Excel does not have out-of-the-box integration with R, is is possible to integrate R-based computation into Excel spreadsheets via RExcel and Revolution Analytics' RevoDeployR web services API.

R: Integrated throughout the analytics stack

As you can see, for organizations who need to create advanced analytics applications, R is integrated throughout the analytics stack: for data access, for presentation of results, and of course for the statistical analysis process itself. This degree of integration by so many companies is indicative of the level of demand for R throughout the enterprise. As the leading provider of commercial software and support for R, Revolution Analytics supports R users througout the organization, helps IT departments integrate Revolution R Enterprise throughout the analytics stack, for high-performance and big data applications based on the R language.

[Update Mar 15 2012: Added SAP HANA to the list.]

Revolution R Enterprise: production-grade analytics software built upon the powerful open source R statistics language.

Oracle's Big Data Appliance to include R

At the Oracle OpenWorld conference in San Francisco today, Oracle announced the new Oracle Big Data Appliance, "a new engineered system that includes an open source distribution of Apache™ Hadoop™, Oracle NoSQL Database, Oracle Data Integrator Application Adapter for Hadoop, Oracle Loader for Hadoop, and an open source distribution of R." Oracle's foray into the Hadoop and NoSQL spaces has captured most of the attention, but it's the inclusion of open source R that I find most interesting. Oracle is clearly finding demand from its customers for advanced analytics capabilities for big data, and is looking to R to fill that gap. I agree with Ed Dumbill's assessment:

Big data isn't much use until you can make sense of it, and the inclusion of R in Oracle's big data appliance bears this out. It also sets up R as a new industry standard for analytics: something that will raise serious concern among vendors of established statistical and analytical solutions SAS and SPSS.

Of course, this isn't the first time that R has been embedded into a data warehousing appliance. IBM Netezza's iClass device integrates with Revolution R, and AsterData, the Teradata Data Warehouse Appliance, and Greenplum all provide connections to R as well. Here at Revolution Analytics, we think that such enterprise-level integrations with R serve to grow the R ecosystem and serve as validation of R as a key platform for advanced analytics. As CEO Norman Nie said to GigaOm this weekend, 

"Oracle’s announcement to embed R demonstrates validation for the leading statistics language and offers further evidence that R is a key weapon in advanced analytics today", said Dr. Nie. "As the Enterprise R company, Revolution Analytics has seen an enormous demand for R solutions, so it’s no surprise that Oracle and other companies are looking to further evangelize and distribute R among the enterprise."

In today's announcement, Oracle also announced another new component, "Oracle R Enterprise". Details are sparse on what exactly this is, so far, other than it allows you to "run existing R applications and use the R client directly against data stored in Oracle Database", presumably thanks to an R package created by Oracle. Timothy Prickett Morgan at The Register has a couple of extra details:

The Big Data Appliance also includes the R programming language, a popular open source statistical-analysis tool. This R engine will integrate with 11g R2, so presumably if you want to do statistical analysis on unstructured data stored in and chewed by Hadoop, you will have to move it to Oracle after the chewing has subsided.

This approach to R-Hadoop integration is different from that announced last week between Revolution Analytics, the so-called Red Hat for stats that is extending and commercializing the R language and its engine, and Cloudera, which sells a commercial Hadoop setup called CDH3 and which was one of the early companies to offer support for Hadoop.

We look forward to hearing more about support for R from Oracle and other enterprise vendors. As the R ecosystem continues to grow into the enterprise space, this can only mean more investment in the R project as a whole.

Oracle Press Releases: Oracle Unveils the Oracle Big Data Appliance

Sharing live R functions with OpenCPU

OpenCPU is a new initiative from R user Jeroen Ooms to make innovations in statistics, visualization and data-science more widely applicable. Based on open-source principles, it's a web-based service that lets you upload data visualizations and analyses as R scripts, and allow others to run them on demand. For example, you can upload a script to visualize a company's stock performance, which can then be embedded as a live chart, like this:

That chart was generated on-demand by the OpenCPU R server, and should display an up-to-date stock history for IBM. It was generated by a few lines of R code that Jeroen published to OpenCPU, and I called from this blog post using the OpenCPU API. You can publish your own code there as well, and share new data science techniques implemented in R in the same way. 

As an open platform, be aware that any everything you do and post on OpenCPU should be considered public, and so isn't suitable for anything that contains private or sensitive data. If you're looking for a secure platform for distributing R code, where access to sensitive or resource-intensive code needs to be restricted to authenticated users, then you should take a look at the RevoDeployR platform from Revolution Analytics. (Jeroen gave a presentation about the use of RevoDeployR to address such issues at the useR! 2010 conference.)

OpenCPU: Documentation

Calling R from JasperReports Server

Revolution Analytics today announced a partnership with Jaspersoft, the makers of the most widely-used business intelligence software in the world. With this partnership, Revolution R and Jaspersoft software work together to bring the power of analytics coded in R to business users working with Business Intelligence (BI) dashboards and reports.

If you're a Jaspersoft or R developer and would like to try your hand at integrating the results of R scripts into Jaspersoft, you can download the free connection between JasperReports Server and RevoDeployR, the Web Services framework for Revolution R. (If you don't have them, you can download the open-source Jaspersoft distribution from the Jaspersoft website, and RevoDeployR is available for free download to academics.)

Also for developers, we'll be hosting a live webinar on March 16 on Integrating R into 3rd Party and Web Applications Using RevoDeployR which will include a section on using the RevoConnectR for JasperReports Server connector. Complete details and registration for the webinar are available here.

Revolution Analytics Partners: Jaspersoft

SAS vs Open Source, ctd

Following up on the story from last week, where SAS CEO Jim Goodnight said he "hadn't noticed" competition from open-source alternatives, open-source BI vendor Pentaho's "Chief Geek" James Dixon responds:

What this means is that SAS has moved from the Igorance phase to the Ridicule phase of battling open source, they only have Fighting and Losing to go.

There are some interesting parallels with Microsoft's reaction to the insurgency of Linux 10 years ago, as James analyzes in the full post linked below.

James Dixon’s Blog: SAS under pressure from Pentaho

SAS vs Open Source

SAS CEO Jim Goodnight elicited guffaws from the audience during his keynote at JSM this year when he said "I have a problem with government using open source software". Now it seems Goodnight is at the center of another controversy related to open source, having dismissed the growth of open-source software for business intelligence in an interview with CBR online:

Asked whether open source BI and data integration software from the likes of Jaspersoft, Pentaho and Talend is a growing threat, [Goodnight] said: "We haven't noticed that a lot. Most of our companies need industrial strength software that has been tested, put through every possible scenario or failure to make sure everything works correctly."

Between the growth of R for data analysis, Talend for ETL and the likes of Jaspersoft and and Pentaho for BI -- all areas where SAS has pricey commercial solutions -- it seems hard to believe that SAS hasn't noticed the impact of open source software in commercial environments. And, as Pentaho's Vinay Joosery pointed out today in a stinging response,

"Denying a market reality doesn't help your business in the long term."

Well said, Vinay.

CNR Online: Pentaho fires back across SAS' bows over "limited" open source appeal

Competition to SAS from yet another angle

Steve Miller at Information Management discusses the NYT story about SAS from last weekend, and notes that open-source R and major new players in the commercial BI space aren't the only threats to SAS's revenues. SAS is also facing competition from a low-priced commercial clone: WPS, created by a small software company based in the UK. WPS allows analysts to run SAS code and replicate the outputs without the need for a SAS license. And much like SAS has recently added facilities to call out to R, WPS can also interface with R thanks to the "Bridge to R" software created by third-party (or is it fourth-party by this stage?) software firm.

SAS clearly sees WPS as a threat: they recently sued WPS in the High Court in London, and the suit seeks to halt the sales of the WPS software. Interestingly, the lawsuit regards not the duplication of the SAS language itself, but instead relates to a violation of the terms of a student license of SAS:

“The lawsuit alleges that World Programming has used the SAS Learning Edition product for purposes of developing, testing and benchmarking its World Programming System software in breach of the terms of the SAS Learning Edition Licensing Agreement,” said John Boswell, Vice President and General Counsel at SAS. 

WPS is fighting back:

"SAS are claiming that the way in which WP has used a $100 learning package bought on Amazon was not permitted by the terms of the click wrap license.

SAS is alleging that WP have run "the SAS Learning Edition software for the purposes of observing its operation and functions in order to develop the WPS software in a way which reproduces the operation and functions of the SAS System". European law provides that one can lawfully observe the operation and functioning of a computer program and develop interoperable or alternative solutions which perform the same function. So far as we can see, WP have honoured both the spirit and the letter of the law. We are surprised that a company of SAS’s repute would want to pursue this action"

The case continues.

Information Management: More on the Statistical Revolution – a SAS Story (continued)

R for Actuaries

The Society of Actuaries publishes a quarterly newsletter called CompAct, and since October 2008 it has included R Corner, a column of R tips for beginning actuarial users. The column is written by Steve Craighead, an actuarial consultant based in Atlanta. Columns so far have covered:

• Introduction to R (p24)
• Reading data files (p11)
• Data objects (p9)
• Models and model formulae (p14) 

Society of Actuaries: CompAct Electronic Newsletter (PDF archive)

IBM to acquire SPSS

Many news outlets are reporting today that IBM has announced that it will acquire SPSS for 1.2 billion dollars in cash. That's $50 a share, a significant premium on the recent trading range of $30-$35. IBM clearly sees value in predictive analysis.

IBM has a strong history in supporting open-source projects: they are a significant contributor to Linux, for example. Given this, and SPSS's recent flirtations with R, I wonder if IBM -- or maybe a rival -- will take an interest in R, too.

The Register: Big Blue blows a billion on predictive software seller