Revolutions

R turns 12; R 2.14.2 is out

As promised by the R Core Group, R 2.14.2 is out. This is the final patchlevel of the R 2.14.x series (R 2.15.0 is due on March 30), and so R 2.14.2 will be the R engine for the next release of Revolution R Enterprise in a couple of months. 

Today also marks the 12th anniversary since R 1.0.0 was released on February 29, 2000. If course, R had been in development for several years before then, as you can see in this abbreviated history of R: 

• 1993: Research project in Auckland, NZ by Ross Ihaka and Robert Gentleman
• 1995: Released as open-source software (Generally compatible with the 'S' language)
• 1997: R core group formed
• 2000: (February 29) R 1.0.0 released
• 2004: First international user conference in Vienna
• 2012: (February 29) R 2.14.2 released

But if you count 1.0.0 as "Day 0", then today marks R's twelfth birthday (and the third February 29 since Day 0). You can download the latest version of R from the CRAN network as the mirrors propagate.

R-announce mailing list: R 2.14.2 is released + R anniversary

Last chance for early-bird registations to useR! 2012

If you're planning to attend the international R user conference useR! 2012 in Nashville, TN in June, today is the last day to register and get a discounted registration. (And don't forget to check out the tutorials when you register.) Details at the useR! site linked below.

useR! 2012: Registration

Strata 2012 begins; R books on sale

I'm at the Strata 2012 conference in Santa Clara today and tomorrow (Revolution Analytics is a sponsor and giving talks). Wifi service looks pretty good so far, so I'll blog when I can. In the meantime, in recognition of the conference, several O'Reilly Media books on data and analytics are 50% off today and tomorrow, including several R-related books: R in a Nutshell, Getting Started with R, R Cookbook, Getting Started with Rstudio, Data Mashups in R, and Parallel R. The discounted books are available at the link below.

O'Reilly Media: Data Development and Resources

Webinar tomorrow: Big-data statistics with Revolution R with IBM Netezza

As explained in detail by Michele Chambers at the IBM Netezza blog, there are two keys to getting fast performance with statistical analysis on massive data sets with R:

1. Massive parallelization: break the problem down into small pieces, and run them in parallel
2. Bring the R engine to the data (not the other way around), to avoid data transfer delays

In a live webinar tomorrow, Wednesday February 29 (at 11AM Pacific), IBM Netezza's William Zanine and Revolution Analytics' Derek Norton will demonstrate this principle in action, with Revolution R Enterprise for IBM Netezza. Here's the description of the webinar:

Everyone involved in high-stakes analytics wants power, speed and flexibility regardless of the size of the data set and complexity of the analysis. Trailblazing organizations that have deployed IBM Netezza Analytics with their IBM Netezza data warehouse appliances (TwinFin) with Revolution R Enterprise are getting all three. Register for this webinar to find out how.

To set the stage, we'll provide a brief overview of the "R" statistical analysis language, the Revolution R Enterprise framework (with R at its core) as well in-database analytics on IBM Netezza Analytics Appliances. We'll be talking about what Revolution Analytics brings to IBM Netezza, and vice versa.

Then, we'll complete a model-building exercise from start to finish using the combined Revolution R Enterprise and IBM Netezza solution and demonstrate the performance and flexibility of the integrated offer. Join us as we:

• Begin with data visualizations and summary statistics to gain an understanding of our data 
• Split the data into training and test sets for model building
• Build models and
• Measure accuracy on both our training and test set and visualize the results.

Register today to learn why leveraging the Revolution R Enterprise and IBM Netezza analytics framework delivers power, speed and flexibility in a modern analytics platform. 

To register for the webinar (or if you're reading this after the live session, to find the slides and replay links), follow the link below.

Revolution Analytics webinars: Turbo-Charge Your Analytics with IBM Netezza and Revolution R Enterprise: A Step-by-Step Approach for Acceleration and Innovation

R integrated throughout the enterprise analytics stack

The past couple of years have seen a dramatic growth in the use of the R language in the enterprise. R has always been pervasive in academia for research and teaching in statistics and data science, and as new graduates trained in R have migrated to the workplace the demand for R in corporations has become more and more intense. 

Database vendor Oracle estimates that "R has attracted over two million users since its introduction". James Kobielus, noted Forrester analyst and predictive analytics expert, recently said in PCWorld that "R has become a real ubiquitous force in advanced analytics. It's everywhere. Enterprise adoption of it has been growing steadily. When we ask our customers what they're using for statistical modeling they'll say SAS or [IBM's] SPSS, but they increasingly say R in the same breath." 

With rapid growth of the use of R in the enterprise comes a corresponding increase in demand for enterprise support for R from its users, and demand for integration of R into corporate systems from IT (both areas in which Revolution Analytics provides software and expertise). Most large organizations have a sophisticated infrastructure devoted to data analysis, with an "analytics stack" of software to provide data warehousing and query, predictive analytics, reporting, presentation, and Business Intelligence (BI). As a result software vendors at every layer of this stack have added functionality to integrate R, accommodating demand from both users and IT, and to serve the needs of data-driven business decision makers. Let's take a look at some of the applications within the analytics stack which now provide integration with R. 

The Data Layer

The data layer is where the lifeblood of the analysis — the data — is stored and prepared. Especially for high-performance and big-data applications, analytics based in R can benefit from the infrastructure that the data layer provides. The IBM blog has a great post with an in-depth discussion about integrating R with the data layer.

IBM Netezza, the high-performance data-warehousing appliance, is integrated with R (in partnership with Revolution Analytics). R users can use Revolution R Enterprise to run massively-parallel computations in R within the IBM Netezza appliance and implement high-performance, big-data analytics (with high-frequency financial data, for example). [Update: a free webinar on February 29 will describe the integration between IBM Netezza and Revolution R Enterprise in detail.] 

Oracle announced last year a forthcoming connection between R and Oracle, which was made available in February 2012. Oracle R Enterprise is aimed at statisticians who are "don't know SQL" and are "not familiar with DBA tasks". It is available as part of the Oracle Advanced Analytics option (priced at around $23,000 per core), and provides a "transparency layer" for with functions to connect to Oracle  and use R functionality in the Oracle database. Oracle also maintains the open-source ROracle package which provides similar functionality for open-source R. 

Cloudera's Distribution Including Apache Hadoop provides support for R in partnership with Revolution Analytics. This connection makes it possible to manipulate Hadoop data stores in R directly from HDFS and HBASE, and give R programmers the ability to write MapReduce jobs in R using Hadoop Streaming.

IBM BigInsights, the Hadoop platform from IBM, is also integrated with R and Revolution R Enterprise. BigInsight queries can make use of the Map-Reduce construct while running R computations in parallel.

Teradata's Enterprise Data Warehousing platform provides in-database analytics using R via the free teradataR package. This package allows R users to connect to Teradata, create data frames linked to Teradata and to call in-database analytic functions. The Teradata Aster MapReduce Platform also provides integration with R. 

Sybase RAP, the edition of the Sybase database for financial data, provides integration with R. Providing the R language alongside Sybase RAP allows for faster algorithm development and extensive backward testing on historical data. Sybase also regularly highlights R integration in its financial newsletters and webinars.

SAP HANA provides in-database analytics based on R with its SAP BusinessObjects Predictive Analysis module.

The Analytics Layer

The Analytics layer is where the magic happens: statistical modeling, predictive analytics and custom data visualization. Fed with (usually) structured data sourced from the Data Layer, R is widely used here to categorize, predict, and generally provide insight into corporate data stores. In many organizations, older data analysis tools remain in use, and so interfaces to R have been added provide support for analysts and data scientists who prefer to use R and to fill in the gaps of these legacy tools with modern, high-performance analytics.

Revolution Analytics is the leading commercial organization focused on software and support for R. Its Revolution R Enterprise software extends open-source R with productivity interfaces, high-performance statistical computing, big-data analytics, and enterprise integration of R. 

SAS has been a statistical analysis workhorse since the early 70's. Now, with so many graduates in statistics trained in R instead of SAS, SAS has introduced the ability to call R from SAS/IML. (It's also possible to call R directly from base SAS thanks to a free package developed at Roche Pharmaceuticals.) SAS JMP, the point-and-click data analysis package, now also provides support for R.

IBM SPSS Statistics, the popular desktop data analysis software known simply as SPSS before being acquired by IBM in 2010, provides integration to R via the Statistics Programmability Extension module.

RStudio, an open-source software company, provides an integrated development environment for developing code in the R language. 

Matlab, a numerical computing language used by engineers, also offers the ability to call R from Matlab on Windows.

Zementis software allows models created with R to be scored on massive data sets using the ADAPA Decision Engine and Revolution R Enterprise.

The Presentation Layer

Data analysis makes the most impact in the enterprise when it can be readily acted upon by decision makers: often, business executives not steeped in the arcana of data warehousing or statistical analysis. As a result, many reporting and business intelligence tools now make it possible to make it possible to incorporate the resuts of analyses generated in R in the presentation layer, in a format tuned to the needs of a business audiecne.  

Jaspersoft's business intelligence software makes it possible to incorporate the results of R-based analytics into BI dashboards and reports, via integration with Revolution R Enterprise.

TIBCO Spotfire's interactive business intelligence dashboards make it possible to share results and models from R. 

You might not think of Microsoft Excel as more of a spreadsheet than a presentation tool, but it is very widely used on the desktop as a "container" for static and interactive reports based on statistical analysis. While Excel does not have out-of-the-box integration with R, is is possible to integrate R-based computation into Excel spreadsheets via RExcel and Revolution Analytics' RevoDeployR web services API.

R: Integrated throughout the analytics stack

As you can see, for organizations who need to create advanced analytics applications, R is integrated throughout the analytics stack: for data access, for presentation of results, and of course for the statistical analysis process itself. This degree of integration by so many companies is indicative of the level of demand for R throughout the enterprise. As the leading provider of commercial software and support for R, Revolution Analytics supports R users througout the organization, helps IT departments integrate Revolution R Enterprise throughout the analytics stack, for high-performance and big data applications based on the R language.

[Update Mar 15 2012: Added SAP HANA to the list.]

Revolution R Enterprise: production-grade analytics software built upon the powerful open source R statistics language.

RHadoop updated: improved performance and more control

Revolution Analytics' open-source RHadoop project, which provides integration between R and Hadoop, has been updated with the release of version 1.2 of the "rmr" package. New in this version: support for binary I/O formats, which improves on the text-only interfact by allowing use of faster and more space-efficient data formats like R's native serialization format. This version also improves the performance of the reduce step (to get around the fact that list appends in R are not constant-time operations), and provides control to the Hadoop user to do things like set number of reducers on a per-job basis.

Find more details about these and other updates in rmr 1.2 (available now) at the link below.

RHadoop: Overview of rmr v1.2

Revolution Analytics at Strata 2012

One of my favourite conferences, Strata: Making Data Work, starts tomorrow in Santa Clara, CA. Revolution Analytics is a proud sponsor, and I'll be there with the team to listen to some great talks and to meet other R users at our booth in the exhibition hall. There will be several R-related talks and tutorials during the conference, including two from the Revolution team as detailed in the announcement below:

Over the past year, "big data" has emerged as perhaps the single most significant trend in enterprise IT. Organizations are generating massive volumes of data on a daily basis and must properly process and analyze it for better operational insight. The R statistical language has concurrently emerged as a leading tool for big data analysis, due to its modern, flexible nature and unparalleled processing power.

Join Joseph Rickert of Revolution Analytics at 9:00 AM on February 28 at the O'Reilly Strata Conference for an in-depth overview of how R can be effectively used for data mining. His talk will include a comprehensive overview of R and how it can be properly leveraged for advanced statistical analysis for data sets of all sizes. Additionally, Antonio Piccolboni will be presenting on how R and Hadoop have been effectively integrated for big data analysis at 10:40 AM on February 29.

See you there!

Strata 2012: Schedule 

Because it's Friday: A better way to fold a map

Ever been frustrated by a paper map that, once unfolded, is impossible to figure out how to refold? The solution is the Miura-Ori sheet, which unfolds easily and naturally re-folds itself into a compact form:

Although this folding technique was first proposed (in a 1989 paper) by Japanese astrophysicist Koryo Miura to fold solar arrays for satellites, it is also perfect for paper maps. As explained at Eric Meltzer's blog (from where the above image was taken), 

A Miura sheet has only one degree of freedom, and can be thought of as having only two states: fully open, or fully closed. Since reversing one fold in the sheet (that is, making a “mountain” into a “valley”) requires reversing all of the adjacent folds as well, the Miura sheet feels as though it has a memory, and is very resistant to deformation.

Despite Meltzer being "a loss as to why all maps aren’t folded this way" (he has a Kickstarter project to create a Miura tourist map for San Francisco), I'm certain I've seen and used such maps several times. (I'm pretty sure the RATP printed a pocket-size Métro/RER map that folded out like this.) When I was a backpacker in Europe in the early 90's, I recall that many of the local tourist maps folded this way, often with some first cardstock attached to two diagonally-opposite corner squares, both to serve as a protective cover for the map and as a "handle" for opening and closing the map. If you want to try and make such a map yourself, the folding technique is quite straightforward: 

Eric Meltzer: The Miura-ori and how to fold it

Creating beautiful maps with R

Spanish R user and solar energy lecturer Oscar Perpiñán Lamigueiro has written a detailed three-part guide to creating beautiful maps and choropleths (maps color-coded with regional data) using the R language. Motivated by the desire to recreate this graphic from the New York Times, Oscar describes how he creates similar high-quality maps using R.

In Part 1, Oscar grabbed some voting data from Spanish elections and shapefiles (detailed map coordinates) from Spain's Official Statistics department (INE) and set to work in R. He then used the maptools package and lattice graphics to create this choropleth of voting patterns in Spain:

In Part 2, he uses the raster package to import pixel-based environmental data from NASA, to create a map that combines population density, land use and land cover in a single map. (Personally, I find it very difficult to interpret such multivariate maps encoded in multiple dimensions of color, but the process is interesting.)

Finally in Part 3, Oscar uses the pxR package to read map data in the PC-Axis format, and the gridSVG package to create an interactive SVG map. If you have a modern browser, you can click the image below, and then annotations will appear as move your mouse over the regions. [Updated: as Oscar explains in the blog, the annotiations require Javascript to work, so I've changed the image below to link to Oscar's annotated version.]

With these examples, you can see how you can get close to the types of maps featured in the New York Times (which, themselves, often involve R in their production). You can find the data and R code used to create these maps at Oscar's blog linked below.

Omnia sunt Communia!: Maps with R (III)

Analyzing weblog data with R

The R-chart blog explains how to read a weblog file into R, so you can analyze traffic to a website. For example, here's a page request chart created with R:

Now, charts like this are stock-in-trade for tools like Google Analytics, but this is still useful if you want to look at the performance of a site that hasn't been instrumented for analytics. It also allows you to analyze other data in the weblog, such as frequency of HTTP status codes:

(Looks like this particular site is experiencing a lot of page-not-found 404 errors.) With access to the raw weblog data in R, you could subject it to any kind of statistical analysis. 

You can find the R code and lots of tips for dealing with weblog data in R at the link below.

R-chart: Log File Analysis with R