Revolutions

Announcing Revolution R Enterprise 6.2

We are pleased to announce that Revolution R Enterprise Release 6.2 is available to new subscribers today. This new software release from Revolution Analytics includes a number of key new features:

• Support for open source R 2.15.3, the latest stable release of R.  Since Release 2.14.2, the R Project has added 89 new features, 11 performance enhancements and 139 bug fixes.  Moreover, there are 137 CRAN packages whose latest version depends on R 2.15.3.

• The new High-Speed Teradata Connector, which is based on Teradata Parallel Transporter. According to benchmark testing performed by the Teradata Center of Excellence, the High-Speed Connector is at least six times faster than the previously available ODBC connector, as shown in the chart below.

• Stepwise Regression for Linear Models. Stepwise Regression is a method for automated selection of variables in a predictive model. For an excellent article about Stepwise Regression in action, read this post. 

• The Parallel Random Number Generator provides an R interface to the parallel random number generators supplied with the Intel MKL libraries, enabling high quality parallel random numbers to be used in distributed computations performed by RevoScaleR’s rxExec function. This capability is a necessary foundation for randomized decision trees (trademarked by Breiman and Cutler as "Random Forests"), and is also useful for simulation and Monte Carlo analysis.

• Revolution R Enterprise users sometimes want to "roll up" data from one level to another — for example, to summarize transaction facts at the customer level. The rxCube and rxSummary functions now provide options to write by-group counts or summary statistics directly to a high performance .xdf file for further analysis. Users also now have more control over the summary statistics that are reported.

• Release 6.2 also includes some performance enhancements, including a fast fixed format text data source capability, as well as optimized memory utilization for sorts and merges.

• For customers who use our RevoDeployR capability for enterprise deployment of analytics there are several enhancements, including priority scheduling for asynchronous jobs, the ability to execute external scripts, plus enhanced versioning and lifecycle management for repository-managed files and scripts.

Revolution R Enterprise is available to new customers today. If you are an existing subscriber you can expect to receive an email on Monday, April 29 with detailed instructions on how to download and install the new software. (If you don't receive your update notification, please contact the Support team.) If you'd like to learn more about the new features, you can find an overview in this press release, or you can join the webinar What's New in Revolution R Enterprise 6.2 on Wednesday, May 1. 

R version 3 released

The R language marks a major milestone today with the release of R 3.0.0 (codename: "Masked Marvel"). The increment in the version number reflects not a fundamental change in the R langauge itself, but a recognition that the R codebase has matured to a point where closing out the 2.x series makes sense. 

Nonetheless, this release does include some major behind the scenes updates, not least of which is the introduction of big vectors to R, which eliminates some big data restrictions in the core R engine by allowing R to better use the memory available on 64-bit systems. Tal Galili lists the new functionality available in R 3.0.0 and provides a guide to upgrading and re-installing packages.

From everyone here at Revolution Analytics, our thanks go to the members of the R Core development team, who have volunteered so much time and expertise to furthering the R Project. The world of statistical computing would be a much poorer place without their contributions.

If you build R yourself, R version 3.0.0 is available for download in source from CRAN, and pre-built binaries (for Windows, Mac and Linux) will be available in the next couple of days. For Revolution R Enterprise users, the next release (version 6.2) will be based on the recently-released final Rv2 engine (R 2.15.3). We're currently working on integration of Rv3 for inclusion in a major update to Revolution R Enterprise in late 2013.

R-announce mailing list: R 3.0.0 is released

R 2.15.3 is released

The final installment of the R 2.x series is now available: R 2.15.3 was released this morning. If you build R yourself, the source files can be downloaded from CRAN now; pre-built binaries for Windows, Mac and Linux will be available from the various CRAN mirrors over the next few days. This update mainly fixes a few minor bugs, and is a drop-in replacement for R 2.15.2.

As announced in January, the next version of R will be R 3.0.0 and will be released in April. No major changes to the language are planned, but internally R will support larger objects to better make use of platforms with large amounts of RAM. This isn't likely to affect R users, but package authors (especially for packages using C code) should test their packages against the development version of R 3.0.0 now to avoid any surprises.

R-announce mailing list: R 2.15.3 is released

R version 3 scheduled for April

Ringing in the New Year, Peter Dalgaard announced yesterday on behalf of the entire R Core Team that the R language will graduate to Version 3 around April 1. This is only the third time that R has incremented its primary version number. Version 1.0.0 (released on February 29, 2000) was the first version deemed stable for production use. R moved to version 2.0.0 on October 4, 2004 once some major language features (the S4 object system) and platforms (MacOS) were established. Version 3.0.0 is scheduled for April 1 this year, and reflects not a major departure from the current 2.15 version, but instead recognizes the establishment of major functionality introduced during the 2.x series. From the announcement:

Major R releases have not previously marked great landslides in terms of new features. Rather, they represent that the codebase has developed to a new level of maturity. This is not going to be an exception to the rule.

Version 1.0.0 was released at a point in time when we felt that we had reached a level of completeness and stability high enough to characterize a full statistical system, which could be put to production use.

Version 2.0.0 came out after strong enhancements of the memory management subsystem as well as several major features, including Sweave.

Version 3.0.0, as of this writing, contains only really major new feature: The inclusion of long vectors (containing more than 2^31-1 elements!). More changes are likely to make it into the final release, but the main reason for having it as a new major release is that R over the last 8.5 years has reached a new level: we now have 64 bit support on all platforms, support for parallel processing, the Matrix package, and much more.

Although many people won't notice the difference, the introduction of long vectors to R is in fact a significant upgrade, and required a lot of work behind-the-scenes to implement in the core R engine. It will allow data frames to exceed their current 2 billion row limit, and in general allow R to make better use of memory in systems with large amounts of RAM. Many thanks go to the R core team for making this improvement.

Also announced was the final release of the 2.15 series: R 2.15.3 is scheduled for release sometime in March. You can read the complete announcement from the R Core Team at the link below.

R-announce mailing list: R version 3.0.0

Applications open for 2013 John M Chambers Statistical Software Award

In 1998, John Chambers won the ACM Software Systems award for the S language, which the ACM heralded as having "forever altered how people analyze, visualize, and manipulate data". The R language is "not unlike S", and John continues pushing the frontiers of statistical computing as a member of the R Core Group. Ang his legacy continues in another way: he donated his ACM prize to a fund for the John M. Chambers Statistical Software Award. Recipients of the award include authors of several R packages, including Michael Kane (for the bigmemory package), Yihui Xie (animation), Hadley Wickham (reshape and ggplot), and Deepayan Sarkar (for lattice, now a part of standard R). 

Applications are now open to individuals or teams for the 2013 prize, which offers $1000 in cash and a travel allowance to the JSM conference in Montreal, where the award will be presented. The official announcement hasn't yet been updated on the web, so I've included the announcement after the jump below.

What's new in Revolution R Enterprise 6.1

We're pleased to announce that the latest update to Revolution R Enterprise is available today! Existing subscribers will soon receive an email with update instructions, and the free academic distribution will be updated later today. Version 6.1 adds a frequently-requested big-data statistical modeling algorithm, adds new connectivity option for Hadoop, improves performance, and provides new security and installation options for IT. Here's a summary of the new features:

• Decision Trees for Big Data. The new “rxDTree” function is a powerful tool for fitting classification and regression trees, which are among the most frequently used algorithms for data analysis and data mining. The implementation provided in Revolution Analytics’ RevoScaleR package is parallelized, scalable, distributable and designed with big data in mind. Revolution R Enterprise continues to offer a wide range of other big-data analysis algorithms, including summary statistics, crosstabs, regression, generalized linear models and K-means clustering. 
• Improved performance for ‘Big Data’ files. RevoScaleR’s ‘XDF’ file format provides fast access to big data. With new compression technology the size of XDF files can be reduced, allowing for higher-performance analytics throughput and faster transfers into clusters or cloud processing systems.
• Improved Linux installer. The installation process on Linux servers has been streamlined to meet stringent IT requirements, especially for non-root installs.
• SiteMinder single-sign for applications: Authorized users of applications built on Revolution R Enterprise deployed via the RevoDeployR Web Services API may authenticate using CA SiteMinder^®.
• Analyze data from Hadoop Distributed File System (HDFS). With more and more data stored in Hadoop, this new option lets data scientists read data from HDFS and apply big-data statistical models from Revolution R Enterprise.

I'm especially excited about this last feature, which makes it possible to feed structured data files in Hadoop directly to the big-data statistical algorithms in the RevoScaleR package, as demonstrated in the video below. It pairs well with the RHadoop project: if you don't have structured data in Hadoop already, use the rmr package and map-reduce to create a structured data file in HDFS, and then analyze it with RevoScaleR.

 

You can find more details about the features in Revolution R Enterprise in this table.

R 2.15.2 now available

As promised, the source distribution for R 2.15.2 is now available for download from the master CRAN repository. (Binary distributions for Windows, MacOS and Linux will be available from the CRAN mirror network in the coming days.) This latest point-update — codenamed "Trick or Treat" — improves the performance of the R engine and adds a few minor but useful features. Detailed changes can be found in the NEWS file, but highlights of the improvements include: 

• New statistical analysis method: Multistratum MANOVA
• Hyman's method of constructing monotonic interpolation splines is now available.
• Improved support for Polish language users
• Functions in the parallel package (such as parLapply) will make use of a default cluster if one is specified.
• Improved performance and reduced memory usage for some commonly-used functions including array, rep, tabulate and hist
• Increased memory available for data on 64-bit systems (increased to 32Gb from 16Gb)
• Several minor bugfixes

There is likely to be at least one further update to the 2.15.x series: a round-up of any further changes will probable be released as 2.15.3 shortly before R 2.16.0 is released, most likely around March 2013.

r-announce mailing list: R 2.15.2 is released

 

Revolution Analytics receives Top Innovator award for Data Science Technology

A big thank-you to all the R users out there who voted for Revolution R Enterprise in DataWeek Awards. We're so pleased to be recognized by the voters and the DataWeek judging panel with the Top Innovator Award for Data Science Technology. We're looking forward to the awards ceremony next week at DataWeek SF (in San Francisco, September 24-27). If you'd like to come along and join us and 1000 of the other most innovative data-centric companies who are attending, speaking, and participating in this first annual event, you can register here with the code DataWeek25 for a 25% discount. 

DataWeek is the first week-long conference and festival to showcase the innovations behind the Data Revolution. The founders of Techweek in Chicago and the Data 2.0 Summit in San Francisco are organizing this year’s DataWeek 2012 Conference & Festival, September 22-27 in San Francisco, CA. Join over 200 speakers from Google, Twitter, IBM, Cloudera, CitrusLeaf, Dun & Bradstreet, and hundreds of other data-centric companies for a 4-day conference inside a week-long series of data events, workshops, hackathons, and hiring mixers, and meetups!

You can find more information about DataWeek and Revolution Analytics' Data Science Technology Top Innovator Award at the link below.

Revolution Analytics Press Releases: Revolution Analytics Wins Top Data Science Technology Innovator Award by DataWeek

R 2.15.1 includes performance improvements inspired by dataframe package

The latest update to open-source R, R 2.15.1, was released this morning. (You can grab sources now, and binary versions will hit the CRAN mirrors over the next couple of days.) In addition to several new features and bug fixes (including the new globalVariables function, which will be a boon to package developers), this update also includes some significant performance improvements inspired by the dataframe package.

At the useR! 2012 conference last week, Google's Tim Hesterberg introduced the dataframe package (available now on CRAN), which has been in use for the last three years amongst Google's 500+ R users. (You can download Tim's PDF slides here.) The package makes no functional changes to R; instead, it improves the implementation of data frames to reduce the number of temporary copies made of data. Tim reported that using the dataframe package with R 2.15.0 improved performance by 21% for creation and column subscripting, and by 14% for row subscripting.

Tim mentioned during the talk that r-core member Luke Tierney was in the process of incorporating performance improvements from the dataframe package into base R, and indeed several such improvements are noted in the NEWS file. All the improvements are devoted to reducing the number of times R makes temporary internal copies of data, which improves both speed and memory usage of R. And because these are low-level improvements at R's core, these improvements will affect many R functions, not just those related to data frames.

If you've built R 2.15.1 already, have you noticed performance improvements? Let us know in the comments.

R-announce mailing list: R 2.15.1 is released

Announcing Revolution R Enterprise 6.0

Revolution Analytics is proud to announce the latest update to our enhanced, production-grade distribution of R, Revolution R Enterprise. This update expands the range of supported computation platforms, adds new Big Data predictive models, and updates to the latest stable release of open source R (2.14.2), which improves performance of the R interpreter by about 30%.

This release expands the range of big-data statistical analysis with support for Generalized Linear Models (GLM). Logistic (Binomial) Poisson, Gamma and Tweedie models are all supported with a high-performance C++ implementation, and you can also model any distribution in the GLM family with a custom link function written in R. Big Data GLM has been a common request from many of our customers, and beta testers have been blown away by the speed of the implementation. For example here's an example of a Tweedie regression on 8.5 million insurance claims in less than 2 and a half minutes (skip ahead to 1:10 for the demo):

 

We can make these computations fast by streaming data through the CPU instead of loading it all at once (which also means no data-size limitations), and by distributing the computations across multiple processors on a server or multiple nodes in a cluster. Revolution R Enterprise 6.0 now supports Linux-based clusters running IBM's Platform LSF grid software, in addition to clusters running Microsoft HPC Server. And with HPC server, you can now run big-data analytics in the Azure Cloud, as shown in this demo.

One other new feature relates to accessing data. Revolution R Enterprise has always included our XDF data format: a high-performance local filestore for Big Data in R. Making a local copy of data in XDF is a very useful if you plan to analyze the same data several times, and want to avoid the network latency of copying the data over and over. But if you don't need a local copy, you can now use the RevoScaleR data step and big-data algorithms on the source data (SAS, SPSS, ASCII or ODBC) directly. (By the way, you don't need a SAS or SPSS license to access data files in those formats, which is useful if you've switched to an R-only environment.) For example, here's an analysis of commuter behavior from 2010 NYC Community Health Survey working directly from the source SAS file:

 

If you're already a subscriber to Revolution R Enterprise, you'll receive an email with upgrade instructions today. (If you'd like to become a subscriber, click here and we'll be happy to help.) As always, academic users can download and use the latest version free of charge.

For more information about Revolution R Enterprise, follow the link below. And if you try out 6.0, let us know what you think in the comments.

Revolution Analytics: Revolution R Enterprise