Revolutions

Ross Ihaka on the history of the R project

Ross Ihaka, one of the co-creators of R (along with Robert Gentleman), recently gave an interview to the University of Auckland's alumni magazine, Ingenio. In the article, he shares the story of the genesis of R in the early 1990s:

The story all began back in the early 1990s when the internet was in its infancy and computers at the University of Auckland were boxy Macintoshes with floppy disks. Ross had done his undergraduate degree at Auckland then his masters and PhD at Berkeley, returning to a lecturing post in the Department of Statistics. Undergraduate students were using what Ross calls “old and clunky programmes” for their data analysis, and he thought there had to be a better way.

Ihaka also expresses some well-deserved pride in the success of R while being humble about his role and that of the other volunteers who worked on the R project:

Ross says that that observing the global labour of love has tempered his cynicism. “R changed my opinion of humanity to some extent, to see how people are really willing to freely give of themselves and produce something larger than themselves without any thought of personal glory. There’s a lot of work with no recognition.”

To read the complete article and learn more about the history of R from its co-creator, follow the link below.

Ingenio: R - the ultimate virus

Get Involved with the R Consortium

by Joseph Rickert

The R Consortium, the non-profit trade organization formed under the Linux Foundation to support the R language and the R Community, is beginning to build real momentum. First of all, two new companies recently joined the Consortium: Avant which provides online personal and auto loans and Procogia, a consulting firm that helps companies make data-driven business decisions. Having companies join the R Consortium that are not directly involved in making R tools and extensions is an exciting development that shows that reach of R and its importance to data-driven business of all types.

Last month the R Consortium announced the results of the first broad call for proposals. The ISC funded 7 projects and 3 working groups. The projects included:

A unified Framework for doing distributed computing in R proposed by Michael Lawrence (an R core member) and Edward Ma and Indrajit Roy was awarded $10K. This project grew out of the workshop on distributed computing that was held at HP in January 2015. The immediate work is to build out the ddR package to include more algorithms and add more "back end" distributed platforms. However, since the project could have significant influence on future work on distributed computing it is also being organized as working group (See below.)

An effort proposed by Kirill Müller to improve the DBI interface was awarded $25K. The general idea is to create a DBI specification and centralized tests and boiler that will make it easier to connect to databases while improving the existing DBI interfaces RMySQL, RPostgress and RSQLite.

The creation of a one-day RIOT (R Implementation, Optimization and Tooling) workshop aimed at identifying R language development and tooling opportunities was awarded $10K. Proposed by Mark Hornick, Lucas Stadler and Adam Welc, it is focused on increasing the involvement of the R Community in this area.

RL10N, a project proposed by Richie Cotton and Thomas Leeper to make it easier to localize R packages was awarded $10K. The work here will focus on improving the msgtools package and developing two new packages: MTurkR for managing local translation and translateR for automated translations.

SatRdays, proposed by Steph Locke and Gergely Daroczi, an effort to create a series of one-day community-led, regional R conferences was awarded $10K.

John Blishak, Jonah Duckles, Laurent Gatto, David LeBauer and Greg Wilson of Software Carpentry were awarded $10K to create a two-day, in-person course to train R instructors. The course is intended to teach already accomplished R programmers to teach R programming to adult learners.            

Edzer Pebesma of the institute for Geoinformatics at University of Muenster was awarded $10K to create an R package and other materials to simplify the R analysis of geospatial data. The idea is comply with the Simple Features standard for the access and manipulation of spatial vector data.

A new development that came out of the recent round of funding is that of organizing collaborative projects as "Working Groups" under the ISC. The idea here is for the ISC to provide the structure for projects of some significance that may not need immediate funding but require consensus and collaboration before progress can be made. As you might expect, the ISC is hesitant to fund any project that has the potential to influence the direction of the development of the R language without achieving some level of consensus from the knowledgeable and interested members of the extended R Community.

These considerations led to an ISC vote to organize a working group around the ddR projects in addition to providing initial funding. Even if the very promising technical approach implemented in ddR turns out to be the preferred way for providing a foundation for distributed computing in R, discussions about what “backends” to build out ought to include representatives from the broader R Community. Should connecting to Spark, for example, be a priority for the R community?

Two additional working groups were also authorized by the ISC.

Adam Welc, Lucas Stadler and Mark Hornick proposed a working group to assess R’s native API. Their proposal asserted the current API is broader than necessary, redundant and, in many cases, tied to the internals of GNUR. The working group will seek consensus from the R Core group, R developers and the greater R Community on designing an easy to understand, consistent and verifiable API.

Shivank Agrawal, Santosh K Chaudhari, Chen Liang, Qin Wang, Vlad Sharanhovich and Mark Hornick to seek direction and consensus for developing tools for R to determine code coverage upon execution of a test suite and to promote code coverage more systematically within the R ecosystem.

I am hopeful that the ISC working groups will be able to foster meaningful discussion and involve the greater R Community in the work of the ISC.

Finally, Gábor Csárdi is making very good progress with R-Hub, a platform for building, testing and validating R packages. This major project, funded last year with an initial budget of $80K, should streamline and improve the package building experience when completed. The following figure from a recent presentation the R Consortium’s board illustrates the R-Hub Architecture. The part in green indicates the prototype structures that have been built out.

For the details, check out the project on Github.

Having been a member of the R Consortium’s Infrastructure Steering Committee (ISC) for nearly a year now, it is clear to me that, in the long run, the success of the R Consortium will hinge on three factors: (1) the impact of the projects it funds, (2) the ability of the ISC to guide, and in the case of working groups, help manage projects to successful completions and (3) the interest and participation of the greater R Community.                          

We will know if we are succeeding at this last factor if we can attract the Community experts into the working groups and continue to receive proposals for high quality projects.

Here is my take on problems that the ISC perceived with proposals that were rejected during the last call for proposals and some ideas about formulating a fundable project.

Projects that were rejected displayed one, or more of the of the following shortcomings. They were either: too specific, too small, too ambitious for resources requested, too open ended or proposed to do something that already exists or has been partially implemented somewhere else.

On the flip side, here are some ideas that you might find helpful in writing a proposal for this next round of funding:

• Think big: shoot for something that will benefit a sizable portion of the R Community for years to come
• Collaborate: seek expert opinion about your ideas and find potential collaborators
• Do your homework: make sure you understand what relevant work already exists
• Write a detailed proposal and carefully estimate work, resources and money required
• Be careful what you ask for!

If you think that you have an idea for a project that can benefit a significant portion of the R Community, please find some collaborators, strap yourselves in and start writing. The window for submitting a proposal for the next round of funding closes on July 10th. To submit a proposal or find more information, please visit https://www.r-consortium.org/about/isc/proposals or email [email protected].

Here is the link to the mailing lists for the ISC working groups.

SQL Server 2016 launch showcases R

Microsoft officially launched SQL Server 2016 at the Data Driven event in New York City last week, and R featured prominently.

One of the highlights from Joseph Sirosh's keynote was a demonstration of the World Wide Telescope, an online tool that allows budding astronomers (and professionals!) to explore the visible universe using imagery from the Digitized Sky Survey. The Survey contains images from thousands of galaxies that have not yet been classified by astronomers into one of the standard forms: elliptical, spiral, or lenticular. In the demo, the World Wide Telescope automatically classifies the galaxies in mere seconds:

 

The automated classification was based on a random forests model, using the manually-classified galaxies in the database as the training set. The model was trained and run using R (specifically, SQL Server R Services) running within SQL Server 2016. You can see the R code embedded within the T-SQL used in SQL Server in the screenshot below.

Notably, R wasn't the only open-source techology featured in this demo. In fact, the demo was running SQL Server on Linux, which is in preview now and will be available in 2017.

If you'd like to explore R and SQL Server in more detail, the launch also saw the release of a number of in-depth videos featuring many of the developers involved in the project. (While I had a small cameo in the Scott Guthrie's keynote presentation on some of the team involved in the project, the names below deserve much more credit for the R integration.) Click on the links below for the videos:

• Scaling your advanced analytics in SQL Server 2016 with R Server: Bill Jacobs.
• R Services in SQL Server 2016: Dotan Elharrar and UC Jayachandran.
• R Services FAQ SQL Server 2016: Dotan Elharrar, Vijay Jayaseelan, Jasraj Dange, and UC Jayachandran.
• Building intelligent applications using SQL Server R services: an End to End Walkthrough, Gopi Kumar and Hang Zhang.

Official Microsoft Blog: SQL Server 2016: The database for mission-critical intelligence 

 

R Tools for Visual Studio preview now available

It's official! R Tools for Visual Studio, until now only available as a private preview, is now in public preview and available to everyone as a free, open-source download.

RTVS is an add-in for Microsoft Visual Studio, which adds R language development capabilities to the popular Windows-based IDE. If you don't already have Visual Studio, you can download Visual Studio Community for free. Then, download and install RTVS to add the "R Tools" menu to Visual Studio.

RTVS in action. Weather chart R code adapted from Bradley Boehmke.

To get started with R in Visual Studio, just create a new project and choose the "R" template. To interact with R directly, you can type commands directly to the R command line while recalling and editing previous commands. Intellisense tab-completion makes it easy to find and type commands, and get quick access to R documentation. R objects you create appear in the Variable Explorer window, and charts appear in the R Plot window. You can create and edit R scripts using a syntax-aware folding editor, and set breakpoints and step into running code with the R code debugger. Once your R script or RMarkdown document is working, you can check it into Git or GitHub. And because you're working within the Visual Studio environment you have access to all of its standard features as well, including the ability to simultaneously edit code in Python, C++, or any other supported language, use any Visual Studio Extension, and have complete control over appearance, window placement, and key bindings (even Vim!). 

This preview release is just the beginning: we have plenty of plans to extend RTVS further. Requests from our private preview include support for creating R packages within RTVS, and support for non-Windows platforms. (This release requires the Windows-based Visual Studio, and doesn't support the cross-platform Visual Studio Code IDE.) If you have preferences or suggestions for features we can add, please take the RTVS Survey, or submit an issue (or even a pull request!) to the RTVS Github repository. (RTVS is released under the MIT open-source license.) And if you encounter any issues with the preview release (or just want to send comments), you can provide a "smile" or "frown" via the Feedback menu in the R Tools menu.

If you'd like to check out a tour of RTVS in action, RTVS's product manager John Lam presents this lap around the features. 

 

Get started with R Tools for Visual Studio here, or for more information check out the official blog post by Shahrokh Mortavazi linked below.

Microsoft Machine Learning Blog: R Tools for Visual Studio Announced

H2O World 2015

by Joseph Rickert

The second, annual H2O World conference finished up yesterday. More than 700 people from all over the US attended the three-day event that was held at the Computer History Museum in Mountain View, California; a venue that pretty much sits well within the blast radius of ground zero for Data Science in the Silicon Valley. This was definitely a conference for practitioners and I recognized quite a few accomplished data scientists in the crowd. Unlike many other single-vendor productions, this was a genuine Data Science event and not merely a vendor showcase. H2O is a relatively small company, but they took a big league approach to the conference with an emphasis on cultivating the community of data scientists and delivering presentations and panel discussions that focused on programming, algorithms and good Data Science practice.

The R based sessions I attended on the tutorial day were all very well done. Each was designed around a carefully crafted R script performing a non-trivial model building exercise and showcasing one or more of the various algorithms in the H2O repertoire including GLMs, Gradient Boosting Machines, Random Forests and Deep Learning Neural Nets. The presentations were targeted to a sophisticated audience with considerable discussion of pros and cons. Deep Learning is probably H2O's signature algorithm, but despite its extremely impressive performance in many applications nobody here was selling it as the answer to everything. 

The following code fragment from a script (Download Deeplearning)that uses deep learning to identify a spiral pattern in a data set illustrates the current look and feel of H2O's R interface. Any function that begins with h2o. runs in the JVM not in the R environment. (Also note that if you want to run the code you must first install Java on your machine, the Java Runtime Environment will do. Then, download the H2O R package Version 3.6.0.3 from the company's website. The scripts will not run with the older version of the package on CRAN.)

### Cover Type Dataset
#We important the full cover type dataset (581k rows, 13 columns, 10 numerical, 3 categorical).
#We also split the data 3 ways: 60% for training, 20% for validation (hyper parameter tuning) and 20% for final testing.
#
df <- h2o.importFile(path = normalizePath("../data/covtype.full.csv"))
dim(df)
df
splits <- h2o.splitFrame(df, c(0.6,0.2), seed=1234)
train <- h2o.assign(splits[[1]], "train.hex") # 60%
valid <- h2o.assign(splits[[2]], "valid.hex") # 20%
test <- h2o.assign(splits[[3]], "test.hex") # 20%
#
#Here's a scalable way to do scatter plots via binning (works for categorical and numeric columns) to get more familiar with the dataset.
#
#dev.new(noRStudioGD=FALSE) #direct plotting output to a new window
par(mfrow=c(1,1)) # reset canvas
plot(h2o.tabulate(df, "Elevation", "Cover_Type"))
plot(h2o.tabulate(df, "Horizontal_Distance_To_Roadways", "Cover_Type"))
plot(h2o.tabulate(df, "Soil_Type", "Cover_Type"))
plot(h2o.tabulate(df, "Horizontal_Distance_To_Roadways", "Elevation" ))
#
#### First Run of H2O Deep Learning
#Let's run our first Deep Learning model on the covtype dataset. 
#We want to predict the `Cover_Type` column, a categorical feature with 7 levels, and the Deep Learning model will be tasked to perform (multi-class) classification. It uses the other 12 predictors of the dataset, of which 10 are numerical, and 2 are categorical with a total of 44 levels. We can expect the Deep Learning model to have 56 input neurons (after automatic one-hot encoding).
#
response <- "Cover_Type"
predictors <- setdiff(names(df), response)
predictors
#
#To keep it fast, we only run for one epoch (one pass over the training data).
#
m1 <- h2o.deeplearning(
model_id="dl_model_first", 
training_frame=train, 
validation_frame=valid, ## validation dataset: used for scoring and early stopping
x=predictors,
y=response,
#activation="Rectifier", ## default
#hidden=c(200,200), ## default: 2 hidden layers with 200 neurons each
epochs=1,
variable_importances=T ## not enabled by default
)
summary(m1)
#
#Inspect the model in [Flow](http://localhost:54321/) for more information about model building etc. by issuing a cell with the content `getModel "dl_model_first"`, and pressing Ctrl-Enter.
#
#### Variable Importances
#Variable importances for Neural Network models are notoriously difficult to compute, and there are many [pitfalls](ftp://ftp.sas.com/pub/neural/importance.html). H2O Deep Learning has implemented the method of [Gedeon](http://cs.anu.edu.au/~./Tom.Gedeon/pdfs/ContribDataMinv2.pdf), and returns relative variable importances in descending order of importance.
#
head(as.data.frame(h2o.varimp(m1)))
#
#### Early Stopping
#Now we run another, smaller network, and we let it stop automatically once the misclassification rate converges (specifically, if the moving average of length 2 does not improve by at least 1% for 2 consecutive scoring events). We also sample the validation set to 10,000 rows for faster scoring.
#
m2 <- h2o.deeplearning(
model_id="dl_model_faster", 
training_frame=train, 
validation_frame=valid,
x=predictors,
y=response,
hidden=c(32,32,32), ## small network, runs faster
epochs=1000000, ## hopefully converges earlier...
score_validation_samples=10000, ## sample the validation dataset (faster)
stopping_rounds=2,
stopping_metric="misclassification", ## could be "MSE","logloss","r2"
stopping_tolerance=0.01
)
summary(m2)
plot(m2)

First notice that it all looks pretty much like R code. The script mixes standard R functions and H2O functions in a natural way. For example, h20.tabulate() produces an object of class "list" and h20.deeplearning() yields a model object that plot can deal with. This is just really baseline stuff that has to happen to provide make H2O coding feel like R. But note that the H2O code goes beyond this baseline requirement. The functions h2o.splitFrame() and  h2o.assign() manipulate data residing in the JVM in a way that will probably seem natural to most R users, and the function signatures also seem to be close enough "R like" to go unnoticed. All of this reflects the conscious intent of the H2O designers not only to provide tools to facilitate the manipulation of H2O data from the R environment, but also to try and replicate the R experience.

An innovative new feature of the h20.deeplearning() function itself is the ability to specify a stopping metric. The parameter setting: (stopping_metric="misclassification", ## could be "MSE","logloss","r2" ) in the specification of model m2 means that the neural net will continue to learn until the specified performance threshold is achieved. In most cases, this will produce a useful model in much less time than it would take to have the learner run to completion. The following plot, generated in the script referenced above, shows the kind of problem for which the Deep Learning algorithm excels.

Highlights of the conference for me included the presentations listed below. The videos and slides (when available) from all of these presentations will be posted on the H2O conference website. Some have been posted already and the rest should follow soon. (I have listed the dates and presentation times to help you locate the slides when they become available)

Madeleine Udell (11-11: 10:30AM)  presented the mathematics underlying the new algorithm, Generalized Low Rank Models (GLRM), she developed as part of her PhD work under Stephen Boyd, professor at Stanford University and adviser to H2O. This algorithm which generalizes PCA to deal with heterogeneous data types shows great promise for a variety of data science applications. Among other things, it offers a scalable way to impute missing data. This was possibly the best presentation of the conference. Madeleine is an astonishingly good speaker; she makes the math exciting.

Anqi Fu (11-9: 3PM)  presented her H2O implementation of the GLRM. Anqi not only does a great job of presenting the algorithm, she also offers some real insight into the challenges of turning the mathematics into production level code.  You can download one of Anqi's demo R scripts here: Download Glrm.census.labor.violations. To my knowledge, Anqi's code is the only scalable implementation of the GLRM. (Madeleine wrote the prototype code in Julia.)

Matt Dowle (11-10), of data.table fame, demonstrated his port of data.table's lightning fast radix sorting algorithm to H2O. Matt showed a 1B row X 1B row table join that runs in about 1.45 minutes on a 4 node 128 core H2O cluster. This is very impressive result, but Matt says he can already do 10B x 10B row joins, and is shooting for 100B x 100B rows.

Professor Rob Tibshirani (11-11: 11AM) presented work he is doing that may lead to lasso based models capable of detecting the presence of cancer in tissue extracted from patients while they are on the operating table! He described "Customized Learning", a method of building individual models for each patient. The basic technique is to pool the data from all of the patients and run a clustering algorithm. Then, for each patient fit a model using only the data in the patient's cluster. This is exciting work with the real potential to save lives.

Professor Stephen Boyd (11-10: 11AM) delivered a tutorial on optimization starting with basic convex optimization problems and then went on to describe Consensus Optimization, an algorithm for building machine learning models from data stored at different locations without sharing the data among the locations. Professor Boyd is a lucid and entertaining speaker, the kind of professor you will wish you had had.

Arno Candel (11-9: 1:30PM) presented the Deep Learning model which he developed at H2O. Arno is an accomplished speaker who presents the details with great clarity and balance. Be sure to have a look at his slide showing the strengths and weaknesses of Deep Learning. 

Erin LeDell (11-9: 3PM) de-mystified ensembles and described how to build an ensemble learner from scratch. Anyone who wants to compete in a Kaggle competition should find this talk to be of value.

Szilard Pafka (11-11:3PM), in a devastatingly effective, low key presentation, described his efforts to benchmark the open source, machine learning platforms R, Python scikit, Vopal Wabbit, H2O, xgboost and Spark MLlib. Szilard downplayed his results, pointing out that they are in no way meant to be either complete nor conclusive. Nevertheless, Szilard put some considerable effort into the benchmarks. (He worked directly with all of the development teams for the various platforms.) Szilard did not offer any conclusions, but things are not looking all that good for Spark. The following slide plots AUC vs file size up to 10M rows.

Szilard's presentation should be available on the H2O site soon, but it is also available here.

I also found the Wednesday morning panel discussion on the "Culture of Data Driven Decision Making" and the Wednesday afternoon panel on "Algorithms -Design and Application" to be informative and well worth watching. Both panels included a great group of articulate and knowledgeable people.

If you have not checked in with H2O since the post I wrote last year, here' on one slide, is some of what they have been up to since then. 

 Congratulations to H2O for putting on a top notch event!

Using the dplyrXdf package

by Hong Ooi
Sr. Data Scientist, Microsoft

In a previous post I introduced the dplyrXdf package, which combines the ease of use and simplicity of dplyr with the scalability of Revolution Analytics' xdf data format. Most of dplyrXdf works the same way as dplyr, so if you know how to use the latter, then you also (mostly) know how to use the former. However, there are some places in which the two packages are different. These will be described below.

Single-table verbs

As with the previous post, I’ll use the nycflights13 flights data to demonstrate the use of dplyrXdf.

library(dplyrXdf)  # also loads dplyr
library(nycflights13)
 
# write the data as an xdf file
flightsXdf <- rxDataFrameToXdf(flights, "flights.xdf", overwrite=TRUE)

The .rxArgs parameter

The RevoScaleR functions typically have several arguments beyond those used by dplyrXdf verbs. While usually you don’t need to touch these, it can sometimes be useful to do so. For example, when using mutate or transmute, you could specify more complicated transformations via a transformFunc (see the help for rxTransform). Similarly, rather than chaining together a mutate and a summarise — which would involve creating an intermediate file — you could incorporate the variable transformation into the summarise itself. More low-level uses of such arguments include setting the block size for an xdf file, changing the compression level, limiting the number of rows, and so on.

Most of the one-table dplyrXdf verbs accept an .rxArgs argument as a way of transmitting these extra arguments to the underlying RevoScaleR code. This should be a named list specifying the names and values of the arguments to be passed. The exact arguments will vary depending on the verb in question; here is a list of the verbs and the underlying RevoScaleR function that they call:

• filter and select: rxDataStep
• mutate and transmute: rxDataStep
• summarise: depending on the method chosen, rxCube or rxSummary
• arrange: rxSort
• distinct: rxDataStep
• factorise: rxFactors
• doXdf: rxDataStep

You should use the .rxArgs argument with caution, as some verbs may modify the data as part of their normal functioning, so the results you get back may not be as expected. It’s also easy to write convoluted code that makes your dplyrXdf pipelines harder to read. However, when working with big datasets this feature can help save a lot of processing time by avoiding unnecessary disk traffic.

See the help for each of the RevoScaleR functions for which arguments they support, and that you can pass via.rxArgs.

Transforming data with mutate and transmute

These verbs work much the same way as in dplyr. You can also use the .rxArgs parameter to specify additional transformation arguments to rxDataStep. Here are some examples to illustrate the use of mutate and transmute:

Introducing the dplyrXdf package

by Hong Ooi
Sr. Data Scientist, Microsoft

The dplyr package is a popular toolkit for data transformation and manipulation. Over the last year and a half, dplyr has become a hot topic in the R community, for the way in which it streamlines and simplifies many common data manipulation tasks.

Out of the box, dplyr supports data frames, data tables (from the data.table package), and the following SQL databases: MySQL/MariaDB, SQLite, and PostgreSQL. However, a feature of dplyr is that it’s extensible: by writing a specific backend, you can make it work with many other kinds of data sources. For example the development version of the RSQLServer package implements a dplyr backend for Microsoft SQL Server.

The dplyrXdf package implements such a backend for the xdf file format, a technology supplied as part of Revolution R Enterprise. All of the data transformation and modelling functions provided with Revolution R Enterprise support xdf files, which allow you to break R’s memory barrier: by storing the data on disk, rather than in memory, they make it possible to work with multi-gigabyte or terabyte-sized datasets.

dplyrXdf brings the benefits of dplyr to xdf files, including support for pipeline notation, all major verbs, and the ability to incorporate xdfs into dplyr pipelines. It also provides some additional benefits which are more specific to working with xdfs:

• The RevoScaleR functions require keeping track of where your data is saved. In some situations, writing a function’s output to the same file as its input is allowed, while in others, it causes problems. You can often end up with many different version of the data scattered around your filesystem, introducing reproducibility problems and making it difficult to keep track of changes. dplyrXdf abstracts this task of file management away, so that you can focus on the data itself.

• Related to the above, the source xdf to a dplyrXdf pipeline is never modified. This provides a measure of security, so that even if there are bugs in your code (maybe you meant to use a mutate rather than a transmute), the original data is safe.

• Consistency of interface: functions like rxCube and rxSummary use formulas in different ways, because they are designed to do slightly different things. Similarly, many RevoScaleR functions use factors but don’t automatically create those factors; or they require handholding when trying to combine factor with non-factor data. With dplyrXdf, you don’t have to remember which formula syntax goes with which function, or create factors yourself. If you do have to create factors, it provides a new verb (factorise) to streamline this as well.

• The verbs in dplyrXdf all read from xdf files and write to xdf files. The data is thus never read entirely into memory, so a dplyrXdf pipeline will work with datasets that are arbitrarily large.

Obtaining dplyrXdf

The package is available for download from Github, at https://github.com/RevolutionAnalytics/dplyrXdf. If you have the devtools package installed, you can download and install it from within R using the command devtools::install_github("Hong-Revo/dplyrXdf").

Note that dplyrXdf is a shell on top of the functionality provided by Revolution R Enterprise, which is the commercial (paid) distribution of R from Revolution Analytics. You must be an RRE customer to make use of dplyrXdf. It will not work with Revolution R Open or standard open source R, as these distributions don’t support xdf files.

A sample dplyrXdf pipeline

For this example I’ll use the flights dataset from the nycflights13 package. This is one of the datasets used in the dplyr vignettes, and crops up in many other places besides.

library(dplyrXdf)  # also loads dplyr
library(nycflights13)
 
# write the data as an xdf file
flightsXdf <- rxDataFrameToXdf(flights, "flights.xdf", overwrite=TRUE)

Consider a simple task: get the average delay and total distance covered (in kilometers) in the first half of 2013, by carrier, sorted by descending delay. This isn’t very complicated, conceptually speaking: we want to do a row selection, then some basic transformations, followed by a summary, and then order the output by one of the columns.

This translates into the following sequence of RevoScaleR function calls:

# select the rows
flights_rx1 <- rxDataStep(flightsXdf, outFile="flights_rx1.xdf",
                          rowSelection=month <= 6 & year == 2013,
                          overwrite=TRUE)
 
# variable transformations
flights_rx2 <- rxDataStep(flights_rx1, outFile="flights_rx2.xdf",
                          transforms=list(dist_km=distance*1.6093,
                                          delay=(arr_delay + dep_delay)/2),
                          overwrite=TRUE)
 
# convert carrier into a factor variable (or rxSummary will complain)
flights_rx3 <- rxFactors(flights_rx2, factorInfo="carrier",
                         outFile="flights_rx3.xdf", overwrite=TRUE)
 
# use rxSummary to get the summary table(s) (could also use rxCube twice)
flights_rx4 <- rxSummary(~delay:carrier + dist_km:carrier, data=flights_rx3,
                         summaryStats=c("mean", "sum"))
 
# extract the desired tables from the rxSummary output
flights_rx4_1 <- flights_rx4$categorical[[1]][c("carrier", "Means")]
names(flights_rx4_1)[2] <- "mean_delay"
 
flights_rx4_2 <- flights_rx4$categorical[[2]][c("carrier", "Sum")]
names(flights_rx4_2)[2] <- "sum_dist"
 
# merge the tables together
flights_rx5 <- merge(flights_rx4_1, flights_rx4_2, by="carrier", all=TRUE)
 
# sort the results
flights_rx5 <- flights_rx5[order(flights_rx5$mean_delay, decreasing=TRUE), ]
 
head(flights_rx5)
#>    carrier mean_delay     sum_dist
#> 11      OO   72.16667     2750.294
#> 7       F9   24.81886   873367.110
#> 6       EV   21.80613 23400472.426
#> 16      YV   20.78378   128401.219
#> 8       FL   15.61127  1979744.767
#> 1       9E   13.85877  7160918.303

The equivalent in dplyrXdf would be the following pipeline:

flightsSmry <- flightsXdf %>%
    filter(month <= 6, year == 2013) %>%
    mutate(dist_km=distance*1.6093, delay=(arr_delay + dep_delay)/2) %>%
    group_by(carrier) %>%
    summarise(mean_delay=mean(delay), sum_dist=sum(dist_km)) %>%
    arrange(desc(mean_delay))
 
head(flightsSmry)
#>   carrier mean_delay     sum_dist
#> 1      OO   72.16667     2750.294
#> 2      F9   24.81886   873367.110
#> 3      EV   21.80613 23400472.426
#> 4      YV   20.78378   128401.219
#> 5      FL   15.61127  1979744.767
#> 6      9E   13.85877  7160918.303

Even with this very straightforward example, dplyrXdf hides the complexity of calling RevoScaleR functions while retaining their power. In particular, note the following:

• There is no need to keep track of input and output file locations: the verbs in the dplyrXdf pipeline will automatically create files and reuse them as needed. Files that are no longer used will be deleted, so there won’t be multiple orphaned files cluttering up your hard disk.

• The summarise verb is much simpler to work with than the RevoScaleR rxSummary function. It doesn’t require scanning through a list of output objects to find the information you’re after, and it accepts grouping variables of any type (numeric, character or factor).

• The pipeline notation makes it clear at a glance what is the sequence of operations being carried out. This is one of the major benefits of dplyr, and is now also available for those working with xdf files.

Single-table verbs

dplyrXdf supports all the basic dplyr single-table verbs:

• filter and select to choose rows and columns

• mutate and transmute to do data transformation

• group_by to define groups

• summarise and do to carry out computations on grouped data

• arrange to sort by variables

• rename to rename columns

• distinct to drop duplicates

Under the hood, they work by translating your pipeline into calls to the base RevoScaleR functions for working with xdf files: for example, mutate calls rxDataStep to compute transformations; arrange calls rxSort, and so on.

Most of these verbs work exactly as they do in dplyr. Thus if you know how to use dplyr, then you also know how to use the bulk of dplyrXdf.

Two-table verbs

dplyrXdf supports the main table-join verbs from dplyr: left_join, right_join, inner_join and full_join. The syntax is the same as for the dplyr versions, including joining on non-matching column names. The underlying implementation uses rxMerge with the appropriate arguments for each type of join.

For example, one of the joins in the dplyr two-table verbs vignette joins the flights table with the airports table, using the columns dest (in flights) and faa (in airports). The same code in dplyr also works in dplyrXdf:

airportsXdf <- rxDataFrameToXdf(airports, "airports.xdf", overwrite=TRUE)
flightsJoin <- left_join(
    flightsXdf %>% select(year:day, hour, origin, dest, tailnum, carrier),
    airportsXdf,
    by=c("dest"="faa"))

head(flightsJoin)
#>   year month day hour origin dest tailnum carrier faa
#> 1 2013    10   1   19    JFK  ABQ  N554JB      B6 ABQ
#> 2 2013    10   2   20    JFK  ABQ  N607JB      B6 ABQ
#> 3 2013    10   3   19    JFK  ABQ  N591JB      B6 ABQ
#> 4 2013    10   4   20    JFK  ABQ  N662JB      B6 ABQ
#> 5 2013    10   5   19    JFK  ABQ  N580JB      B6 ABQ
#> 6 2013    10   6   19    JFK  ABQ  N507JB      B6 ABQ
#>                                name      lat       lon  alt tz dst
#> 1 Albuquerque International Sunport 35.04022 -106.6092 5355 -7   A
#> 2 Albuquerque International Sunport 35.04022 -106.6092 5355 -7   A
#> 3 Albuquerque International Sunport 35.04022 -106.6092 5355 -7   A
#> 4 Albuquerque International Sunport 35.04022 -106.6092 5355 -7   A
#> 5 Albuquerque International Sunport 35.04022 -106.6092 5355 -7   A
#> 6 Albuquerque International Sunport 35.04022 -106.6092 5355 -7   A

Tbls and file management

To facilitate the task of file management, dplyrXdf defines a new tbl_xdf class that extends the RxXdfData class. This is what allows it to keep track of which data sources should remain untouched, and which can be modified or overwritten as part of a pipeline. To the base RevoScaleR functions, a tbl_xdf object is just a normal xdf data source; thus, existing code dealing with xdfs should work with minimal modification. However, the verbs implemented in dplyrXdf will recognise when they are passed a tbl_xdf, as opposed to a normal xdf, in which case they will delete their input file after writing the output file. Thus there is always only one file that represents the latest stage of a pipeline.

To create a tbl_xdf yourself, just call the tbl function on an xdf data source:

flightsTbl <- tbl(flightsXdf)
flightsTbl
#> tbl_xdf Source
#> "C:\Users\hongooi\Documents\revoDplyr\blog\flights.xdf"
#> fileSystem: 
#>     fileSystemType: native

However this should rarely (if ever) be necessary, as dplyrXdf will automatically create a tbl for you if you pass a raw xdf to a pipeline.

dplyrXdf creates its output files in R’s temporary directory, so when you close your R session, these files will be deleted. This saves you having to manually delete files that are no longer in use, but it does mean that you must copy the output of your pipeline to a permanent location if you want to keep it around. You can use file.copy and rxXdfFileName for this purpose:

# same dplyrXdf pipeline from before
flightsSmry <- flightsXdf %>%
    filter(month <= 6, year == 2013) %>%
    mutate(dist_km=distance*1.6093, delay=(arr_delay + dep_delay)/2) %>%
    group_by(carrier) %>%
    summarise(mean_delay=mean(delay), sum_dist=sum(dist_km)) %>%
    arrange(desc(mean_delay))
 
# store the result of the pipeline where it won't get deleted
outFile <- "./flightsSmry.xdf"
file.copy(rxXdfFileName(flightsSmry), outFile, overwrite=TRUE)

Non-xdf and non-local data sources

Despite the name, dplyrXdf supports all file data sources defined by RevoScaleR, not just xdf files. This includes delimited text (RxTextData), SAS datasets (RxSasData) and SPSS datasets (RxSpssData). If you pass one of these data sources to a dplyrXdf pipeline, it will import the data to an xdf file first before executing the rest of the pipeline.

For the moment, dplyrXdf only supports files stored in the local filesystem. Support for datasets stored in HDFS (using the RxHdfsFileSystem class) may appear in a future version. For data stored in a SQL database, consider using the dplyr backend for that database, if available.

Conclusion

This article has been a quick executive-summary introduction to dplyrXdf. In a following article, I’ll discuss the differences between dplyr and dplyrXdf; while most verbs work the same way in both packages, there will be areas where they diverge (mostly due to the structural difficulties of working with big data).

Like dplyr, dplyrXdf is a package under active development. If you have any suggestions on features to add (including bits of dplyr that have been left out) or bugs that need fixing, please contact me at [email protected].

R User Groups Highlight R Creativity

by Joseph Rickert

I have been a big fan of R user groups since I attended my first meeting. There is just something about the vibe of being around people excited about what they are doing that feels good. From a speaker's perspective, presenting at an R user Group meeting must be the rough equivalent of doing "stand-up" at a club where you know mostly everyone and you are pretty sure people are going to like your material. So while user groups don't necessarily ignite R creativity (people don't do their best work just to present at an R User group meeting), they do help to shine the spotlight on some really good stuff.

I attend all of the Bay Area useR group meetings, and quite a few other R related events throughout the year, but I only get to experience a small fraction of what is going on in the R world. In the spirit of sharing the "wish I was there" feeling, here are a few recent user group presentations from around the globe that look like they were informative, entertaining and motivating.

Tommy O'Dell gave "Welcome to dply" talk to the Western Australia R Group (WARG) on September 10th. This is a very good presentation until near the very end when it becomes an absolutely great presentation!! Apparently, motivated by a desire to use dplyr with R 2.12, an older R version of R not supported by dplyr, Tommy deconstructed the dplyr "magic" to write his own package, rdplyr. This is a wonderful example of how curiosity and open source can open up many possibilities. The following slide comes from the section where Tommy explains some of the problems he encountered and how he worked through them.

On the 16th of September, Kevin Little gave a talk to MadR about how he recovered after "hitting the wall" in failed first attempt to interface to the SurveyMonkey API using the Rmonkey package. Kevin's description of how he worked through the process which included wading into some JSON scripting is a motivational case study. Kevin wrote a blog post that provides background for the project and has made his slides available here.

Also in September Jim Porzak, a long-time contributor to the San Francisco Bay Area R community, described a detailed customer segmentation analysis in a presentation to BARUG. The following slide examines the stability of the clusters.

Finally, there is a small treasure trove of relatively recent work at the BaselR presentations page. These include a presentation from Aimee Gott on the Mango Solutions development environment and one from Anne Kuemmel on using simulations to calculate confidence intervals in pharma applications. Also have a look at Daniel Sabanes Bove's presentation on using R to produce Microsoft PowerPoint presentations, and some thoughtful advice from Reinhold Koch on how to go about creating a lively R community within your company.

_______________________________________________________________________________________

_______________________________________________________________________________________

 Let us all adopt this mindset!!

The R Consortium Gears Up For Business

by Joseph Rickert

This week, the Infrastructure Steering Committee (ISC) of the R Consortium unanimously elected Hadley Wickham as its chair thereby also giving Hadley a seat on the R Consortium board of directors. Congratulations Hadley!!

This is a major step forward towards putting the R Consortium in business. Not only is the ISC the group that will decide on what projects the R Consortium will undertake, but it will also be responsible for actually getting the work done. (Look here for the charter of the ISC. )

The whole process of funding, soliciting, selecting and executing projects will work something like this: The board of directors under the leadership of its chair, Richard Pugh of Mango Solutions, will establish a budget for projects. The ISC will solicit proposals for new projects both from R Consortium member companies and from the R Community at large.  With approval from the board, the ISC will decide which projects to fund. From there on, the ISC will assemble resources and manage the work. That’s the plan. The devil, of course, is in the details. There is much work to be done to put all of the necessary infrastructure in place, but Hadley’s election makes it possible for the ISC to begin bootstrapping the process.

So, while there is currently no formal proposal process in place, and the ISC and the R Consortium are not ready to begin the process of soliciting proposals from the public, it is not too early for the R Community to begin thinking about what work needs to be done. Now, is the time to begin thinking on a grand scale; well, at least on a scale that might be a bit more ambitious than creating a single R package.

What type of project might make the cut? I don’t want to set up any constraints here, or limit possibilities. But, just to pick one application area, it seems to me that that there were more than a few ideas kicked around in the HP Workshop on Distributed Computing in R held earlier this year that could be formulated into exciting and important projects. How about a unified interface for distributed computing in R?

If you have an idea for a project that you think would benefit the general R community but is more complicated than writing a simple package please start thinking about how you would write up your ideas, elaborating on the benefits to the R Community, technical feasibility, required resources etc. And, stayed tuned to R Consortium Announcements for information on when the proposal process will begin.

I’ll finish here by congratulating Hadley one more time, and state that I am very pleased to have the opportunity to work with him and the other members of the committee. I expect that with Hadley’s technical leadership, the guidance of the board of directors, and the participation of committed R users that the R Consortium will become an effective advocate and source of support for the R Community.

You can write to the ISC at:  [email protected]

Some facts about the R Consortium
Founded: June 19, 2015
Status: The R Consortium is organized as a Linux Foundation Collaborative Project
Member organizations: Alteryx, Google, Hewlett Packard, Ketchum Trading, Mango Solutions, Microsoft, Oracle, The R foundation, RStudio and Tibco
Board of Directors: David Smith (Microsoft), Hadley Wickham (RStudio), John Chambers (R Foundation), J.J. Allaire (RStudio), Louis Bajuk-Yorgan (Tibco) and Chair, Richard Pugh (Mango Solutions)
ISC Members: Hadley Wickham (RStudio), Joseph Rickert (Microsoft), Luke Tierney (R Foundation) and Stephen Kaluzny (Tibco)

 

 

 

Watch 18 years of R development in 15 minutes

R is an incredibly active software project. Since the first code was checked into Subversion back on 18 September 1997, there have been more than 100,000 updates to the R sources by the 20-some members of the R Core Team. As you can see from the activity graphs from the GitHub mirror of the R sources, the pace of development has been frequent and steady over the lifetime of the project:

It's hard to get a sense of the pace of activity from that chart alone, but Matt Dowle (he of the data.table package) published this mesmerizing visualization of R development to YouTube:

 

Matt created this visualization using Gource, using the aforementioned R GitHub mirror as a data source. It's cool to see all of the R developers moving around the source code tree as it grows over time. If you'd like to drill down into the individual contributors, Romain Francois has some R code and charts (created in 2009) that do exactly that.