I’m delighted to announce the release of version 1.0.0 of the dplyrXdf package. dplyrXdf began as a simple (relatively speaking) backend to dplyr for Microsoft Machine Learning Server/Microsoft R Server’s Xdf file format, but has now become a broader suite of tools to ease working with Xdf files.
This update to dplyrXdf brings the following new features:
- Support for the new tidyeval framework that powers the current release of dplyr
- Support for Spark and Hadoop clusters, including integration with the sparklyr package to process Hive tables in Spark
- Integration with dplyr to process SQL Server tables in-database
- Simplified handling of parallel processing for grouped data
- Several utility functions for Xdf and file management
- Workarounds for various glitches and unexpected behaviour in MRS and dplyr
Spark, Hadoop and HDFS
New in version 1.0.0 of dplyrXdf is support for Xdf files and datasets stored in HDFS in a Hadoop or Spark cluster. Most verbs and pipelines behave the same way, whether the computations are taking place in your R session itself, or in-cluster (except that they should be much more scalable in the latter case). Similarly, dplyrXdf can handle both the scenarios where your R session is taking place on the cluster edge node, or on a remote client.
For example, here is some sample code where we extract a table from Hive, then create a pipeline to process it in the cluster:
If you are logged into the edge node, dplyrXdf also has the ability to call sparklyr to process Hive tables in Spark. This can be more efficient than converting the data to Xdf format, since less I/O is involved. To run the above pipeline with sparklyr, we simply omit the step of creating an Xdf file:
SQL database support
One of the key strengths of dplyr is its ability to interoperate with SQL databases. Given a database table as input, dplyr can translate the verbs in a pipeline into a SQL query which is then execute in the database. For large tables, this can often be much more efficient than importing the data and running them locally. dplyrXdf can take advantage of this with an MRS data source that is a table in a SQL database, including (but not limited to) Microsoft SQL Server: rather than importing the data to Xdf, the data source is converted to a dplyr tbl and passed to the database for processing.
Parallel processing and grouped data
Even without a Hadoop or Spark cluster, dplyrXdf makes it easy to parallelise the handling of groups. To do this, it takes advantage of Microsoft R Server's distributed compute contexts: for example, if you set the compute context to "localpar", grouped transformations will be done in parallel on a local cluster of R processes. The cluster will be shut down automatically when the transformation is complete.
More broadly, you can create a custom backend and tell dplyrXdf to use it by setting the compute context to "dopar". This allows you a great deal of flexibility and scalability, for example by creating a cluster of multiple machines (as opposed to multiple cores on a single machine). Even if you do not have the physical machines, packages like AzureDSVM and doAzureParallel allow you to deploy clusters of VMs in the cloud, and then shut them down again. For more information, see the “Parallel processing of grouped data” section of the Using dplyrXdf vignette.
Data and file management
New in dplyrXdf 1.0.0 is a suite of functions to simplify managing Xdf files and data sources:
- HDFS file management: upload and download files with
hdfs_file_download; copy/move/delete files with
hdfs_file_remove; list files with
hdfs_dir; and more
- Xdf data management: upload and download datasets with
compute; import/convert to Xdf with
as_xdf; copy/move/delete Xdf data sources with
delete_xdf; and more
- Other utilities: run a block of code in the local compute context with
local_exec; convert an Xdf file to a data frame with
as.data.frame; extract columns from an Xdf file with methods for
Obtaining dplyr and dplyrXdf
dplyrXdf 1.0.0 is available from GitHub. It requires Microsoft R Server 8.0 or higher, and dplyr 0.7 or higher. Note that dplyr 0.7 will not be in the MRAN snapshot that is your default repo, unless you are using the recently-released MRS 9.2; you can install it, and its dependencies, from CRAN. If you want to use the SQL Server and sparklyr integration facility, you should install the odbc, dbplyr and sparklyr packages as well.