Revolutions

The R Consortium has funded half a million dollars to R projects

The R Consortium passed a significant milestone this month: since its inception, the non-profit body has provided more than US$500,000 in grant funding to project proposed by the R Community. The R Consortium uses the dues from its member organizations to fund grant proposals, which are reviewed twice a year by its Infrastructure Steering Committee. (If you'd like to propose a project, proposals for the next round are being accepted through April 1.)

New projects funded in this round include:

• Creating a new data type for R to unify the "units" and "errors" packages
• Updating the R module for the Simplified Wrapper and Interface Generator (SWIG) to support modern R programming practices like reference classes
• Providing an API and test framework to underlie the "future" package
• A package to process spatiotemporal data held on servers with a dplyr-like syntax

With these new grants, the R Consortium has funded 21 projects in total, from R packages to community events to developer tools. You can read more about the new projects in the announcement linked below.

R Consortium: Announcing the second round of ISC Funded Projects for 2017

New releases: Microsoft R Client 3.4.3, Microsoft ML Server 9.3

An update to Microsoft R Client, Microsoft's distribution of open source R with additional proprietary packages — including RevoScaleR (for data analysis at scale) and MicrosoftML (for machine learning) — is now available. Microsoft R Client 3.4.3 updates the R engine to R 3.4.3, and (on Linux) now supports deploying computations to a remote SQL Server with the sqlrutils package.

Microsoft R Client 3.4.3 is free to download and use, and as the video above shows is designed for developing analytical applications that will be deployed to production servers. It works with Microsoft Machine Learning Server 9.3, also released today. You can use Microsoft ML Server to scale your R analysis to data sets of any size or workloads of any intensity: as a server or cluster of servers on premises or in the Azure cloud, or as part of a hybrid architecture with Azure Stack.

For more information on the new capabilities in Microsoft ML Server and Microsoft R Client, take a look at the announcement linked below.

Machine Learning Blog: Introducing the Microsoft Machine Learning Server 9.3 Release

 

Because it's Friday: Faster Than Light

Faster than Light, a short film by Adam Stern, captures a single idea as well as any good sci-fi short story. It's also very pretty, and surprisingly moving. Well worth your 15 minutes. (Via Kottke)

That's all from us for this week. We'll be back with more next week, and in the meantime have a great weekend!

 

How to set up a sparklyr cluster in 5 minutes

If you've ever wanted to play around with big data sets in a Spark cluster from R with the sparklyr package, but haven't gotten started because setting up a Spark cluster is too hard, well ... rest easy. You can get up and running in about 5 minutes using the guide SparklyR on Azure with AZTK, and you don't even have to install anything yourself. I'll summarize the steps below, but basically you'll run a command-line utility to launch a cluster in Azure with everything you need already installed, and then connect to RStudio Server using your browser to analyze data with sparklyr.

Step 1: Install the Azure Distributed Data Engineering Toolkit (aztk). For this, you'll need a Unix command line with Python 3 installed. I'm on Windows, so I used a bash shell from the Windows Subsystem for Linux and it worked great. (I just had to use pip3 instead of pip to install, since the default there is Python 2.) The same process should work with other Linux distros or from a Mac terminal.

Step 2: Log into the Azure Portal with your Azure subscription. If you don't have an Azure subscription, you can sign up for free and get $200 in Azure credits.  

Step 3: Back at the command line, set up authentication in the secrets.yaml file. You'll be using the Azure portal to retrieve the necessary keys, and you'll need to create an Azure Batch account if you don't have one already. (Batch is the HPC cluster and job-management service in Azure.) You can find step-by-step details in the aztk documentation.

Step 4: Configure your cluster defaults in the cluster.yaml file. Here you can define the default VM instance size used for the cluster nodes; for example vm_size: standard_a2 gives you basic 2-core nodes. (You can override this in the command line, but it's convenient to set it here.) You'll also need to specify a dockerfile here that will be used to set up the node images, and for use with sparklyr you'll need to specify one that includes R and the version of Spark you want. I used:

docker_repo: aztk/r-base:spark2.2.0-r3.4.1-base

This provides an image with Spark 2.2.0, R 3.4.1, and a suite of R packages pre-installed, including sparklyr and tidyverse. (You could provide your own dockerfile here, if you need other things installed on the nodes.)

Step 5: Privision a Spark cluster. This is the easy bit: just use the command line tool like this:

aztk spark cluster create --id mysparklyr4 --size 4

In this case, it will launch a cluster of 4 nodes, each with 2 cores (pre the vm_size option configured above.) Each node will be pre-installed with R and (Warning: the default quotas for Azure Batch are laughably low: for me it was 24 cores total at first. You can get your limit raised fairly easily, but it can take a day to get approval.) Provisioning a cluster takes about 5 minutes; while your waiting you can check on the progress by clicking on the cluster name in the "Pools" section of your Azure Batch account within the Azure Portal.

Once it's ready, you'll also need to provide a password for the head node unless you set up ssh keys in the secrets.yaml file. 

Step 6: Connect to the head node of the Spark cluster. Normally you'd need to find the IP address first, but aztk makes it easy with its ssh command:

aztk spark cluster ssh --id mysparklyr4

(You'll need to provide a password here, if you set one up in Step 5.) This gives you a shell on the head node, but more importantly it maps the ports for Spark and RStudio server, so that you can connect to them using http://localhost URLs in the next step. Don't exit from this shell until you're done with the next steps, or the port mappings will be cancelled. 

Step 7: Connect to RStudio Server

Open a browser window on your desktop, and browse to http://localhost:8787. This will open up RStudio Server in your browser. (The default login is rstudio/rstudio.) To be clear, RStudio Server is running on the head node of your cluster in Azure Batch, not on your local machine: the port mapping from the previous step is redirecting your local port 8787 to the remote cluster.

From here, you can use RStudio as you normally would. In particular, the sparklyr package is already installed, so you can connect to the Spark cluster directly and use RStudio Server's built-in features for working with Spark.

One of the nice things about using RStudio Server is that you can shut down your browser or even your machine, and RStudio Server will preserve its state so that you can pick up exactly where you left off next time you log in. (Just use aztk spark cluster ssh to reapply the port mappings first, if necessary.)

Step 8: When you're finished, shut down your cluster using the aztk spark cluster delete command. (While you can delete the nodes from the Pools view in the Azure portal, the command does some additional cleanup for you.) You'll be charged for each node in the cluster at the usual VM rates for as long as the cluster is provisioned. (One cost-saving option is to use low-priority VMs for the nodes, for savings of up to 90% compared to the usual rates.)

That's it! Once you get used to it, it's all quick and easy -- the longest part is waiting for the cluster to spin up in Step 5. This is just a summary, but the full details see the guide SparklyR on Azure with AZTK.

Analyzing accelerometer data with R

Using your smartphone (any modern phone with a built-in accelerometer should work), visit the Cast Your Spell page created by Nick Strayer. (If you need to type it to your phone browser directly, here's a shortlink: bit.ly/castspell .) Scroll down and click the "Press To Cast!" button, and then wave your phone like a wand using one of the shapes shown.

The app will attempt to detect which of the four "spells" you gestured. It was pretty confident in its detection when I cast "Incendio", but your mileage may vary depending on your wizarding ability and the underlying categorization model. 

Nick Strayer described how he built this application in a presentation at Data Day Texas last month. The app itself was built using Shiny with the shinysense package (on Github) to collect movement data from the phone. Nick trained a convolutional neural network model (from his own casting gesture data) using the keras package to classify gestures into one of the four "spells". (Interesting side note: because CNNs aren't time-dependent, you can gesture in reverse and still pass the classification test.)

It's almost like magic! For the complete details on how the Cast your Spell app was constructed, see Nick Strayer's presentation at the link below.

Data Day Texas 2018: Making Magic with Keras and Shiny (Nick Strayer)

Machine Learning in R with TensorFlow

Modern machine learning platforms like Tensorflow have to date been used mainly by the computer science crowd, for applications like computer vision and language understanding. But as JJ Allaire pointed out in his keynote at the RStudio conference earlier this month (embedded below), there's a wealth of applications in the data science domain that have yet to be widely explored using these techniques. This includes things like time series forecasting, logistic regression, latent variable models, and censored data analysis (including survival analysis and failure data analysis).

The keras package for R provides a flexible, high-level interface for specifying machine learning models.(RStudio also provides some nice features when using the package, including a dynamically-updated convergence chart to show progress.) Networks defined with keras are flexible enough to specify models for data science applications, that can then be optimized using frameworks like Tensorflow (as opposed to traditional maximum-likelihood techniques), without limitations on data set size and with the ability to apply modern computational hardware.

For learning materials, RStudio's Tensorflow Gallery provides a good place to get started with several worked examples using real-world data. The book Deep Learning with R (Chollet and Allaire) provides even more worked examples translated from the original Python. If you want to dive into the mathematical underpinnings, the book Deep Learning (Goodfellow et al) provides the details there.

RStudio blog: TensorFlow for R

 

Because it's Friday: Hold the Door

You probably saw Boston Dynamics' robots achieve another milestone this week: not only can one of their robots open and pass through a door, it will cooperate and politely hold the door as a fellow robot passes through:

While impressive, the feat still isn't quite up to human standards:

GREAT JOB BOSTON DYNAMICS. YOU NAILED THE WAY I, A NORMAL HUMAN, ALWAYS ENTER A ROOM. pic.twitter.com/OGmGTKHnC1

— Jesse McLaren (@McJesse) February 13, 2018

That's all for the blog for this week. We'll be back next week — have a great weekend!

A fresh look for base graphics

While ggplot2 (and its various extensions) is often the go-to package for graphics in R these days, if you need to step outside the boundaries of what ggplot2 can do, you can always step back to base R graphics (and the built-in lattice package) and customize to your hearts content.

The problem is that (unlike for ggplot2) the default look for base graphics is kinda ... meh. That being said, the base graphics system offers almost unlimited flexibility, both via function options and via the par system for modifying layouts and graphic defaults. As Colin Gallespie explains in a recent blog post, you can take a scatterplot that looks, by default, like this:

and with a little extra code, make it look like this:

Find out how in Colin's blog post, linked below.

The Jumping Rivers Blog: Styling Base R Graphics

What does Microsoft do with R?

I was genuinely chuffed to get a shout-out in the most recent episode of Not So Standard Deviations, the awesome statistics-and-R themed podcast hosted by Hilary Parker and Roger Peng. In that episode, Roger recounts his recent discovery of the Microsoft ecosystem of tools for R, which he (jokingly) dubbed the "Microsoft-verse".

While we're flattered by the allusion to the tidyverse, in general Microsoft's developments with R are designed to work with the entire R ecosystem rather than be distinct from it. Here's a quick overview of what Microsoft has developed around R. It's in three sections: the first two don't require any special version of R, and only the third section requires a Microsoft-specific R distribution.

Thanks again to Hilary and Roger for another entertaining episode of NSS Deviations and for giving me the impetus to write all of this down. (This started as an email, but I quickly realized it was getting too long and became this blog post instead.) If you have any questions or feedback, let me know in the comments section of this post.

R available from within Microsoft products

You can call R from within some data oriented Microsoft products, and apply R functions (from base R, from packages, or R functions you've written) to the data they contain.

• SQL Server (the database) allows you to call R from SQL, or publish R functions to a SQL Server for database adminstrators to use from SQL. 
• Power BI (the reporting and visualization tool) allows you to call R functions to process data, create graphics, or apply statistical models to data.
• Visual Studio (the integrated development environment) includes R as a fully-supported language with syntax highlighting, debugging, etc.
• R is supported in various cloud-based services in Azure, including the Data Science Virtual Machine and Azure Machine Learning Studio. You can also publish R functions to Azure with the AzureML package, and then call those R functions from applications like Excel or apps you write yourself.

Open source R tools and packages from Microsoft

Microsoft provides various open source tools to help people use R. This includes R packages published on CRAN and in Github.

• Microsoft R Open, Microsoft's distribution of open-source R. The only difference with CRAN R is that comes bundled with Intel Math Kernel Libraries (which makes vector and matrix operations faster on multi-core machines), and that it uses a static mirror of CRAN so packages don’t change from day to day (but you can always use the "checkpoint" package described below to get the latest-and-greatest, if you want). 
• MRAN, which is the download repository for Microsoft R Open, and also hosts daily archive snapshots of the entire CRAN system (from 2015 to the present). These snapshots are used for reproducibility by Microsoft R Open, the checkpoint package (see below), and anyone who wants a non-changing CRAN image. (The Rocker docker images are configured to use these static snapshots, for example.)
• The checkpoint package, which provides a simple interface to those static CRAN snapshots, for reproducibility. (In short: add checkpoint("2018-02-13") to make R install and use packages from that date for your project, now and in the future, including when you share scripts with someone else.)
• The foreach and iterators packages, for parallel programming. Microsoft also provide “backends” to use different parallel programming systems for a foreach loop, like doParallel (use a local machine or cluster), and doAzureParallel (spin up a cluster in Azure and to the parallel iterations there).

Parallel and distributed algorithm implementations of statistics and machine learning algorithms

Microsoft has implemented a suite of algorithms for statistics and machine learning. These either serve as replacements for existing R functions or packages, or add new capabilities. They are designed for performance and to work without data size limitations. (In general, these algorithms are closed-source and only available within Microsoft R products, and not on CRAN or Github.)

• The RevoScaleR package provides new implementations of some of R's statistical functions (for example: rxGlm is the equivalent of R's glm), but are designed to work with data sizes much larger than available memory. It also uses parallel computing to speed things up when running on a multi-core server, in a Hadoop or Spark cluster, or in a SQL Server database.
• The MicrosoftML package provides high-performance implementations of various machine-learning algorithms (like neural networks and random forests), and also some pre-trained machine learning models like sentiment analysis for text.
• Those two packages can optionally make use of the XDF file format, a binary on-disk data format designed for performance and parallel processing. Within R, XDF objects behave much like data frames, and you can also apply tidyverse data functions with the dplyrXdf package.
• The mrsdeploy package allows you to publish custom R scripts and functions (including ones using the packages above) to a server as an API that can be called from other applications. 
• Microsoft Machine Learning Server includes R and the packages above. So do SQL Server, and HDInsight (the big-data platform on Azure). Microsoft also provides paid support for R to customers of these products.
• Microsoft R Client also includes those packages. It's free to use for developing applications, but it's performance-limited to non-production workloads.

Diversity scholarships for upcoming R conferences

One of the greatest things about the R community is its diversity. This is largely thanks to organizations like Forwards and R-Ladies, who have been instrumental in welcoming women and other under-represented groups to the world of R. Likewise, conferences in the R community encourage diversity, with open codes of conduct, facilitations like on-site child-care, and by offering scholarships for travel and lodging to encourage attendees from diverse backgrounds. 

Here are three upcoming R community events that are offering diversity scholarships:

The rOpenSci Unconference (May 21-22, Seattle) is now accepting self-nominations to participate. You can request travel support as part of the application process, and the organizers "strongly encourage applications from women and other underrepresented genders, people of color, people who are LGBTQ, people with disabilities or any other underrepresented minorities in research". Applications are open now.

The eRum 2018 Conference (May 14-16, Budapest, Hungary) will include a Package Development Workshop for Women presented by Forwards, which is also offering scholarships to attend (thanks to funding from the R Consortium). The deadline to apply is February 25.

The useR! 2018 Conference (July 10-13, Brisbane, Australia) is offering scholarships for those seeking financial aid to attend, and in addition R-Ladies Melbourne is offering two $500 travel scholarships. Applications are open now.

If you're thinking about applying, Forwards offers useful tips for applying for diversity scholarships.