Revolutions

Because it's Friday: Powerpoint Punchcards

A "Turing Machine" — a conceptual data processing machine that processes instructions encoded on a linear tape — is capable of performing the complete range of computations of any modern computer (though not necessarily in a useful amount of time). Tom Wildenhain demonstrates the Turing-competeness of Powerpoint, where the "tape" is a series of punch-cards controlled by the animations feature:

Of the many things you can do with Powerpoint, but probably shouldn't do, this ranks right up there.

That's all from us for this week. See you back here on Monday, and have a great weekend!

Make pleasingly parallel R code with rxExecBy

Some things are easy to convert from a long-running sequential process to a system where each part runs at the same time, thus reducing the required time overall. We often call these "embarrassingly parallel" problems, but given how easy it is to reduce the time it takes to execute them by converting them into a parallel process, "pleasingly parallel" may well be a more appropriate name.

Using the foreach package (available on CRAN) is one simple way of speeding up pleasingly parallel problems using R. A foreach loop is much like a regular for loop in R, and by default will run each iteration in sequence (again, just like a for loop). But by registering a parallel "backend" for foreach, you can run many (or maybe even all) iterations at the same time, using multiple processors on the same machine, or even multiple machines in the cloud.

For many applications, though, you need to provide a different chunk of data to each iteration to process. (For example, you may need to fit a statistical model within each country — each iteration will then only need the subset for one country.) You could just pass the entire data set into each iteration and subset it there, but that's inefficient and may even be impractical when dealing with very large datasets sitting in a remote repository. A better idea would be to leave the data where it is, and run R within the data repository, in parallel.

Microsoft R 9.1 introduces a new function, rxExecBy, for exactly this purpose. When your data is sitting in SQL Server or Spark, you can specify a set of keys to partition the data by, and an R function (any R function, built-in or user-defined) to apply to the partitions. The data doesn't actually move: R runs directly on the data platform. You can also run it on local data in various formats

The rxExecBy function is included in Microsoft R Client (available free) and Microsoft R Server. For some examples of using rxExecBy, take a look at the Microsoft R Blog post linked below.

Microsoft R Blog: Running Pleasingly Parallel workloads using rxExecBy on Spark, SQL, Local and Localpar compute contexts

Where Europe lives, in 14 lines of R code

Via Max Galka, always a great source of interesting data visualizations, we have this lovely visualization of population density in Europe in 2011, created by Henrik Lindberg:

 

Impressively, the chart was created with just 14 lines of R code:

(To recreate it yourself, download the GEOSTAT-grid-POP-1K-2011-V2-0-1.zip file from eurostat, and move the two .csv files inside in range of your R script.) The code parses the latitude/longitude of population centers listed in the CSV file, arranges them into a 0.01 by 0.01 degree grid, and plots each row as a horizontal line with population as the vertical axis. Grid cells with zero populations cause breaks in the line and leave white gaps in the map. It's quite an elegant effect!

Update:  James Cheshire created a version of this concept back in 2013 (again using R of course). You can see (and purchase a print) of a map of the world with population lines, which was used for the Royal Geographical Society’s Annual Conference in 2013.

dv01 uses R to bring greater transparency to the consumer lending market

The founder of the NYC-based startup dv01 watched the 2008 financial crisis and was inspired to bring greater transparency to institutional investors in the consumer lending market. Despite being an open-source shop, they switched their data services to Microsoft SQL Server to provide better performance (reducing latency for queries from tens of seconds to under two seconds). They also use R for modeling, as you can see in the video below:

Principal Data Scientist Wei Wu also mentions Microsoft Machine Learning in the video. The MicrosoftML package, included in SQL Server R Services and Microsoft R Server 9, provides parallel implementations of many popular algorithms as R functions. The video below (presented by Gleb Krivosheev and Yunling Wang) provides a brief overview of some of the new capabilities in Microsoft ML, including the new pre-trained models for sentiment analysis and image featurization.

To try out the MicrosoftML package, download SQL Server 2016 or Microsoft R Server 9.

Using checkpoint with knitr and RStudio

The knitr package by Yihui Xie is a wonderful tool for reproducible data science. I especially like using it with R Markdown documents, where with some simple markup in an easy-to-read document I can easily combine R code and narrative text to generate an attractive report with words, tables and pictures in HTML, PDF or Word format. Say, something like this:

In that document, the numerical weather records and the chart were generated by R,  combined into a document using R Markdown, and then generated as a word file with knitr. (You can find the R Markdown file to generate that report, and the R script to download the data, in my weather-report repository.)

Another useful tool for reproducible data science is the checkpoint package. It helps you manage the ever-changing ecosystem of R packages on CRAN, by making it easy to "lock in" specific versions of R packages. With a single call to the checkpoint function — say checkpoint("2017-04-25"), for April 25, 2017 — you can automatically find all the packages used by your current R project (i.e. the current folder) and install them as they used to be on the specified date. A colleague or collaborator can use the same script to get the same versions too, and so be confident of reproducing your results without having to worry a newer package version may have affected the results. By the way, those package versions get installed in a special folder (.checkpoint, in your home directory), so they won't change the results of any other R projects, either. 

RStudio includes a very useful tool for working with R Markdown and knitr: you can press the "Knit" toolbar button to process the document with a single click. For that to work, it does require certain R packages to be available for use behind the scenes. In normal circumstances RStudio will offer to install them, but the process doesn't work when a checkpoint folder is active. A simple workaround is to include a file in the same folder (I call mine knitr-packages.R) with the following lines:

library("formatR")
library("htmltools")
library("caTools")
library("bitops")
library("base64enc")
library("rprojroot")
library("rmarkdown")
library("evaluate")
library("stringi")

Although you never run that file directly, the checkpoint process will discover it and ensure the necessary packages are installed for RStudio to perform its magic. (In my tests this works with recent versions of RStudio including the latest, 1.0.143). All you need to do is make sure you run checkpoint from the R command line (just press Control-ENTER on the corresponding line in the .Rmd file) before attempting to knit. Simple!

R 3.4.0 now available

R 3.4.0, the latest release of the R programming language (codename: "You Stupid Darkness"), is now available. This is the annual major update to the R language engine, and provides improved performance for R programs. The source code was released by the R Core Team on Friday and binaries for Windows, Mac and Linux are available for download now from CRAN.

The most significant change in this release is that the JIT ('Just In Time') byte-code compiler is now enabled at level 3 by default. This means that top-level loops will be compiled before being run, and all functions will be compiled on their first or second use. Byte-compilation has been available for packages for some time, but the inclusion of this just-in-time compiler means you'll see similar performance benefits for your own R code without having to take any additional steps to compile it.

There have been various other performance improvements as well. Sorting vectors of numbers is faster (thanks to the use of the radix-sort algorithm by default). Tables with missing values compute quicker. Long strings no longer cause slowness in the str function. And the sapply function is faster when applied to arrays with dimension names.

Also on the performance front, R now will use some higher-performance BLAS routines for linear algebra operations (for example, DGEMV is now used instead of DGEMM). R falls back to a single-threaded (but higher-precision) built-in linear algebra functions when missing values are present in the data, and the check for that situation has also been sped up which should further improve performance.

For package authors using compiled C/C++/Fortran code, there is a new function to register the routines that should be exposed to other packages. Registering functions speeds up the process of calling compiled code, particularly on Windows systems. The gain is measured in the order of microseconds per call, but when these functions are called thousands or millions of times the impact can be noticeable.

This update also makes a number tweaks to the language to improve consistency, especially in corner cases. R is now more consistent in its handling of missing values when constructing tables. Some common programmer errors, like comparing a vector with a zero-length array, now generate warnings. And there have also been some accuracy improvements for extreme values in some statistical functions.

This is just a summary of the changes; check out the announcement linked below for the detailed list. As always, big thanks go to the volunteers of the R Core Group for continuing to push the boundaries of R's capabilities with each annual release and the several patches that improve it each year. (By the way, the R Foundation now accepts donations via its website. If you benefit from using R, consider contributing.)

R-announce mailing list: R 3.4.0 is released

Because it's Friday: Secrets of the London Underground

I guess these have been around for a few years now, but I recently stumbled across these short videos dedicated to each of the 11 lines of the London Underground. Presented by Geoff Marshall (who once set a record for visiting all the stations in a single day), each video includes interesting trivial on the history, operations and art and architecture of the stations on each line. Something to check out for your next visit to London.

That's all from us for this week — it's been a big week of announcements! We'll be back on Monday with more posts. Have a great weekend!

Reproducible Data Science with R

Yesterday, I had the honour of presenting at The Data Science Conference in Chicago. My topic was Reproducible Data Science with R, and while the specific practices in the talk are aimed at R users, my intent was to make a general argument for doing data science within a reproducible workflow. Whatever your tools, a reproducible process:

• Saves time,
• Produces better science,
• Creates more trusted research,
• Reduces the risk of errors, and
• Encourages collaboration.

Sadly there's no recording of this presentation, but my hope is that the slides are sufficiently self-contained. Some of the images are links to further references, too. You can browse them below, or download (CC-BY) them from the SlideShare page.

Thanks to all who attended for the interesting questions and discussion during the panel session!

SQL Server 2017 to add Python support

One of the major announcements from yesterday's Data Amp event was that SQL Server 2017 will add Python as a supported language. Just as with the continued R support, SQL Server 2017 will allow you to process data in the database using any Python function or package without needing to export the data from the database, and use SQL Server itself as an operationalization platform for production applications using Python code. In addition, the high-performance and distributed statistical and machine learning functions from the RevoScaleR and MicrosoftML packages in Microsoft R will be available as Python functions for use within SQL Server.

The video below provides more details on the capabilities and the underlying architecture, and includes a demo of deploying and running Python code within SQL Server 2017.

SQL Server 2017 will also be available for the first time on Linux (including within Docker containers), and the new version is currently in public preview. In case you missed it yesterday, check out our post on the many new features supporting R in Microsoft R Server 9.1. And for more on the many updates coming in SQL Server 2017, take a look at the official blog post linked below.

SQL Server Blog: Delivering AI with data: the next generation of Microsoft’s data platform

Microsoft R Server 9.1 now available

During today's Data Amp online event, Joseph Sirosh announced the new Microsoft R Server 9.1, which is available for customers now. In addition the updated Microsoft R Client, which has the same capabilities for local use, is available free for everyone on both Windows and — new to this update — Linux.

This release adds many new capabilities to Microsoft R, including: 

Based on R 3.3.3. This update is based on the latest release of open source R from the R Foundation, bringing many improvements to the core language engine.

Microsoft R Client now available on Linux. The Microsoft R Client is available free, and has all of the same capabilities as Microsoft R Server for a standalone machine (or, you can use it to push heavy workloads to a remote Microsoft R Server, SQL Server or compute cluster). With this release it is now also available on Linux (specifically, Ubuntu, Red Hat / CentOS and SUSE) in addition to Windows.

New function for "pleasingly parallel" R computations on data partitions. The new rxExecBy function allows you to apply any R function to partitions of a data set, and compute on the partitions in parallel. (You don't have to manually split the data first; this happens automatically at the data source, without the need to move or copy the data.) You can use any parallel platform supported in Microsoft R: multiple cores in a local context, multiple threads in SQL Server, or multiple nodes in a Spark cluster.

New functions for sentiment scoring and image featurization have been added to the built-in MicrosoftML package. The new getSentiment function returns a sentiment score (on a scale from "very negative" to "very positive") for a piece of English text. (You can also featurize text in English, French, German, Dutch, Italian, Spanish and Japanese as you build your own models.) The new featurizeImage function decomposes an image into a few numeric variables (using a selection of ResNet recognizers) which can then be used as the basis of a predictive model. Both functions are based on deep neural network models generated from thousands of compute-hours of training by Microsoft Research.

Interoperability between Microsoft R Server and sparklyr. You can now use RStudio's sparklyr package in tandem with Microsoft R Server in a single Spark session

New machine learning models in Hadoop and Spark. The new machine learning functions introduced with Version 9.0 (such as FastRank gradient-boosted trees and GPU-accelerated deep neural networks) are now available in the Hadoop and Spark contexts in addition to standalone servers and within SQL Server. 

Production-scale deployment of R models. The new publishService function creates a real-time web-service for certain RevoScaleR and MicrosoftML functions that is independent of any underlying R interpreter, and can return results with millisecond latency. There are also new tools for launching and managing asynchronous batch R jobs.

Updates for R in SQL Server. There are also improvements for R Services in SQL Server 2016, in addition to the new R functions described above. There are new tools to manage installed R packages on SQL Server, and faster single-row scoring. Even more new capabilities are coming in SQL Server 2017 (now in preview for both Windows and Linux), including the ability to use both R and Python for in-database computations.

For more on the updates in Microsoft R Server 9.1, check the official blog post by Nagesh Pabbisetty linked below.

SQL Server Blog: Introducing Microsoft R Server 9.1 release