Revolutions

Because it's Friday: Goodbye, 2016

Between the deaths of beloved heroes and heroines, the civil unrest and political upheavals, and a slew of natural disasters, 2016 wasn't the greatest year. If you made a movie about it, this is what the trailer would look like:

We'll be taking a few days off to recover from 2016. See you back here on January 3 and in the meantime, a very happy (and welcome!) New Year to all of our readers. 

Power BI custom visuals, based on R

You've been able to include user-defined charts using R in Power BI dashboards for a while now, but a recent update to Power BI includes seven new custom charts based on R in the customs visuals gallery. You can see the new chart types by visiting the Power BI Custom Visuals Gallery and clicking on the "R-powered visuals" tab.

The new custom visuals are listed below. Click on the visual names to see an example, and click on the "GitHub" link at the bottom of the pop-up to see the actual R code used.

• Clustering, based on the NbClust package
• Clustering with Outliers, based on the dbscan package
• Correlation Plot, based on the corrplot package
• Decision Tree, based on the rpart and the rpart.plot packages
• Forecasting, based on the forecast package
• Association Rules, based on the arules and arulesViz packages
• Smoothing Splines, based on the splines package

You can also combine custom R charts with other Power BI tools to create interactive R charts. A favorite of mine is the Timeline slicer, which allows you to select a range from a date variable. (The dashboard, including the R charts, is updated using only data from the date range you select.) You can also use the built-in date slicer for similar effect, and Tomaž Kaštrun demonstrates how to combine it with a custom R chart in his blog post R graphs and Tables in Power BI Desktop.

Using R to prevent food poisoning in Chicago

There are more than 15,000 restaurants in Chicago, but fewer than 40 inspectors tasked with making sure they comply with food-safety standards. To help prioritize the facilities targeted for inspection, the City of Chicago used R to create a model that predicts which restaurants are most likely to fail an inspection. Using this model to deploy inspectors, the City is able to detect unsafe restaurants more than a week sooner than by using traditional selection methods, and cite 37 additional restaurants per month.

Chicago's Department of Public Health used the R language to build and deploy the model, and made the code available as an open source project on GitHub. The reasons given are twofold:

An open source approach helps build a foundation for other models attempting to forecast violations at food establishments. The analytic code is written in R, an open source, widely-known programming language for statisticians. There is no need for expensive software licenses to view and run this code.

Releasing the model as open source has had benefits for beyond Chicago as well: Montogomery County, MD adopted the process and also saw improvements in its food safety inpection process.

You can see how the model is used in practice in the video below from PBS NewsHour. Fast forward to the 3:00 mark to see the Tom Schenk, Chief Data Officer for the City of Chicago, describe how the data science team there used R to develop the model. (There's also a close-up of R code using the data.table package around the 6:45 mark.)

The video also describes the Foodborne Chicago Twitter detection system for flagging tweets describing food poisoning in Chicago (also implemented with R).

PBS NewsHour: Up to code? An algorithm is helping Chicago health officials predict restaurant safety violations (via reader MD)

Combine choropleth data with raster maps using R

Switzerland is a country with lots of mountains, and several large lakes. While the political subdivisions (called municipalities) cover the high mountains and lakes, nothing much of economic interest happens in these places. (Raclette and sailing are wonderful, but don't count for our purposes.) For this reason, the Swiss Federal Statistical Office publishes the boundaries of the "productive" parts of the municipalities, and as this choropleth of average age in Swiss municpalities created by Timo Grossenbacher shows, leaving out the non-productive parts leaves us with a very different-looking Switzerland.

The choropleth would be more recognizable by filling in the non-productive areas with a traditional relief map, which is exactly what Timo does (along with breaking the age scale into discrete categories, for improved interpretability) in the publication-quality map below.

Timo's blog post, Beautiful thematic maps with ggplot2 (only), details the process of building maps like this using the ggplot2 package (and just a few others) for R. There are lots of useful nuggets of advice within the tutorial, including:

• To run scripts in a "clean" R session, to avoid conflicts with packages and objects that happen to be hanging around. Timo suggests unloading packages and removing objects, but a quicker and easier way is to simply launch a new R session with "R --vanilla". The --vanilla option prevents R from running any initialization scripts (that might load packages) or loading any objects from a saved workspace.
• To import geographic boundaries using the readOGR function, and the use of coord_equal to display them as a map without distortion.
• To choose a useful color scale for continuous variables (like age) with the viridis package, and how to discretize them into buckets to improve visibility of regional differences.
• To define a ggplot2 theme according to your presentation style guide (here, a light grey background, a specific font, and faint grid lines).
• Importing a TIFF relief map of Switzerland using the raster function, and overlaying the choropleth (and its holes for the non-productive regions) on top of it to create the combined map.

For the complete tutorial, including links to the code and data, check out Timo's blog post linked below.

Timo Grossenbacher: Beautiful thematic maps with ggplot2 (only)

Update Dec 29: A couple of minor corrections based on feeback from Timo Grossenbacher

Parallelizing Data Analytics on Azure with the R Interface Tool

by Le Zhang (Data Scientist, Microsoft) and Graham Williams (Director of Data Science, Microsoft)

In data science, to develop a model with optimal performance, exploratory experiments on different sets of hyper-parameters are often performed. Preliminary analyses on smaller data can be performed on a single machine, while the experimental one on large-scale data by sweeping multi-sets of parameters can be run on a cluster to boost the computational efforts. Scalable computation resources which can be easily managed are desired for such an application scenario. This blog post shares a walk-through using the Azure Interface tool that

• operates and manages Azure cloud instances directly from R using the AzureSMR package, and
• executes scalable analytical jobs on deployed instances using customized Microsoft R Server computation contexts, which are easily specified in R using an "interface object".

The overall architecture of the Interface Tool is described in the graphic below. The Master script manages Azure instance deployment and operation, as well as settings of analytics execution specifications. The Worker script contains the actual analytics which is submitted onto the Azure instance and run according to the configured context.

 

A predictive maintenance example is presented to illustrate the efficacy of the interface tool. Predictive maintenance is widely employed within the airline and manufacturing industries to reduce operational costs. One of the problems in a predictive maintenance scenario is to diagnose whether equipment is working under a healthy status or not by analyzing historical sensor measurements. Based on the data set of sensor measurements, a well-trained model is able to classify a machine as “low risk” or “high risk” of failure. To obtain an optimal model for accurate prediction, hyper-parameters should be swept to perform a search on the domain of definition. There are two critical hyper-parameters in the health status recognition that need experimental analysis for optimal prediction: 

The Basics of Bayesian Statistics

Bayesian Inference is a way of combining information from data with things we think we already know. For example, if we wanted to get an estimate of the mean height of people, we could use our prior knowledge that people are generally between 5 and 6 feet tall to inform the results from the data we collect. If our prior is informative and we don't have much data, this will help us to get a better estimate. If we have a lot of data, even if the prior is wrong (say, our population is NBA players), the prior won't change the estimate much. You might say that including such "subjective" information in a statistical model isn't right, but there's subjectivity in the selection of any statistical model. Bayesian Inference makes that subjectivity explicit.

Bayesian Inference can seem complicated, but as Brandon Rohrer explains, it's based on straighforward principles of conditional probability. Watch his video below for an elegant explanation of the basics.

If you'd like to try out some Bayesian statistics yourself, R has many packages for Bayesian Inference. 

Data Science and Robots Blog: How Bayesian inference works

 

 

Because it's Friday: A Christmas Destiny

This video isn't CGI. It's more like a Machinima Christmas pantomime: carefully selected costumes, choreographed performances, and sharp editing of some long takes in the world of Destiny, one of my favourite games. The only thing added that isn't from the game is the music.

That's all from us for this week. To all who celebrate, enjoy Christmas, and to everyone have a great weekend. We'll be back as usual next week -- see you on Monday!

Merry ChRistmas!

Christmas day is soon upon us, so here's a greeting made with R:

Each frame is a Voronoi Tesselation: about 1,000 points are chosen across the plane, which each generate a polygon comprising the region closer to it than any other selected point. These process is repeated for three designs (a heart, the word "Merry", and the word "Xmas"), and the Voronoi tiles are colored according to whether the generating point is within the design. In the transitions between the designs, the points migrate from one selected point in one design to a selected counterpoint in the next. The counterpoint is randomly chosen for the "heart" transition (generating a chaotic assembly effect); and a nearby point is chosen for the "Merry" transition (for a much smoother effect).

There's some interesting R-based magic behind how the animation was created as well. The three designs were created in Adobe Illustrator. Getting shapes from Illustrator into R is a surprisingly simple two-line process: you simply import the design using fromJSON, and extract the component representing line segments which can be immediately drawn in R.  The tesselations were drawn using the geom_voronoi_tile() ggplot2 geom in the Github version of the ggforce package, and colored by checking whether the generating point falls inside the design using the in.out function of the mgcv package. The smooth transition of the generating points from one design to the next were calculated using the tweenr package, and the whole thing was animated using the animation package.

For the complete details on how this lovely greeting was created, including the R code, follow the link to creator Thomas Lin Pedersen's blog post below.

Data Imaginist: Data Driven Christmas Card Animation with Voronoi Tiles

Take a Test Drive of the Linux Data Science Virtual Machine

If you've been thinking about trying out the Data Science Virtual Machine on Linux, but don't yet have an Azure account, you can now take a free test drive -- no credit card required! Just visit the Linux DSVM Marketplace page and click the blue button:

The Linux Data Science Virtual Machine includes all of the tools a modern data scientist needs, in one easy-to-launch package. With it, you can try exploring data with Apache Drill, train deep neural networks for computer vision with MXNet, develop AI applications with the Cognitive Toolkit, or create statistical models with big data in R with Microsoft R Server 9.0.

For details on how to test drive Linux Data Science Virtual Machine, follow the link below.

Cortana Intelligence and Machine Learning Blog: New Additions to the Data Science Virtual Machine – Test Drive, Community Forums, Deep Learning 

 

Introducing the AzureSMR package: Manage Azure services from your R session

by Alan Weaver, Advanced Analytics Specialist at Microsoft

Very often data scientists and analysts require access to back-end resources on Azure. For example, they may need to start a virtual machine or resize a Hadoop cluster. This typically requires making a request to the IT department and patiently waiting. 

AzureSMR is a simple R package that enables those users to do many of those operations themselves.

It’s very easy to script commonly-used functions which can be run without having to navigate the portal or wizards. AzureSMR uses the Azure Systems Management API and leverages standard packages such as httr, so it can easily run in any R session (you don't need Microsoft R Server).  You can also manage multiple Azure subscriptions from within the same session.

The AzureSMR functions currently addresses the following Azure Services:

• Azure Blob: List, Read and Write to Blob Services
• Azure Resources: List, Create and Delete Azure Resource. Deploy ARM templates.
• Azure VM: List, Start and Stop Azure VMs
• Azure HDI: List and Scale Azure HDInsight Clusters
• Azure Hive: Run Hive queries against a HDInsight Cluster
• Azure Spark: List and create Spark jobs/Sessions against a HDInsight Cluster(Livy)

For example, here's how you would use the AzureSMR package to find, start and stop an Azure virtual machine:

azureListVM(sc, resourceGroup = "Analytics")   # LIST VMS
azureStartVM(sc, vmName = "MYVM")              # START A VM
azureStopVM(sc, vmName = "MYVM")               # STOP A VM

Or, you could spin up a Spark cluster in HDInsight (though you should choose better passwords!):

azureCreateHDI(sc,
  resourceGroup = "myResourceGroup",
  clusterName   = "myhdicluster",
  location      = "northeurope", 
  storageAcc    = "azuresmrteststorage", 
  version       = "3.5", 
  sshUser       = "hdiuser", 
  sshPassword   = "Password123!!",
  adminUser     = "admin", 
  adminPassword = "Password123!!"
)

Or create an Azure Storage account and list and create files:

azureCreateStorageAccount(sc, storageAccount = "azuresmrteststorage", 
                              resourceGroup = "myResourceGroup")
azureCreateStorageContainer(sc, "opendata")
azureListStorageContainers(sc, storageAccount = "azuresmrteststorage", 
                               resourceGroup ="myResourceGroup")
azurePutBlob(sc, contents=”XXX”, blob = "testdir/testfile.csv")
azureBlobCD(sc, "/testdir")
azureBlobLS(sc)

For a detailed list of the available functions and their syntax please refer to the Help pages. A vignette is available to help set up Authentication and there is also a Getting started tutorial.

To get started with the AzureML package, simply use the devtools package to install it from Github.

if(!require("devtools")) install.packages("devtools")
devtools::install_github("Microsoft/AzureSMR")
library(AzureSMR)

We welcome your feedback on this package! Contact us via the Github repo, linked below.

Github (Microsoft): AzureSMR, an R package for managing a selection of Azure resources