Revolutions

XKCD "Curve Fitting", in R

You probably saw this XKCD last week, which brought a grimace of recognition to statisticians everywhere:

It's so realistic, that Barry Rowlingson was able to reproduce all but two of the "charts" above with a simple R function (and a little help from the xkcd ggplot2 theme):

And now for @revodavid et al, with the xkcd package and font! (still two more to do...) pic.twitter.com/3aVHis23Gl

— Barry Rowlingson (@geospacedman) September 21, 2018

You can find the R code behind Barry's reproductions here, and I'm sure he'd welcome contributions for the remaining two charts :). 

 

3-D shadow maps in R: the rayshader package

Data scientists often work with geographic data that needs to be visualized on a map, and sometimes the maps themselves are the data. The data is often located in two-dimensional space (latitude and longitude), but for some applications we have a third dimension as well: elevation. We could represent the elevations using contours, color, or 3-D perspective, but with the new rayshader package for R by Tyler Morgan-Wall, it's easy to visualize such maps as 3-D relief maps complete with shadows, perspective and depth of field:

👌 Dead-simple 3D surface plotting in the next version of rayshader! Apply your hillshade (or any image) to a 3D surface map. Video preview with rayshader's built-in palettes. #rstats

Code:

elmat %>%
sphere_shade() %>%
add_shadow(ray_shade(elmat)) %>%
plot_3d(elmat) pic.twitter.com/FCKQ9OSKpj

— Tyler Morgan-Wall (@tylermorganwall) July 2, 2018

Tyler describes the rayshader package in a gorgeous blog post: his goal was to generate 3-D representations of landscape data that "looked like a paper weight". (Incidentally, you can use this package to produce actual paper weights with 3-D printing.) To this end, he went beyond simply visualizing a 3-D surface in rgl and added a rectangular "base" to the surface as well as shadows cast by the geographic features. He also added support for detecting (or specifying) a water level: useful for representing lakes or oceans (like the map of the Monterey submarine canyon shown below) and for visualizing the effect of changing water levels like this animation of draining Lake Mead.

The rayshader package is implemented as an independent R package; it doesn't require any external 3-D graphics software to work. Not only does that make it easy to install and use, but it also means that the underlying computations are available for specialized data analysis tasks. For example, research analyst David Waldran used a LIDAR scan of downtown Indianapolis to create (with the lidR package) a 3-D map of the buildings, and then used the ray_shade function to simulate the shadows cast by the buildings at various times during a winter's day. Averaging those shadows yields this map of the shadiest winter spots in Indianapolis:

The rayshader package is available for download now from your local CRAN mirror. You can also find an overview, and the latest version of the package at the Github repository linked below.

Github (tylermorganwall): rayshader, R Package for Producing and Visualizing Hillshaded Maps from Elevation Matrices, in 2D and 3D

 

R developer's guide to Azure

If you want to run R in the cloud, you can of course run it in a virtual machine in the cloud provider of your choice. And you can do that in Azure too. But Azure provides seven dedicated services that provide the ability to run R code, and you can learn all about them in the new R Developer's Guide to Azure at Microsoft Docs. The services include:

• Data Science Virtual Machine: a customized VM to use as a data science workstation or as a custom compute target. This includes Microsoft R Open, Microsoft Machine Learning Server, and Microsoft ML Services in SQL Server, which can also be run locally.
• ML Services on HDInsight: a cluster-based system for running R analyses on large datasets across many nodes.
• Azure Databricks: a collaborative Spark environment that supports R and other languages.
• Azure Machine Learning Studio: a drag-and-drop app for Azure's machine learning experiments, that supports custom R scripts.
• Azure Batch: a variety of options for economically running R code across many nodes in a cluster.
• Azure Notebooks: a no-cost (but limited) cloud-based version of Jupyter notebooks.
• Azure SQL Database: to run R scripts inside of the SQL Server database engine.

Click on the links above for detailed documentation on how to run R in each of these services. Like all Microsoft Docs this guide is hosted in Github, so if you have suggestions for modifications or additions to this document, you can use the "Content Feedback" link to provide suggestions directly in the repository.

Microsoft Docs: R developer's guide to Azure

Because it's Friday: Fly Strong

I was about the same age as student pilot Maggie Taraska when I had my first solo flight. Unlike Maggie, I didn't have to deal with a busy airspace, or air traffic control, or engines (I was in a glider), or — most significantly — one of my landing wheels falling off during the flight. But Maggie handled the entire situation much more coolly than I remember my own short, uneventual, first solo. Just listen to the radio chatter below. If you're in a rush, you can skip the period from 2:00 – 7:30 while she circles the Beverly (MA) airport waiting for a couple of other planes to land first, but the whole thing is worth a listen to hear how she successfully lands with only one wheel on the left side.

 

That's all from us at the blog for this week. Have a great weekend, and we'll be back next week.

Applications of R presented at EARL London 2018

During the EARL (Enterprise Applications of the R Language) conference in London last week, the organizers asked me how I thought the conference had changed over the years. (This is the conference's fifth year, and I'd been to each one.) My response was that it reflected the increasing maturity of R in the enterprise. The early years featured many presentations that were about using R in research and the challenges (both technical and procedural) for integrating that research into the day-to-day processes of the business. This year though, just about every presentation was about R in production, as a mainstream part of the operational infrastructure for analytics. 

That theme began in earnest with Garret Grolemund's keynote presentation on the consequences of scientific research that can't be replicated independently. (This slide, based on this 2016 JAMA paper was an eye-opener for me.) The R language has been at the forefront of providing the necessary tools and infrastructure to remove barriers to reproducible research in science, and the RMarkdown package and its streamlined integration in RStudio, is particularly helpful. Garret's keynote was recorded but the video isn't yet available; when it's published I highly recommend taking the time to watch this excellent speech.

The rest of the program was jam-packed with a variety of applications of R in industry as well. I couldn't see them all (it was a three-track conference), but every one I attended demonstrated mature, production-scale applications of R to solve difficult business problems with data. Here are just a few examples:

• Rainmakers, a market research firm, uses R to estimate the potential market for new products. They make use of the officer package to automate the generation of Powerpoint reports.
• Geolytix uses R to help companies choose locations for new stores that maximize profitability while reducing the risk of cannibalizing sales from other nearby locations. They use SQL Server ML Services to deploy these models to their clients.
• Dyson uses R (and the prophet package) to forecast the expected sales of new models of vacuum cleaners, hand dryers, and other products, so that the manufacturing plants can ramp up (or down) production as needed.
• Google uses R to design and analyze the results of customer surveys, to make the best decisions of which product features to invest in next.
• N Brown Group, a fashion retailer, uses R to analyze online product reviews from customers.
• Marks and Spencer uses R to increase revenues by optimizing the products shown and featured in the online store.  
• PartnerRe, the reinsurance firm, has built an analytics team around R, and uses Shiny to deploy R-based applications throughout the company.
• Amazon Web Services uses containerized applications with R to identify customers who need additional onboarding assistance or who may be dissatisfied, and to detect fraud.
• Microsoft uses R and Spark (via sparklyr in HDInsight) to support marketing efforts for Xbox, Windows and Surface with propensity modeling, to identify who is most likely to respond to an offer. 

You can see may more examples at the list of speakers linked below. (I blogged about my own talk at EARL earlier this week.) Click through to see detailed summaries and (in most cases) a link to download slides.

EARL London 2018: Speakers 

AI, Machine Learning and Data Science Roundup: September 2018

A monthly roundup of news about Artificial Intelligence, Machine Learning and Data Science. This is an eclectic collection of interesting blog posts, software announcements and data applications from Microsoft and elsewhere that I've noted over the past month or so.

Open Source AI, ML & Data Science News

ONNX 1.3 released. The standard for representing predictive models adds a cross-platform API for executing graphs on optimized backends and hardware.

A new foundation supporting the development of scikit-learn, the Python machine learning library.

Google open sources "dopamine", a Tensorflow-based framework for reinforcement learning research.

Google adds a "What-If Tool" to Tensorboard, to interactively explore the robustness and algorithmic fairness of machine learning models.

Industry News

Google introduces Dataset Search, an index of public datasets from the environmental and social sciences, government, and journalism.

The 2018.3 update to the Alteryx analytics platform brings interactive graphics, and Spark and Jupyter Notebook integration.

AWS Deep Learning AMIs now provide Tensorflow 1.10 and PyTorch with CUDA 9.2 support.

Rigetti Computing launches the Quantum Cloud Service with a $1M prize for the first application demonstrating "quantum advantage".

Thomas Dinsmore's commentary on Forrester Wave rankings for "Multimodal Predictive Analytics and Machine Learning Platforms" and "Notebook-Based Predictive Analytics and Machine Learning Solutions".

Microsoft News

Microsoft acquires Lobe, a Bay Area startup that produced a drag-and-drop interface for building machine learning models.

NVIDIA GPU Cloud now provides ready-to-run containers with GPU-enabled deep learning frameworks for use on Azure.

Microsoft introduces Azure CycleCloud, a tool for creating, managing, operating, and optimizing burst, hybrid, and cloud-only HPC clusters. Schedulers including Slurm, Grid Engine and Condor are supported.

New features in Azure HDInsight: Spark 2.3.0 and Kafka 1.1 support; ML Services 9.3 integration, with updated R engine and new statistical and machine learning algorithms; Apache Phoenix, for SQL-like queries for data in HBase; Apache Zeppelin, web-based notebooks for querying Phoenix tables; and more.

Learning resources

A review of the algorithms behind AutoML systems for model selection and hyperparameter optimization, from the H2O blog.

Joel Grus's criticisms of Jupyter Notebooks as a platform for reproducible and production-ready computing. Yihui Xiu offers an alternative: RMarkdown.

A comprehensive introduction to mixed-effects models, and fitting them in Python.

A collection of videos, presentations and essays by Brandon Rohrer with approachable explanations of the inner workings of deep learning and machine learning algorithms.

A tutorial on using Azure Batch AI to parallelize forecasts of energy demand.

A meticulously-researched accounting of the resources (natural, technological, and human) that enable an Alexa voice query, presented as highly-detailed map.

A review of the book SQL Server 2017 Machine Learning Services with R.

Applications

Data scientists use text similarity analyses in R to try and identify the author of that anonymous NYT op-ed.

Sketch2Code, an application to translate hand drawings into HTML forms.

Measuring building footprints from satellite images, with semantic segmentation.

Near real-time fraud detection for mobile banking, with a classification model implemented in Azure Machine Learning.

Credit card fraud detection with an Autoencoder neural network, using the Azure Data Science VM.

Shell deploys machine learning and AI systems to avert equipment failures, autonomously direct drill-bits underground, and improve safety.

Find previous editions of the monthly AI roundup here

Not Hotdog: A Shiny app using the Custom Vision API

I had a great time at the EARL Conference in London last week, and as always came away invigorated by all of the applications of R that were presented there. I'll do a full writeup of the conference later this week, but in the meantime I wanted to share the materials from my own presentation there, "Not Hotdog: Image Recognition with R and the Custom Vision API". I've embedded the slides below:

In the presentation, I showed how easy it is to build a Shiny app to perform image recognition with the Custom Vision API in Azure. I've provided all of the R code behind the app in this GitHub repository, and thanks to the httr package it takes just a few lines of R code to interface with the API to upload images, train a transfer learning model, and classify new images. It's also handy to be able to use the customvision.ai web portal to review your tagged images and model performance. 

This was my first time building a Shiny app, and it was really easy to learn how to do it thanks to the RStudio tutorials. One feature that proved particularly useful was reactive expressions, which would cache the results of the prediction and prevent a call out to the API every time I touched the "threshold" slider. The prediction API is rate-limited, and this way it was only called when the user actually changes the image URL, a relatively rare event. In all, it took me less than a day to learn Shiny and create the application below. Here it is correctly classifying this image of a hotdog (source), and correctly identifying that this hot dog (source) is not a hotdog.

If you want to try this out yourself, all you'll need is an Azure subscription (new subscribers can also get $200 free credits) and the R code in the repository below.

Github (revodavid): Not Hotdog vision recognition using R and Custom Vision API

Because it's Friday: Hurricane Trackers

With Hurricane Florence battering the US and Typhoon Manghkut bearing down on the Philippines, it's a good time to take a look at the art of visualizing predicted hurricane paths. (By the way, did you know that "typhoon", "hurricane" and "cyclone" are just different names for the same weather phenomenon?) Flowing Data has a good overview of the ways media have been visualizing the predicted path (hat tip: reader MB), including this animation from Axios which does a good job of demonstrating the uncertainty in the forecast:

A good thing to be aware of, though, is that the cones around the predicted tracks do not represent the size of the storm, but rather the uncertainty in the position of the center of the storm.

For a "live" view though, the place I like to look at is the global wind visualization from the Climate Literacy and Energy Awareness Network. Here's how Florence looks at this writing (3:45AM East Coast time). Click the image to see the current animated view.

That's all from us at the blog for this week. For those in the path of the storms, good luck and stay safe. 

 

How many deaths were caused by the hurricane in Puerto Rico?

President Trump is once again causing distress by downplaying the number of deaths caused by Hurricane Maria's devastation of Puerto Rico last year. Official estimates initially put the death toll at 15 before raising it to 64 months later, but it was clear even then that those numbers were absurdly low. The government of Puerto Rico commissioned an official report from the Millikan Institute of Public Health at George Washington University (GWU) to obtain a more accurate estimate, and with its interim publication official toll stands at 2,975. 

Why were the initial estimates so low? I read the interim GWU report to find out. The report itself is clearly written, quite detailed, and composed by an expert team of social and medical scientists, demographers, epidemiologists and biostatisticians, and I find its analysis and conclusions compelling. (Sadly however the code and data behind the analysis have not yet been released; hopefully they will become available when the final report is published.) In short:

• In the earliest days of the hurricane, the death-recording office was closed and without power, which suppressed the official count.
• Even once death certificates were collected, it became clear that officials throughout Puerto Rico has not been trained on how to record deaths in the event of a natural disaster, and most deaths were not attributed correctly in official records. 

Given these deficiencies in the usual data used to calculate death tolls (death certificates) the GWU team used a different approach to calculate the death toll. The basis of the method was to estimate excess mortality, in other words, how many deaths occurred in the post-Maria period compared to the number of deaths that would have been expected if it had never happened. This calculation required two quantitative studies:

• An estimate of what the population would have been if the hurricane hadn't happened. This was based on a GLM model of monthly data from the prior years, accounting for factors including recorded population, normal emigration and mortality rates.
• The total number of deaths in the post-Maria period, based on death certificates from the Puerto Rico government (irrespective of how the cause of death was coded).
• (A third study examined the communication protocols before, during and after the disaster. This study did not affect the quantiative conclusions, but formed the basis of some of the report's recommendations.)

The difference between the actual mortality, and the estimated "normal" mortality formed the basis for the estimate of excess deaths attributed to the hurricane. You can see those estimates of excess deaths one month, three months, and five months after the event in the table below; the last column represents the current official estimate.

These results are consistent in scale with another earlier study by Nishant Kishore et al. (The data and R code behind this study is available on GitHub.) This study attempted to quantify deaths attributed to the hurricane directly, by visiting 3299 randomly chosen households across Puerto Rico. At each household, inhabitants were asked about any household members who had died and their cause of death (related to or unrelated to the hurricane), and whether anyone had left Puerto Rico because of the hurricane. From this survey, the paper's authors extrapolated the number hurricane-related deaths to the entire island. The headline estimate of 4,625 at three months is somewhat larger than the middle column of the study above, but due to the small number of recorded deaths in the survey sample the 95% confidence interval is also much larger: 793 to 8498 excess deaths. (Gelman's blog has some good discussion of this earlier study, including some commentary from the authors.)

With two independent studies reporting excess deaths well into the thousands attributable directly to Hurricane Maria, it's a fair question to ask whether a more effective response before and after the storm could have reduced the scale of this human tragedy.

Milken Institute School of Public Health: Study to Estimate the Excess Deaths from Hurricane Maria in Puerto Rico

If not Notebooks, then what? Look to Literate Programming

Author and research engineer Joel Grus kicked off an important conversation about Jupyter Notebooks in his recent presentation at JupyterCon: 

There's no video yet available of Joel's talk, but you can guess the theme of that opening slide, and walking through the slides conveys the message well, I think. Yuhui Xie, author and creator of the rmarkdown package, provides a detailed summary and response to Joel's talk, where he lists Joel's main critiques of Notebooks: 

1. Hidden state and out-of-order execution
2. Notebooks are difficult for beginners
3. Notebooks encourage bad habits
4. Notebooks discourage modularity and testing
5. Jupyter’s autocomplete, linting, and way of looking up the help are awkward
6. Notebooks encourage bad processes
7. Notebooks hinder reproducible + extensible science
8. Notebooks make it hard to copy and paste into Slack/Github issues
9. Errors will always halt execution
10. Notebooks make it easy to teach poorly
11. Notebooks make it hard to teach well 

Yihui suggests that many of these shortcomings of Notebooks could be addressed through literate programming systems, where the document you edit is plain-text (and so easy to edit, manage, and track), and computations are strictly processed from the beginning of the document to the end. I use the RMarkdown system myself, and find it a delightful way of combining code, output and graphics in a single document, which can in turn be rendered in a variety of formats including HTML, PDF, Word and even PowerPoint.

Yihui expands on these themes in greater detail in his excellent book (with JJ Allaire and Garrett Grolemund), R Markdown: The Definitive Guide, published by CRC Press. Incidentally, the book itself is a fine example of literate programming; you can find the R Markdown source here, and you can read the book in its entirety here. As Joel mentions in his talk, an automatically-generated document of that length and complexity simply wouldn't be possible with Notebooks.

All that being said, RMarkdown is (for now) a strictly R-based system. Are there equivalent literate programming systems for Python? That's a genuine question — I don't know the Python ecosystem well enough to answer — but if you have suggestions please leave them in the comments.

Yihui Xie: The First Notebook War