Revolutions

Spooky! Gravedigger in R

Need something to distract the kids while they're waiting to head out Trick-or-Treating? You could have them try out a Creepy Computer Game in R! Engineer and social scientist Dr Peter Provost translated one of the old BASIC games from the classic book Creepy Computer Games into R: just have them type in this file (and yes, they do need to type it character-by-character to get the full experience) and then source it into R to play the game. The goal is to move your character * to the exit at the SE corner of the graveyard populated by gravestones + while avoiding the skeletons X.

For extra credit, see if they can figure out the bug that causes the move counter to go down, instead of up. It might help to refer to the orginal BASIC sources -- the book is online at the publisher's website as a PDF here.

The Lucid Manager: Celebrate Halloween with Creepy Computer Games in R

Fringe FM conversation on AI Ethics

A few weeks ago, I had a lively conversation with Matt Ward for the Fringe FM podcast, where we discussed artificial intelligence, its applications, and the ethical implications thereof. 

During the conversation I mentioned offhand an article I'd read recently which suggested that program to identify criminals using face recognition in CCTV suffered from a high rate of misidentifications. I couldn't remember the exact rate at the time, and in the quote shown below I said "50% or 90%". Turns out I was being too conservative: the actual rate was 98%. Just as with human-based systems, no AI system is perfect, and classifications are based on a confidence score exceeding some threshold. Set the threshold too low, as was undoubtedly the case here, and the result will be many false positives. It's always important for AI developers to consider the impact of false positives and false negatives, and to take particular care to consider the impact of those negative determinations, especially when they relate to people's lives.

During the podcast I also mentioned the Microsoft group headed by Kate Crawford which is doing some of the fundamental research into AI ethics, but failed to mention the name of the group: Fairness, Accountability, Transparency and Ethics in AI (FATE). You can find their publications in the Microsoft Research Catalog — the AI Now Report in particular is worth reading for its recommendations.

You can find the Fringe FM podcast in your favorite podcast app (mine is Pocket Casts), or listen to the Episode 46 directly at the link below.

Fringe FM: 46. The Ethics of AI in an Era of Big Data | David Smith

Because it's Friday: Parable of the Polygons

What if we lived in a society where everyone really happy about living in a diverse neighborhood? What if people only wanted to move when the disparity was really extreme: say, when fewer than 33% of people nearby looked like them? Well, we'd end up with a society like this:

Still pretty segregated, as it turns out. At Parable of The Polygons (via Miles McBain), an interesting interactive website, you can experiment with the levels of individual bias that lead the eponymous Polygons to switch locations, and see the impact on the overall Polygon society. It's quite enlightening.

That's all from us at the blog for this week. Have a great weekend, and we'll be back next week.

 

 

AI, Machine Learning and Data Science Roundup: October 2018

A monthly roundup of news about Artificial Intelligence, Machine Learning and Data Science. This is an eclectic collection of interesting blog posts, software announcements and data applications from Microsoft and elsewhere that I've noted over the past month or so.

Open Source AI, ML & Data Science News

PyTorch 1.0 is now in preview, and brings a streamlined workflow from model development to production. Relatedly, Sam Charrington compares the growing PyTorch ecosystem with that of Tensorflow.

Tensorflow 1.11 is released, with binaries for cuDNN 7.2 and TensorRT 4, and new functions for querying kernels.

fastai, a new open library for deep learning built on PyTorch, has been released by fast.ai. It provides a single, consistent API for many deep learning applications and data types.

The R packages keras and tensorflow now allow definition of models that take advantage of eager execution.

Anaconda can now install and manage Tensorflow as a conda package.

Industry News

PyXLL, a commercial Excel add-in for calling Python functions from Excel.

IBM releases AI Fairness 360, an open source toolkit for investigating and mitigating bias in machine learning models.

Google open sources a TensorFlow package for Active Question Answering, a reinforcement learning based method to train artificial agents to answer natural-language questions.

Google Cloud Platform adds PyTorch support to several services, including Tensorboard, Kubeflow and Deep Learning VM images.

RStudio Package Manager, a commercial product to help organizations manage R packages, has been released.

RStudio adds support for Stan, the Bayesian modeling language.

Databricks releases MLflow 0.7.0, featuring a new R client API that allows you to log parameters, code versions, metrics, and output files when running R code and visualize the results in MLflow.

Cloudera and Hortonworks are merging. Thomas Dinsmore assesses the impact on the machine learning ecosystem.

Dataiku 5 is released. The enterprise data platform adds support for containerized R and Python recipes, and integration with Tensorflow and Keras.

Microsoft News

Microsoft has opened a new AI research facility in Shanghai in conjunction with INESA, the China organization developing smart city solutions.

SQL Server 2019 will provide integration with "big data clusters": external SQL Server, Spark or HDFS containers managed in Kubernetes. For AI and ML workloads, you can run Spark or SQL Server's R, Python and Java extensions against the cluster data.

Azure Machine Learning Services now provides Python developer libraries for data preparation, experiment tracking and model management, training on GPU clusters, automated model search and hyperparameter optimization, deploying trained models as containers, and many other new capabilities.

Visual Studio Code Tools for AI has been updated to provide a convenient interface to Azure Machine Learning for users of the popular open-source editor.

Speech Service in Azure Cognitive Services is now generally available, and includes a new neural text-to-speech capability for humanlike synthesized spech.

Azure Databricks is now supported in more regions, offers GPU support for deep learning, and Databricks Delta is now available (in preview) for transactional data capabilities.

Microsoft Bot Framework SDK v4 is now available. The Ignite presentation "Creating Enterprise-Scale Intelligent Agents and Bots" provides an overview and several examples.

Cortana Skills Kit for Enterprise, a development platform based on Azure Bot Service and Language Understanding, is now in private preview.  

ONNX Runtime, a high-performance engine for executing trained models represented in the open ONNX format, is now in preview.

PyTorch is now supported in many Azure services, including Azure Machine Learning service, Data Science Virtual Machine, Azure Notebooks, and Visual Studio Code Tools for AI.

The Microsoft Infer.NET machine learning framework has been released as open source. An online book, Model-Based Machine Learning, describes its probabilistic approach with several in-depth examples.

Learning resources

Microsoft Learn, which provides free interactive training for the Azure platform, is now available. Modules include Deep Learning with PyTorch and Computer Vision with Tensorflow.

A tutorial on converting a PyTorch model to ONNX, the cross-platform model-sharing format.

Chromebook Data Science, a free course in Data Science and R from the Johns Hopkins Data Science lab that requires only a browser to take.

Applications

The Ethics Certification Program for Autonomous and Intelligent Systems, a new IEEE standard with the goal of advancing transparency and reducing algorithmic bias in AI systems.

The fast.ai research datasets collection, which includes MNIST, CIFAR 10 and Imagenet, is now available on AWS Open Data.

A comprehensive list of AI Ethics resources, published by fast.ai.

Snip Insights, a Microsoft AI Lab project for image analysis and text extraction from screenshots.

Azure Healthcare AI blueprint, a process to deploy a HIPAA and HITRUST compliant environment in Azure for managing and running healthcare AI experiments.

Find previous editions of the monthly AI roundup here

When the numbers don't tell the whole story

Anscombe's Quartet is a famous collection of four small data sets — just 11 (x,y) pairs each — that was developed in the 1970s to emphasize the fact that sometimes, numerical summaries of data aren't enough. (For a modern take on this idea, see also the Datasaurus Dozen.) In this case, it takes visualizing the data to realize that the for data sets are qualitatively very different, even though the means, variances, and regression coefficients are all the same. In the video below for Guy in a Cube, Buck Woody uses R to summarize the data (which is conveniently built into R) and visualize it using an R script in Power BI.

The video also makes for a nice introduction to R for those new to statistics, and also demonstrates using R code to generate graphics in Power BI.

Computer Vision for Model Assessment

One of the differences between statistical data scientists and machine learning engineers is that while the latter group are concerned primarily with the predictive performance of a model, the former group are also concerned with the fit of the model. A model that misses important structures in the data — for example, seasonal trends, or a poor fit to specific subgroups — is likely to be lacking important variables or features in the source data. You can try different machine learning techniques or adjust hyperparameters to your heart's content, but you're unlikely to discover problems like this without evaluating the model fit.

One of the most powerful tools for assessing model fit is the residual plot: a scatterplot of the predicted values of the model versus the residuals (the difference between the predictions and the original data). If the model fits well, there should be no obvious relation between the two. For example, visual inspection shows us that the model below first fairly well for 19 of the 20 subgroups in the data, but there may be a missing variable that would explain the apparent residual trend for group 19.

Assessing residual plots like these is something of an art, which is probably why it isn't routine in machine learning circles. But what if we could use deep learning and computer vision to assess residual plots like these automatically? That's what Professor Di Cook from Monash University proposes, in the 2018 Belz Lecture for the Statistical Society of Australia, Human vs computer: when visualising data, who wins?  

(The slides for the presentation, created using R, are also available online.) The first part of the talk also includes a statistical interpretation of neural networks, and an intuitive explanation of deep learning for computer vision. The talk also references a related  paper, Visualizing Statistical Models : Removing the Blindfold (by Hadley Wickham, Dianne Cook and Jeike Hofman and published in the ASA Data Science Journal in 2015) which is well worth a read as well.

Maryland's Bridge Safety, reported using R

A front-page story in the Baltimore Sun reported last week on the state of the bridges in Maryland. Among the report's findings:

• 5.4% of bridges are classified in "poor" or "structurally deficient" condition
• 13% of bridges in the city of Baltimore are in "poor" condition

Those findings were the result of analysis of Federal infrastructure data by reporter Christine Zhang. The analysis was performed using R and documented in a Jupyter Notebook published on Github. The raw data included almost 50 variables including type, location, ownership, and inspection dates and ratings, and required a fair bit of processing with tidyverse functions to extract the Marlyland-specific statistics above. The analysis also turned up an unusual owner for one of the bridges: this one — an access road to the Goddard Space Flight Center — is owned by NASA.

You can read the story in the Baltimore Sun, or check out the R analysis in the Github repo linked below.

Github (Baltimore Sun): Maryland bridges analysis (via Sharon Machlis)

A small logical change with big impact

In R, the logical || (OR) and && (AND) operators are unique in that they are designed only to work with scalar arguments. Typically used in statements like

while(iter < 1000 && eps < 0.0001) continue_optimization()

the assumption is that the objects on either side (in the example above, iter and eps) are single values (that is, vectors of length 1) — nothing else makes sense for control flow branching like this. If either iter or eps above happened to be vectors with more than one value, R would silently consider only the first elements when making the AND comparison.

In an ideal world that should never happen, of course. But R programmers, like all programmers, do make errors, and using non-scalar values with || or && is a good sign that something, somewhere, has gone wrong. And with an experimental feature in the next version of R, you will be able to set an environment variable, _R_CHECK_LENGTH_1_LOGIC2_, to signal a warning or error in this case. But why make it experimental, and not the default behavior? Surely making such a change is a no-brainer, right?

It turns out things are more difficult than they seem. The issue is the large ecosystem of R packages, which may rely (wittingly or unwittingly) on the existing behavior. Suddenly throwing an error would throw those packages out of CRAN, and all of their dependencies as well.

We can see the scale of the impact in a recent R Foundation blog post by Tomas Kalibara where he details the effort it took to introduce an even simpler change: making if(cond) { ... } throw an error if cond is a logical vector of length greater than one. Again, it seems like a no-brainer, but when this change was introduced experimentally in March 2017, 154 packages on CRAN started failing because of it ... and that rose to 179 packages by November 2017. The next step was to implement a new CRAN check to alert those package authors that, in the future, their package would fail. It wasn't until October 2018 that the number of affected packages had dropped to a suitable level that the error could actually be implemented in R as well.

So the lesson is this: with an ecosystem as large as CRAN, even "trivial" changes take a tremendous amount of effort and time. They involve testing the impact on CRAN packages, coding up tests and warnings for maintainers, and allowing enough time for packages to be udpated. For an in-depth perspective, read Tomas Kalibera's essay at the link below.

R Developer blog: Tomas Kalibera: Conditions of Length Greater Than One

Because it's Friday: Hey, it's Enrico Pallazzo!

It seemed like such a simple movie. The Naked Gun (1988) is slapstick comedy through-and-through, but I never would have guessed (h/t Steven O'Grady) how much detail and planning went into the jokes, especially the baseball scene at the end. There's lots of interesting behind-the-scenes info in Sporting News's breakdown of the movie. Even Drebin's bungled National Anthem performance was composed in advance for the scene:

That's all from us for this week. Next week, I'm in Australia for the Melbourne R User Group and the eResearch Australiasia conference, so I expect blogging will be a little lighter than usual. Have a great weekend!

The Economist's Big Mac Index is calculated with R

The Economist's Big Mac Index (also described on Wikipedia if you're not a subscriber) was created (somewhat tongue-in-cheek) as a measure to compare the purchasing power of money in different countries. Since Big Macs are available just about everywhere in the world, the price of a Big Mac in Sweden — expressed in US dollars — gives an American traveler a sense of how much more expensive things will be in Stockholm. And comparing the price of a Big Mac in several countries converted to a single baseline currency is a measure of how over-valued (or undervalued) those other currencies are compared to that baseline.

Since its inception in 1986, the Big Mac Index has been compiled and calculated manually, twice a year. But starting with the most recent published index (July 2018, shown above), the index is now calculated with R. This is the first example of a new program at The Economist to publish the data and methods behind its journalism, and here the data and code behind the Big Mac Index have been published as a Github repository. The method behind the index is provided as a richly-commented Jupyter Notebook, where you can also find some additional charts and within-currency analyses not published in the main index.

The repository is published under an open MIT license, meaning you can remix the code to create a new index on prices of another international commodity, provided you can find the data. Find everything you need at the Github repository linked below.

Github (TheEconomist): Data and methodology for the Big Mac index