Revolutions

AI, Machine Learning and Data Science Roundup: June 2019

A monthly roundup of news about Artificial Intelligence, Machine Learning and Data Science. This is an eclectic collection of interesting blog posts, software announcements and data applications from Microsoft and elsewhere that I've noted over the past month or so.

Open Source AI, ML & Data Science News

Tensorflow 2.0 beta is now available, featuring first-class Keras support and with eager execution enabled by default.

MLflow 1.0, the open source platform for managing end-to-end machine learning lifecycles from Databricks, is now available.

Industry News

Facebook open sources PyRobot, a framework that enables AI researchers and students to control a physical robot with a few lines of Python code.

Databricks Connect, a new universal Spark client library with bindings in Python, Scala, Java and R to manage data and compute in hosted Databricks environments.

Google introduces Deep Learning Containers (beta), which come with a Jupyter environment pre-configured to interact with the GCP AI platform.

GCP AI Platform Notebooks, Google's managed JupyterLab service, now support the R language (beta).

Amazon releases GluonTS, an open-source Python toolkit for building deep-learning based time series models.

Microsoft News

Microsoft introduces Immersive Reader, a new Azure Cognitive Service that allows developers to provide assisted reading experiences to non-native speakers and people with dyslexia, ADHD, or visual impairment.

Microsoft open-sources TensorWatch, a Jupyter-based debugging and visualization tool to observe machine learning and deep learning training in process.

Azure Machine Learning Service introduces new time series forecasting capabilities.

Power BI adds AI capabilities: cognitive text and image analysis, and consumption of models from Azure ML Services.

InterpretML, a new open-source Python package for training interpretable models and explaining black-box systems from Microsoft Research.

MLOps, an extension to Azure DevOps for orchestration and management of models in Azure ML Service, such as this Video Anomaly Detection example.

Learning resources

ONNX.JS: demos with code of running GPU-accelerated inference on ONNX models in the browser.

Tutorial: using the new Automated Machine Learning web user interface in the Azure portal.

Video recordings from the 2019 New York R Conference are available to view.

Feature Engineering and Selection, a new book Max Kuhn and Kjell Johnson with implementations in R.

An overview of datatable, Python's version of the R package for efficient, multithreaded processing of out-of-memory datasets.

Learn R and Python in Parallel, an online book by Nailong Zhang useful for learning one language based on your knowledge of the other.

Mastering Shiny, a new book in progress by Joe Cheng, developer of the interactive UI framework for R.

Exploring Data with R, an introduction to R from MSDN Magazine.

Applications

A new paper by leading researchers suggests 10 domains where AI could be applied to address the threat of climate disruption.

Uber uses causal inference in product development, operations analysis and improving user experiences.

The Future Computed: AI and Manufacturing, a 135-page Microsoft e-book featuring applications of AI in manufacturing.

Try out GauGAN, NVIDIA's style transfer algorithm that converts a crude finger-painting into a realistic landscape.

MASS, a new pre-training method that outperforms BERT and GPT in generating realistic natural language text.

FUNIT (Few-Shot Unsupervised Image-to-Image Translation), a NVIDIA research project used to convert images of one animal (or even a human face) to other breeds/species.

Neural Code Search, a Facebook model for using natural language search to find relevant computer code.

Google Research Football, an open-source reinforcement learning simulator to teach an AI agent to play a computer soccer game.

The Dalí Museum in Florida creates an interactive simulation of the iconic artist from historical footage (video).

Editor's note: The monthly roundup will return in August. Find previous editions of the monthly AI roundup here.

AzureVM update: flexible and powerful deployment and management of VMs in Azure

by Hong Ooi, senior data scientist, Microsoft Azure

I'm happy to announce version 2.0 of AzureVM, a package for deploying and managing virtual machines in Azure. This is a complete rewrite of the package, with the objective of making it a truly generic and flexible tool for working with VMs and VM scale sets (clusters).

AzureVM 1.0 was a rather limited package in many respects: it came with only a small selection of DSVM templates, and didn't give you many options for changing them. While you could deploy any arbitrary template, this functionality was actually provided by the underlying AzureRMR package, rather than something that AzureVM added.

Here are the main changes in AzureVM 2.0:

• Separate out deployment of VMs and VM clusters; the latter are implemented as scalesets, rather than simplistic arrays of individual VMs. The methods to work with scalesets are named get_vm_scaleset, create_vm_scaleset and delete_vm_scaleset; get/create/delete_vm_cluster are now defunct.
• New UI for VM/scaleset creation, with many more ways to fine-tune the deployment options, including specifying the base VM image; networking details like security rules, load balancers and autoscaling; datadisks to attach; use of low-priority VMs for scalesets; etc.
• Several predefined configurations supplied to allow quick deployment of commonly used images (Ubuntu, Windows Server, RHEL, Debian, Centos, DSVM).
• Allow referring to existing resources in a deployment (eg placing VMs into an existing vnet), by supplying AzureRMR::az_resource objects as arguments.
• Clear distinction between a VM deployment template and a resource. get_vm and get_vm_scaleset will always attempt to retrieve the template; to get the resource, use get_vm_resource and get_vm_scaleset_resource.
• New VM resource methods: get_public_ip_address, get_private_ip_address.
• New cluster/scaleset resource methods: get_public_ip_address (technically the address for the load balancer, if present), get_vm_public_ip_addresses, get_vm_private_ip_addresses, list_instances, get_instance.
• Use a pool of background processes to talk to scalesets in parallel when carrying out instance operations. The pool size can be controlled with the global options azure_vm_minpoolsize and azure_vm_maxpoolsize.
  

See the README and/or the vignette for more information.

Here are some example deployments. A basic Ubuntu VM:

library(AzureVM)

sub <- AzureRMR::get_azure_login()$

    get_subscription("sub_id")

# default is an Ubuntu 18.04 VM, size Standard_DS3_v2, login via SSH key

# call sub$list_vm_sizes() to get the sizes available in your region

sub$create_vm("myubuntuvm", user_config("myname", "~/.ssh/id_rsa.pub"),

                    location="australiaeast")

A more complex deployment (Windows 10 Pro):

## this assumes you have a valid Win10 desktop license

user <- user_config("myname", password="Use-strong-passwords!")

image <- image_config(

     publisher="MicrosoftWindowsDesktop",

     offer="Windows-10",

     sku="19h1-pro"

)

datadisks <- list(

    datadisk_config(250, type="Premium_LRS"),

    datadisk_config(1000, type="Standard_LRS")

)

nsg <- nsg_config(

    list(nsg_rule_allow_rdp)

)

sub$create_vm("mywin10vm", user,

    config=vm_config(

        image=image,

        keylogin=FALSE,

        datadisks=datadisks,

        nsg=nsg,

        properties=list(licenseType="Windows_Client")

    ),

    location="australiaeast"

)

 

Creating a scaleset:

sub$create_vm_scaleset("myubuntuss", user_config("myname", "~/.ssh/id_rsa.pub"), instances=5,

                       location="australiaeast")

 

Sharing a subnet between a VM and a scaleset:

# first, create the resgroup

rg <- sub$create_resource_group("rgname", "australiaeast")

# create the master

rg$create_vm("mastervm", user_config("myname", "~/.ssh/id_rsa.pub"))

# get the vnet resource

vnet <- rg$get_resource(type="Microsoft.Network/virtualNetworks", name="mastervm-vnet")

# create the scaleset

# since the NSG is associated with the vnet, we don't need to create a new NSG either

rg$create_vm_scaleset("slavess", user_config("myname", "~/.ssh/id_rsa.pub"),

                      instances=5, vnet=vnet, nsg=NULL)

AzureVM 2.0 is available on CRAN now. If you have any questions, or if you run into problems, please feel free to email me or file an issue at the Github repo.

Fun with R and the Noops

Earlier this week, Github introduced Noops, a collection of simple black-box machines with API endpoints, with the goal of challenging developers of all skill levels to solve problems with them. Five "Noops" machines have been released so far along with challenges suitable for beginner programmers, with 15 further machines (and some more complex challenges) to be released over the coming three weeks. The bots available now are:

• Hexbot, which generates random RBG color strings and x/y coordinates
• Vexbot, which generates random endpoint coordinates of (possibly-connected) line segments
• Directbot, which will generate sequences of LOGO-like movements, either at random or according to a pattern you specify
• Drumbot, which generates commands for various pre-defined drumbeats
• Fizzbot, which tests your knowledge of classic programmer interview questions

Descriptions of the Noops and their APIs are provided on GitHub, along with sample code in Javascript and/or Python. However, as long as you can connect to the internet, you can use any language you like to solve the challenges. 

R is an ideal language for interfacing with these bots. The httr package for R makes it easy to interact with APIs using the GET operation, which is also what you use to control most Noops. (Fizzbot also uses the POST verb.) All I need to do is to load the package, and define the endpoint URL for Hexbot. And the data from the bots lends itself to graphical visualization, which is also easy to do with R. So I thought it would be fun to take Hexbot for a spin with R.

Calling the endpoint without any parameters generates a single RGB color specification (for example: #95DDFA), and this is easily done using the GET function from the httr package. Here, I define a function random.color to return a random color generated by Hexbot as a single R character. Luckily, you can pass RGB strings as colors to the base graphics functions in R, so I can use the barplot function to generate a block of random color:

Hexbot can generate more than one color at once, if you pass in the count parameter. For this, we just need to use the query option to the GET function. Here, we define a function to generate n random colors and extract the results as a vector, which we can then display with the barplot function once more:

You can also provide multiple parameters to the endpoint: if you also provide width and height, it will return random x and y locations within those ranges, along with the random colors. This adds some additional complexity when it comes to extracting the data from the payload, but with that done we can display the results as a scatterplot:

Of course, this is all rather pointless, in the sense that I could trivially use R itself to generate random points and colors. But still, it's a useful exercise to work through this yourself if you're not familiar with interacting with HTTP APIs. You'll learn several things along the way:

• about the HTTP GET operation and how to provide query parameters;
• how to navigate the nested lists that httr creates from the JSON content payload; and
• how to unroll those nested lists to extract the data you need in R.

The Noops yet to be revealed promise to offer more complex challenges, so I expect we'll learn about other aspects of HTTP APIs when they're to be released in the coming weeks as well.

Github: Meet the Noops

AzureR and AzureKeyVault

by Hong Ooi, senior data scientist, Microsoft Azure

Just a couple of announcements regarding my family of packages for working with Azure from R.

First, the packages have moved from the cloudyr org on GitHub to the Azure org, thus making them "official". A (rather spartan) homepage is here, containing links to the individual repos:

https://github.com/Azure/AzureR

The cloudyr repos will remain, but going forward they'll be mirrors of the main repos in Azure. Please submit issues and PRs to the Azure repos.

Second, the AzureKeyVault package is now available, on GitHub and on CRAN! This is an interface to the Key Vault service, which allows secure storage of cryptographic keys, certificates, and generic secrets. Both Resource Manager and client interfaces are provided. The package allows you to carry out operations such as encryption and decryption, certificate signing, and managing storage account keys.

Some sample code:

 

Any comments and feedback welcome.