Revolutions

Developments in AzureR

by Hong Ooi, senior data scientist, Microsoft Azure

The AzureR packages have now been on CRAN for a couple of months, so I thought I'd provide an update on developments in the works.

First, AAD authentication has been moved into a new package, AzureAuth, so that people who just want OAuth tokens can get it without any other baggage. This has many new features:

• Supports both AAD v1.0 and v2.0
• Tokens are cached in a user-specific directory using the rappdirs package, typically c:\users\<username>\local\AzureR on Windows and ~/.local/share/AzureR on Linux
• Supports 4 authentication methods: authorization_code, device_code, client_credentials and resource_grant
• Supports logging in with a username or with a certificate

In the longer term, the hope is for AzureAuth to be something like the R equivalent of the ADAL client libraries. Things to add include dSTS, federated logins, and more.

AzureRMR 2.0 has a new login framework and no longer requires you to create a service principal (although you can still provide your own SP if desired). Running create_azure_login() logs you into Azure interactively, and caches your credentials; in subsequent sessions, run get_azure_login() to login without having to reauthenticate.

AzureStor 2.0 has several new features mostly for more efficient uploading and downloading:

• Parallel file transfers, using a pool of background processes. This greatly improves the transfer speed when working with lots of small files.
• Transfer files to or from a local connection. This lets you transfer in-memory R objects without having to create a temporary file first.
• Experimental interface to AzCopy version 10. This lets you do essentially anything that AzCopy can do, from within R. (Note: AzCopy 10 is quite a different beast to AzCopy 8.x.)
• A new framework of generic methods, to organise all the various storage-type-specific functions.

A new AzureKusto package is in the works, for working with Kusto/Azure Data Explorer. This includes:

• All basic functionality including querying, engine management commands, and ingesting
• A dplyr interface written by Alex Kyllo.

AzureStor 2.0 is now on CRAN, and the others should also be there over the next few weeks. As always, if you run into any problems using the packages, feel free to contact me.

AI, Machine Learning and Data Science Roundup: February 2019

A monthly roundup of news about Artificial Intelligence, Machine Learning and Data Science. This is an eclectic collection of interesting blog posts, software announcements and data applications from Microsoft and elsewhere that I've noted over the past month or so.

Open Source AI, ML & Data Science News

ONNX, the open interchange format for AI models, updates to version 1.4.0 with support for models larger than 2Gb.

PT-BERT, a PyTorch implementation of Google's BERT language representation model, adds new pre-trained language models: GPT and Transformer-XL.

Stanford University has released StanfordNLP, a natural language analysis package for Python with pre-trained models for 53 languages.

The Python Foundation releases Python 3.7 on the Windows 10 App Store.

Industry News

Amazon offers recommendations to policymakers on the use of facial recognition technology and calls for regulation of its use.

AWS open-sources the Neo-AI project, a machine learning compiler and runtime that tunes Tensorflow, PyTorch, ONNX, MXNet and XGBoost models for performance on edge devices. 

Google introduces FEAST, an open-source "feature store" for managing and discovering features in machine learning models.

Google releases Natural Questions, a corpus of 300,000 questions and human-annotated answers, for question-answering research.

Uber open-sources Ludwig, a platform for training and testing deep learning models without programming, simply by specifying input and output variables in data.

Microsoft News

Microsoft has joined the SciKit-learn Consortium as a Platinum member.

Azure Data Explorer, a high-performance service for real-time analysis of streaming data, is now generally available.

HDInsight Tools for VSCode, providing a lightweight code editor for HDInsight PySpark and Hive batch jobs, is now available.

Azure Cognitive Services can now be used with a unified API key, and in more regions and with new certifications including HIPAA BAA.

An in-depth InfoWorld review of Azure ML services.

Learning resources

Papers with Code, a repository of machine learning papers with associated implementation code and evaluation metrics.

Practical Deep Learning for Coders, the popular free course from fast.ai, has been updated with all-new content. Microsoft Azure and GCP provide pre-configured environments with the required tools, data and materials.

Recommenders Hub: an open source GitHub repository containing Microsoft’s best practices and state-of-the-art algorithms for building recommendation systems.

Presentation videos and workshop materials from the recent RStudio conference are available.

Tutorial: running trained deep learning models on Apple devices, by converting CoreML exported from PyTorch into ONNX.

Tutorials from the MIT Deep Learning courses: Deep learning basics, Driving scene segmentation, DeepTraffic deep reinforcement learning.

Reference Architecture: Batch scoring of Spark models on Azure Databricks.

Applications

Facebook develops a predictive model for global electricity availability, publishes the output and some code.

The British Broadcasting Corporation releases an R package and a guide for data journalists on creating data visualizations in the BBC News style.

TWIMLAI's podcast series on Microsoft's AI for Earth , AI for Accessibility and AI for Humanitarian Action programs.

A profile of Heather Dowdy, the head of Microsoft's AI for Accessibility program.

Find previous editions of the monthly AI roundup here.

Create data visualizations like BBC News with the BBC's R Cookbook

If you're looking for a guide to making publication-ready data visualizations in R, check out the BBC Visual and Data Journalism cookbook for R graphics. Announced in a BBC blog post this week, it provides scripts for making line charts, bar charts, and other visualizations like those below used in the BBC's data journalism. 

The cookbook is based around the bbplot package (available on Github), which has been in use at the BBC since March 2018 for creating graphics in the BBC News graphics style. Rather than drafting graphics in R and then relying another graphics tool or a design team for publication, R has been a "game changer" for the BBC. It offers more creativity and control while saving time and effort, especially when the chart needs to be reproduced every time the underlying data change. The BBC offers a six-week course for its data journalists internally, and you can learn as well with the free cookbook published by the BBC, linked below.

Github (BBC): BBC Visual and Data Journalism cookbook for R graphics

 

 

Tutorial: Sequential Pattern Mining in R for Business Recommendations

by Allison Koenecke, Data Scientist, AI & Research Group at Microsoft, with acknowledgements to Amita Gajewar and John-Mark Agosta.

In this tutorial, Allison Koenecke demonstrates how Microsoft could recommend to customers the next set of services they should acquire as they expand their use of the Azure Cloud, by using a temporal extension to conventional Market Basket Analysis. 

Problem Statement

Market Basket Analysis (MBA) answers a standard business question: given a set of grocery store receipts, can we find bundles of products often purchased together (e.g., peanut butter and jelly)?  Suppose we instead want to model the evolution of a customer’s services, such as determining whether buying peanut butter in the past indicates a higher likelihood of buying bread in the future.  For this, we apply a sequential version of MBA, sometimes called “sequential itemset mining” or “sequential pattern mining”, to introduce a time component to the analysis [1]. Sequential itemset mining has been applied across many industries, from determining a patient’s sequence of medical prescriptions [2] to detecting misuse intrusions such as application layer attacks [3]. In this tutorial, when given a series of purchases over time, we determine whether we can (1) find product bundles that we expect to be consumed simultaneously, and also (2) examine how these bundles evolve over time.

In the following tutorial, we answer both questions using the R package arulesSequences [4], which implements the SPADE algorithm [5]. Concretely, given data in an Excel spreadsheet containing historical customer service purchase data, we produce two separate Excel sheet deliverables: a list of service bundles, and a set of temporal rules showing how service bundles evolve over time.  We will focus on interpreting the latter result by showing how to use temporal rules in making predictive sales recommendations.

Our running example below is inspired by the need for Microsoft’s Azure Services salespeople to suggest which additional products to recommend to customers, given the customers’ current cloud product consumption services mix.  We’d like to know, for instance, if customers who have implemented web services also purchase web analytics within the next month.  Actual Azure Service names have been removed for confidentiality reasons.

Step 1: Reformat Data into Transaction Matrix