Revolutions

AzureR updates: AzureStor, AzureVM, AzureGraph, AzureContainers

Some major updates to AzureR packages this week! As well as last week's AzureRMR update, there are changes to AzureStor, AzureVM, AzureGraph and AzureContainers. All of these are live on CRAN.

AzureStor 3.0.0

There are substantial enhancements to multiple-file transfers (up and down). You can supply a vector of pathnames to storage_upload/download as the source and destination arguments. Alternatively if you specify a wildcard source, there is now the ability to recurse through subdirectories; the directory structure in the source will be reproduced at the destination.

There are revamped methods for getting storage properties and (user-defined) metadata for storage objects.

You can now use Azurite with AzureStor, by creating a service_specific endpoint (file_endpoint, blob_endpoint, adls_endpoint) with the Azurite URL. AzureStor will print a warning, but create the endpoint anyway.

For other changes, see the NEWS.md file.

AzureVM 2.1.0

You can now create VM scalesets with attached data disks. In addition, you can specify the disk type (Standard_LRS, StandardSSD_LRS, or Premium_LRS) for the OS disk and, for a Linux Data Science Virtual Machine, the supplied data disk. This enables using VM sizes that don't support Premium storage.

AzureGraph 1.1.0 and AzureContainers 1.1.2

These packages have been updated to use the new Microsoft Graph operations, introduced last week, for managing app passwords. As a security measure, app passwords can no longer be manually specified; instead they are auto-generated on the server using a cryptographically secure PRNG.

In AzureGraph, the az_app$update_password() method is defunct; in its place are add_password() and remove_password(). Similarly, in AzureContainers the aks$update_service_password() and aks$update_aad_password() methods no longer accept a manual password as an input.

If you use Graph, and in particular if you use AzureContainers to deploy Azure Kubernetes Service clusters, you should update as soon as possible, as the old versions are incompatible with the new Graph API.

If you run into any problems, please open an issue at the GitHub repo in question.

AzureVM update: flexible and powerful deployment and management of VMs in Azure

by Hong Ooi, senior data scientist, Microsoft Azure

I'm happy to announce version 2.0 of AzureVM, a package for deploying and managing virtual machines in Azure. This is a complete rewrite of the package, with the objective of making it a truly generic and flexible tool for working with VMs and VM scale sets (clusters).

AzureVM 1.0 was a rather limited package in many respects: it came with only a small selection of DSVM templates, and didn't give you many options for changing them. While you could deploy any arbitrary template, this functionality was actually provided by the underlying AzureRMR package, rather than something that AzureVM added.

Here are the main changes in AzureVM 2.0:

• Separate out deployment of VMs and VM clusters; the latter are implemented as scalesets, rather than simplistic arrays of individual VMs. The methods to work with scalesets are named get_vm_scaleset, create_vm_scaleset and delete_vm_scaleset; get/create/delete_vm_cluster are now defunct.
• New UI for VM/scaleset creation, with many more ways to fine-tune the deployment options, including specifying the base VM image; networking details like security rules, load balancers and autoscaling; datadisks to attach; use of low-priority VMs for scalesets; etc.
• Several predefined configurations supplied to allow quick deployment of commonly used images (Ubuntu, Windows Server, RHEL, Debian, Centos, DSVM).
• Allow referring to existing resources in a deployment (eg placing VMs into an existing vnet), by supplying AzureRMR::az_resource objects as arguments.
• Clear distinction between a VM deployment template and a resource. get_vm and get_vm_scaleset will always attempt to retrieve the template; to get the resource, use get_vm_resource and get_vm_scaleset_resource.
• New VM resource methods: get_public_ip_address, get_private_ip_address.
• New cluster/scaleset resource methods: get_public_ip_address (technically the address for the load balancer, if present), get_vm_public_ip_addresses, get_vm_private_ip_addresses, list_instances, get_instance.
• Use a pool of background processes to talk to scalesets in parallel when carrying out instance operations. The pool size can be controlled with the global options azure_vm_minpoolsize and azure_vm_maxpoolsize.
  

See the README and/or the vignette for more information.

Here are some example deployments. A basic Ubuntu VM:

library(AzureVM)

sub <- AzureRMR::get_azure_login()$

    get_subscription("sub_id")

# default is an Ubuntu 18.04 VM, size Standard_DS3_v2, login via SSH key

# call sub$list_vm_sizes() to get the sizes available in your region

sub$create_vm("myubuntuvm", user_config("myname", "~/.ssh/id_rsa.pub"),

                    location="australiaeast")

A more complex deployment (Windows 10 Pro):

## this assumes you have a valid Win10 desktop license

user <- user_config("myname", password="Use-strong-passwords!")

image <- image_config(

     publisher="MicrosoftWindowsDesktop",

     offer="Windows-10",

     sku="19h1-pro"

)

datadisks <- list(

    datadisk_config(250, type="Premium_LRS"),

    datadisk_config(1000, type="Standard_LRS")

)

nsg <- nsg_config(

    list(nsg_rule_allow_rdp)

)

sub$create_vm("mywin10vm", user,

    config=vm_config(

        image=image,

        keylogin=FALSE,

        datadisks=datadisks,

        nsg=nsg,

        properties=list(licenseType="Windows_Client")

    ),

    location="australiaeast"

)

 

Creating a scaleset:

sub$create_vm_scaleset("myubuntuss", user_config("myname", "~/.ssh/id_rsa.pub"), instances=5,

                       location="australiaeast")

 

Sharing a subnet between a VM and a scaleset:

# first, create the resgroup

rg <- sub$create_resource_group("rgname", "australiaeast")

# create the master

rg$create_vm("mastervm", user_config("myname", "~/.ssh/id_rsa.pub"))

# get the vnet resource

vnet <- rg$get_resource(type="Microsoft.Network/virtualNetworks", name="mastervm-vnet")

# create the scaleset

# since the NSG is associated with the vnet, we don't need to create a new NSG either

rg$create_vm_scaleset("slavess", user_config("myname", "~/.ssh/id_rsa.pub"),

                      instances=5, vnet=vnet, nsg=NULL)

AzureVM 2.0 is available on CRAN now. If you have any questions, or if you run into problems, please feel free to email me or file an issue at the Github repo.

Fun with R and the Noops

Earlier this week, Github introduced Noops, a collection of simple black-box machines with API endpoints, with the goal of challenging developers of all skill levels to solve problems with them. Five "Noops" machines have been released so far along with challenges suitable for beginner programmers, with 15 further machines (and some more complex challenges) to be released over the coming three weeks. The bots available now are:

• Hexbot, which generates random RBG color strings and x/y coordinates
• Vexbot, which generates random endpoint coordinates of (possibly-connected) line segments
• Directbot, which will generate sequences of LOGO-like movements, either at random or according to a pattern you specify
• Drumbot, which generates commands for various pre-defined drumbeats
• Fizzbot, which tests your knowledge of classic programmer interview questions

Descriptions of the Noops and their APIs are provided on GitHub, along with sample code in Javascript and/or Python. However, as long as you can connect to the internet, you can use any language you like to solve the challenges. 

R is an ideal language for interfacing with these bots. The httr package for R makes it easy to interact with APIs using the GET operation, which is also what you use to control most Noops. (Fizzbot also uses the POST verb.) All I need to do is to load the package, and define the endpoint URL for Hexbot. And the data from the bots lends itself to graphical visualization, which is also easy to do with R. So I thought it would be fun to take Hexbot for a spin with R.

Calling the endpoint without any parameters generates a single RGB color specification (for example: #95DDFA), and this is easily done using the GET function from the httr package. Here, I define a function random.color to return a random color generated by Hexbot as a single R character. Luckily, you can pass RGB strings as colors to the base graphics functions in R, so I can use the barplot function to generate a block of random color:

Hexbot can generate more than one color at once, if you pass in the count parameter. For this, we just need to use the query option to the GET function. Here, we define a function to generate n random colors and extract the results as a vector, which we can then display with the barplot function once more:

You can also provide multiple parameters to the endpoint: if you also provide width and height, it will return random x and y locations within those ranges, along with the random colors. This adds some additional complexity when it comes to extracting the data from the payload, but with that done we can display the results as a scatterplot:

Of course, this is all rather pointless, in the sense that I could trivially use R itself to generate random points and colors. But still, it's a useful exercise to work through this yourself if you're not familiar with interacting with HTTP APIs. You'll learn several things along the way:

• about the HTTP GET operation and how to provide query parameters;
• how to navigate the nested lists that httr creates from the JSON content payload; and
• how to unroll those nested lists to extract the data you need in R.

The Noops yet to be revealed promise to offer more complex challenges, so I expect we'll learn about other aspects of HTTP APIs when they're to be released in the coming weeks as well.

Github: Meet the Noops

AzureR and AzureKeyVault

by Hong Ooi, senior data scientist, Microsoft Azure

Just a couple of announcements regarding my family of packages for working with Azure from R.

First, the packages have moved from the cloudyr org on GitHub to the Azure org, thus making them "official". A (rather spartan) homepage is here, containing links to the individual repos:

https://github.com/Azure/AzureR

The cloudyr repos will remain, but going forward they'll be mirrors of the main repos in Azure. Please submit issues and PRs to the Azure repos.

Second, the AzureKeyVault package is now available, on GitHub and on CRAN! This is an interface to the Key Vault service, which allows secure storage of cryptographic keys, certificates, and generic secrets. Both Resource Manager and client interfaces are provided. The package allows you to carry out operations such as encryption and decryption, certificate signing, and managing storage account keys.

Some sample code:

 

Any comments and feedback welcome.

Developments in AzureR

by Hong Ooi, senior data scientist, Microsoft Azure

The AzureR packages have now been on CRAN for a couple of months, so I thought I'd provide an update on developments in the works.

First, AAD authentication has been moved into a new package, AzureAuth, so that people who just want OAuth tokens can get it without any other baggage. This has many new features:

• Supports both AAD v1.0 and v2.0
• Tokens are cached in a user-specific directory using the rappdirs package, typically c:\users\<username>\local\AzureR on Windows and ~/.local/share/AzureR on Linux
• Supports 4 authentication methods: authorization_code, device_code, client_credentials and resource_grant
• Supports logging in with a username or with a certificate

In the longer term, the hope is for AzureAuth to be something like the R equivalent of the ADAL client libraries. Things to add include dSTS, federated logins, and more.

AzureRMR 2.0 has a new login framework and no longer requires you to create a service principal (although you can still provide your own SP if desired). Running create_azure_login() logs you into Azure interactively, and caches your credentials; in subsequent sessions, run get_azure_login() to login without having to reauthenticate.

AzureStor 2.0 has several new features mostly for more efficient uploading and downloading:

• Parallel file transfers, using a pool of background processes. This greatly improves the transfer speed when working with lots of small files.
• Transfer files to or from a local connection. This lets you transfer in-memory R objects without having to create a temporary file first.
• Experimental interface to AzCopy version 10. This lets you do essentially anything that AzCopy can do, from within R. (Note: AzCopy 10 is quite a different beast to AzCopy 8.x.)
• A new framework of generic methods, to organise all the various storage-type-specific functions.

A new AzureKusto package is in the works, for working with Kusto/Azure Data Explorer. This includes:

• All basic functionality including querying, engine management commands, and ingesting
• A dplyr interface written by Alex Kyllo.

AzureStor 2.0 is now on CRAN, and the others should also be there over the next few weeks. As always, if you run into any problems using the packages, feel free to contact me.

Use foreach with HPC schedulers thanks to the future package

The future package is a powerful and elegant cross-platform framework for orchestrating asynchronous computations in R. It's ideal for working with computations that take a long time to complete; that would benefit from using distributed, parallel frameworks to make them complete faster; and that you'd rather not have locking up your interactive R session. You can get a good sense of the future package from its introductory vignette or from this eRum 2018 presentation by author by Henrik Bengtsson (video embedded below), but at its simplest it allows constructs in R like this:

a %<-% slow_calculation(1:50)
b %<-% slow_calculation(51:100)
a+b

The idea here is that slow_calculation is an R function that takes a lot of time, but with the special %<-% assignment operator the computation begins and the R interpreter is ready immediately. The first two lines of R code above take essentially zero time to execute. The futures package farms off those computations to another process or even a remote system (you specify which with a preceding plan call), and R will only halt when the result is needed, as in the third line above. This is beneficial in Bengtsson's own work, where he uses the future package to parallelize cancer research on DNA sequences on high-performance computing (HPC) clusters.

The future package supports a wide variety of computation frameworks including parallel local R sessions, remote R sessions, and cluster computing frameworks. (If you can't use any of these, it falls back to evaluating the expressions locally, in sequence.) The future package also works in concert other parallel programming systems already available in R. For example, it provides future_lapply as a futurized analog of lapply, which will use whatever computation plan you have defined to run the computations in parallel.

The future package also extends the foreach package thanks to the updated doFuture package. By using registerDoFuture as the foreach backend, your loops can use any computation plan provided by the future package to run the iterations in parallel. (The same applies to R packages that use foreach internally, notably the caret package.) This means you can now use foreach with any of the HPC schedulers supported by future, which includes TORQUE, Slurm, and OpenLava. So if you you share a Slurm HPC cluster with colleagues in your department, you can queue up a parallel simulation on the cluster using code like this:

library("doFuture")
registerDoFuture()
library("future.batchtools")
plan(batchjobs_slurm)

mu <- 1.0
sigma <- 2.0
x <- foreach(i = 1:3, .export = c("mu", "sigma")) %dopar% {
  rnorm(i, mean = mu, sd = sigma)
}

The future package is available on CRAN now, and works consistently on Windows, Mac and Linux systems. You can learn more in the video at the end of this post, or in the recent blog update linked below.

JottR: Maintenance Updates of Future Backends and doFuture

AzureStor: an R package for working with Azure storage

by Hong Ooi, senior data scientist, Microsoft Azure

A few weeks ago, I introduced the AzureR family of packages for working with Azure in R. Since then, I’ve also written articles on how to use AzureRMR to interact with Azure Resource Manager, how to use AzureVM to manage virtual machines, and how to use AzureContainers to deploy R functions with Azure Kubernetes Service. This article is the next in the series, and covers AzureStor: an interface to Azure storage.

The Resource Manager interface: creating and deleting storage accounts

AzureStor implements an interface to Azure Resource Manager, which you can use manage storage accounts: creating them, retrieving them, deleting them, and so forth. This is done via the appropriate methods of the az_resource_group class. For example, the following code shows how you might create a new storage account from scratch.

library(AzureStor)

# get the resource group for the storage account
rg <- AzureRMR::az_rm$
    new(tenant="{tenant_id}", app="{app_id}", password="{password}")$
    get_subscription("{subscription_id}")$
    get_resource_group("myresourcegroup")

# create the storage account
# by default, this will be in the resource group's region
rg$create_storage_account("mynewstorage")

Without any options, this will create a storage account with the following parameters:

• General purpose account (all storage types supported)
• Locally redundant storage (LRS) replication
• Hot access tier (for blob storage)
• HTTPS connection required for access

You can change these by setting the arguments to create_storage_account(). For example, to create an account with geo-redundant storage replication and the default blob access tier set to “cool”:

AzureVM: managing virtual machines in Azure

by Hong Ooi, senior data scientist, Microsoft Azure

This is the next article in my series on AzureR, a family of packages for working with Azure in R. I’ll give a short introduction on how to use AzureVM to manage Azure virtual machines, and in particular Data Science Virtual Machines (DSVMs).

Creating a VM

Creating a VM is as simple as using the create_vm method, which is available as part of the az_subscription and az_resource_group classes.

library(AzureRMR)
library(AzureVM)

## using the subscription method
sub <- az_rm$
    new(tenant="{tenant_id}", app="{app_id}", password="{password}")$
    get_subscription("{subscription_id}")

myNewVM <- sub$create_vm("myNewVM",
    location="australiaeast",
    username="datascience",
    passkey="fAs30q-2a5vF!Z")  # be sure to choose a strong password!


## using the resource group method
rg <- sub$create_resource_group("myresourcegroup",
    location="australiaeast")

myOtherVM <- rg$create_vm("myOtherVM",
    username="datascience",
    passkey="l3Kgrf21%?0DFm")

Without any other options, this will create a Windows Server 2016 Data Science Virtual Machine, which is pre-installed with several tools useful for analytics: Python, R (and RStudio), Tensorflow, XGBoost, SQL Server, and so on. The size will be a Standard DS3 V2 VM, which has 4 cores, 14GB of memory, 1TB primary disk, and up to 16x28GB data disks.

AzureRMR: an R interface to Azure Resource Manager

In a previous article I announced AzureR, a new family of packages for working with Azure from R. This article goes into more detail on how you can use AzureRMR, the base package of the AzureR family, to manage resources with Azure Resource Manager.

Before you begin

The first thing you have to do is create a service principal. This is a security identity used by apps, services and automation tools for programmatic access to Azure resources. You can think of it as a “user identity” (login and password or certificate) with a role and permissions to access your resources. Creating a service principal is a one-time task, after which you can reuse it for all your AzureRMR sessions.

There are various ways to create a service principal. The easiest is to have your network administrator create it for you (which is probably also the best option in a corporate environment), but if you’re doing it yourself, you can do it via the Azure cloud shell.

AzureR: R packages to control Azure services

by Hong Ooi, senior data scientist, Microsoft Azure

This post is to announce a new family of packages we have developed as part of the CloudyR project for talking to Azure from R: AzureR.

As background, some of you may remember the AzureSMR package, which was written a few years back as an R interface to Azure. AzureSMR was very successful and gained a significant number of users, but it was never meant to be maintainable in the long term. As more features were added it became more unwieldy until its design limitations became impossible to ignore.

The AzureR family is a refactoring/rewrite of AzureSMR that aims to fix the earlier package's shortcomings.

The core package of the family is AzureRMR, which provides a lightweight yet powerful interface to Azure Resource Manager. It handles authentication (including automatically renewing when a session token expires), managing resource groups, and working with individual resources and templates. It also calls the Resource Manager REST API directly, so you don't need to have PowerShell or Python installed; it depends only on commonly used R packages like httr, jsonlite and R6.

Here's what some code using AzureRMR looks like. Note that it uses R6 classes to represent Azure objects, hence the use of $ to call methods.

library(AzureRMR)

az <- az_rm$new(tenant="{tenant_id}",
    app="{app_id}",
    password="{password}")

# get a subscription and resource group
sub <- az$get_subscription("{subscription_id}")
rg <- sub$get_resource_group("rgName")

# get a resource (storage account)
stor <- rg$get_resource(type="Microsoft.Storage/storageAccounts",
    name="myStorage")

# method chaining works too
stor <- az$
    get_subscription("{subscription_id}")$
    get_resource_group("rgName")$
    get_resource(type="Microsoft.Storage/storageAccounts",
        name="myStorage")

# create a new resource group and resource
rg2 <- sub$create_resource_group("newRgName", location="westus")

stor2 <- rg2$create_resource(type="Microsoft.Storage/storageAccounts",
    name="myNewStorage",
    kind="Storage", 
    sku=list(name="Standard_LRS", tier="Standard"))

# delete them
stor2$delete(confirm=FALSE)
rg2$delete(confirm=FALSE)

You can use AzureRMR to work with any Azure service, but a key idea behind it is that it's extensible: you can write new packages that provide extra functionality relevant to specific Azure offerings. This deals with one of the main shortcomings of AzureSMR: as a monolithic package, it became steadily more difficult to add new features as time went on. The packages that extend AzureRMR in this way constitute the AzureR family.

Currently, the AzureR family includes the following packages:

• AzureVM is a package for working with virtual machines and virtual machine clusters. It allows you to easily deploy, manage and delete a VM from R. A number of templates are provided based on the Data Science Virtual Machine, or you can also supply your own template.
• AzureStor is a package for working with storage accounts. In addition to a Resource Manager interface for creating and deleting storage accounts, it also provides a client interface for working with the storage itself. You can upload and download files and blobs, list file shares and containers, list files within a share, and so on. It supports authenticated access to storage via either a key or a shared access signature (SAS).
• AzureContainers is a package for working with containers in Azure: specifically, it provides an interface to Azure Container Instances, Azure Container Registry and Azure Kubernetes Service. You can easily create an image, push it to an ACR repo, and then deploy it as a service to AKS. It also provides lightweight shells to docker, kubectl and helm (assuming these are installed).

I'll be writing future blog posts to describe each of these in further detail.

All the AzureR packages are part of the CloudyR Project, which aims to make R work better with cloud services of all kinds. They are not (yet) on CRAN, but you can obtain them with devtools::install_github.

library(devtools)
install_github("cloudyr/AzureRMR")
install_github("cloudyr/AzureVM")
install_github("cloudyr/AzureStor")
install_github("cloudyr/AzureContainers")

If you use Azure with R, I hope you'll find these packages to be of significant benefit to your workflow. If you have any questions or comments, please contact me at [email protected].

Ed Note: This is the first post in a series. For more detail on the individual packages, here are links to the followup posts:

• AzureRMR: an R interface to Azure Resource Manager
• AzureVM: managing virtual machines in Azure
• AzureContainers: deploy a predictive service with containers
• AzureStor: an R package for working with Azure storage