Revolutions

AI, Machine Learning and Data Science Announcements from Microsoft Ignite

Microsoft Ignite, Microsoft's annual developer conference, wrapped up last week and many of the big announcements focused on artificial intelligence and machine learning. The keynote presentation from Microsoft's Cloud AI lead Eric Boyd showcases the major developments, or you can check out his accompanying blog post for a written summary. 

In this post, I'll dive down into the details of the specific announcements and provide links to the associated recorded presentations from Ignite and other technical documents.

Tools for AI developers

Azure Machine Learning Service (Azure ML) adds several new capabilities. These include a new Python package that developers can use to prepare data for analysis, automate machine learning and deep learning model training, track model performance under various conditions, and automate the model selection process (see below). It supports a variety of execution environments, from local desktops to GPU clusters, and deploy trained models with containers or even high-performance FPGA chips.  (The desktop "Workbench" client, previously in preview, is no longer required and has been deprecated.) Watch the Ignite presentation AI with Azure Machine Learning services: Simplifying the data science process for an overview and demo.

Automated Machine Learning, a new Azure ML service in preview, automates the selection of machine learning algorithms (and associated parameters and hyperparameters) to optimize predictive performance. This blog post provides the technical details, and another blog post describes some practical applications. A Microsoft Research paper describes the probabilistic matrix factorization technique behind the optimization. This Ignite presentation provides further details and includes a demo.  

Visual Studio Code Tools for AI has been updated to provide a convenient interface to Azure ML for users of the popular open-source editor.

SQL Server 2019 is now in public preview, with many improvements shown in this Ignite keynote. Updates include Spark support, and added Linux support for in-database R and Python (watch this Ignite presentation for details).

Azure Databricks is now supported in more regions, offers GPU support for deep learning, and Databricks Delta is now available (in preview) for transactional data capabilities.

Customized AI

Microsoft Bot Framework SDK v4 is now available. The Ignite presentation "Creating Enterprise-Scale Intelligent Agents and Bots" provides an overview and several examples.

Cortana Skills Kit for Enterprise, a development platform based on Azure Bot Service and Language Understanding is now in private preview.  

Pre-trained AI

Speech Service in Azure Cognitive Services is now generally available, and includes a new neural text-to-speech capability for humanlike synthesized speech. This Ignite presentation provides an overview.

This Ignite presentation provides an overview of Cognitive Services with some demos with applications of the pre-trained AI capabilities for computer vision, speech, search and more.

Programs

AI for Humanitarian Action announced. With this new AI for Good program, MIcrosoft will dedicate $40m over five years to deploy AI technologies to assist with disaster recovery; the needs of children, refugees and displaced people; and human rights. A short video summarizes the initiative.

And that's just the highlights — for example, I haven't included the many AI capabilities added to products like Windows and Office. (However I can't resist mentioning the new Insert Data from Picture feature in Excel, which I'm sure I'll be making a lot of use of.) For more from Ignite, take a look at Buck Woody's index of presentations at Ignite, with highlights of videos focused on AI, machine learning, and data science.

Who wrote that anonymous NYT op-ed? Text similarity analyses with R

In US politics news, the New York Times took the unusual step this week of publishing an anonymous op-ed from a current member of the White House (assumed to be a cabinet member or senior staffer). Speculation about the identity of the author is, of course, rife. Much of the attention has focused on the use of specific words in the article, but can data science provide additional clues? In the last 48 hours, several R users have employed text similarity analysis to try and identify likely culprits.

David Robinson followed a similar process to the one he used to identify which of President Trump's tweets were written by him personally (as opposed to a staffer). He downloaded the tweets written by 69 Cabinet members and WH staff who maintain accounts on on Twitter (about 1.5 million words total). Then, he used the TD-IDF (term-frequency inverse-document-frequency) distance measure to find the rate at which certain words are more frequently (or less frequently) by the each subject. Comparing those rates to the rate of use in the op-ed. After ruling out the VP (and the President himself), this analysis suggests Secretary of State Pompeo as a suspect, in part due to his preference for the word "malign".

David Robinson's analysis

Michael Kearney also used tweets as the comparison text, but limited the suspect pool to cabinet members only. He compared the text of those tweets to the NYT op-ed on using 107 numeric features, such as use of capitalizaion, punctuation, whitespace, and sentence length and structure. On that basis, again ruling out the President and VP, the closest match is Secretary Pompeo.

Michael Kearney's analysis

Kanishka Misra also used tweets from cabinet members, but used the Burows' Delta metric to compare their text to that of the op-ed; those tweets with similar absolute word frequencies to the op-ed would be "closest" by this measure. Secretary of Transportation Elaine Chao is the closest match using this method.

Kanishka Misra's analysis

The authors take great pains to point out the limitations of these analyses: not all possible authors are active on Twitter; some of those likely don't write all their own tweets; Twitter is a medium that uses very different language than a newspaper editorial; and it's even possible the author was deliberately concealing their authorship through their language choices. But even so, the use of data has highlighted some unusual turns of phrase in the original op-ed that might have otherwise gone unobserved.

In each case the analysis was performed using the R language, and the authors make use of a common set of R packages to assist with the analysis:

• rvest to scrape data from the NYT website,
• rtweet, to download the text of tweets from likely suspects,
• tidytext, to tokenize and otherwise prepare the texts for analysis, and
• widyr to reshape the data for distance calculations

The full R code behind the analysis is provided by each author, and if nothing else provides interesting examples of different methods for text similarity analysis. These articles are a great demonstration of the power of rapid, in-depth data science with R: the op-ed was only published two days ago!

 

Three Twitter Threads

I've been heads-down this week preparing for some upcoming talks, so not as much blogging as usual this week. But there have been some interesting conversations on Twitter this week that you may be interested to check out if you're not on the platform.

Steph Lock shares her go-to R packages for every stage of the data science process:

My #rstats #datascience goto 📦
IO: odbc readxl httr
EDA: DataExplorer
Prep: tidyverse
Sampling: rsample modelr
Feature Engineering: recipes
Modelling: glmnet h2o FFTrees
Evaluation: broom yardstick
Deployment: sqlrutils AzureML opencpu
Monitoring: flexdashboard
Docs: rmarkdown

— Steph Locke (@SteffLocke) April 28, 2018

Rachel Thomas and Jeremy Howard advocate thinking differently about AI development, and not falling into the trap of thinking "bigger is always better" when it comes to data (a sentiment I wholeheartedly agree with):

Innovation come from doings things differently, not doing things bigger. @jeremyphoward https://t.co/3TJYs8OCbr pic.twitter.com/I55a6gT1OF

— Rachel Thomas (@math_rachel) May 2, 2018

I wondered what was so different about Python compared to R when it comes to package management, and got some really thoughtful responses:

Serious question: I use R, not Python, and while there's the occasional version/package issue in #rstats it's rarely a big deal. But I hear about this from Python devs all the time. What's so different about Python that this is such a thing? https://t.co/g8ddQu2gpt

— David Smith (@revodavid) April 30, 2018

Twitter definitely has its bad side, but there's a lot of really interesting conversation on the platform as well. Click on each tweet to see the conversations these generated.

A few podcast recommendations

After avoiding the entire medium for years, I've been rather getting into listening to podcasts lately. As a worker-from-home I don't have a commute (the natural use case of podcasts, I guess), but I have been travelling a lot more recently and it's been great to listen to during long flights. It turns out there are a lot of great podcasts out there about R, data science, and AI, so if you're looking for something to listen to here's what's currently in rotation for me:

• Not So Standard Deviations. Roger Peng and Hilary Parker chat about data science, real-world data analysis, and building ancient versions of R. 
• DataFramed. DataCamp's offical podcast features interviews with prominent members of the R and Data Science communities.
• The R Podcast. Practical advice on using R, and interviews with R developers.
• This Week in Machine Learning and AI. Interviews with AI and machine learning practitioners, with a good mix of technical and application-oriented topics.
• The Microsoft Research Podcast. Frequently features researchers working on artificial intelligence and novel data analysis methodology.
• And, for general interest / entertainment: Bombshell, Marketplace and (our namesake, but no relation) Revolutions.

What other podcasts are you enjoying right now? Let us know your recommendations in the comments.

A chart of Bechdel Test scores

A movie is said to satisfy the Bechdel Test if it satisfies the following three criteria:

1. The movie has at least two named female characters
2. ... who have a conversation with each other
3. ... about something other than a man

The website BechdelTest.com scores movies accordingly, granting one point for each of the criteria above for a maximum of three points. The recent Wonder Woman movie scores the full three points (Diana and her mother discuss war, for example), while Dunkirk, which features no named female characters, gets zero. (It was still a great film, however.)

The website also offers an API, which enabled data and analytics director Austin Wehrwein to create this time series chart of Bechdel scores for movies listed on BechdelTest.com:

This chart only includes ratings for that subset of movies listed on Bechdeltest.com, so it's not clear whether this is representative of movies as a whole. Austin suggests combining these data with the broader data from IDMB, so maybe someone wants to give that a try. Austin's R code for generating the above chart and several others is available at the link below, so click through for the full analysis.

Austin Wehrwein: A quick look at Bechdel test data (& an awtools update) (via Mara Averick)

A case study in messy data analysis: the Australian same-sex marriage survey

Last month the Australian people signaled their approval of legalizing same-sex marriage by a 62%:38% margin in a national survey. (On a personal note, I was elated and relieved by the result: my husband and I have discussed eventually retiring to Australia, and with this decision our marriage would be recognized there.) While fears of a surprise Brexit-like electoral backlash proved unfounded, researchers including R user Miles McBain explored the results for correlations to demographic variables. This process wasn't as simple as it might have been though: the Australian Bureau of Statistics released the results as a pair of Excel files that violate just about every good practice for sharing data in spreadsheets:

Miles shares the R code he used to extract useful data from this spreadsheet as a blog post that makes a great case study in dealing with messy data using R. The post demonstrates how he used the read_excel function (from readxl package) to extract specific sub-tables from the spreadsheet by specifying row and column ranges, and then use the dplyr package to clean up and merge the data. If you want to explore the data yourself, you can find the R code and the source data in this Github repository.

In a follow-up post, Miles combines the same-sex marriage survey data with Australian Census data to explore various demographic relationships. Unlike the US Census data (which is easily accessible in R thanks to the tidycensus package), there's no interface package for Australian Census data. (Selected tables are available in the Census2016 package, however.) Instead, Miles demonstrates how to use R to download and extract data from the the "Census DataPacks" (CSV data files and Excel data dictionaries) provided by the Australian Bureau of Statistics.  Yet more data wrangling allows Miles to create summary charts of the responses, such as this chart of proportion voting No by percent of the district population declaring a religious affiliation, broken down by state. As you may expect, those districts with more religious populations voted No at greater rates.

Both of these post provide great examples of working with government data, which is often provided in inconvenient formats with messy structures. Follow the links below for step-by-step guides, including the R code used to extract the data, structure it for analysis, and create useful charts.

Medium (Miles McBain): Tidying the Australian Same Sex Marriage Postal Survey Data with R; Combining Australian Census data with the Same Sex Marriage Postal Survey in R

Good practices for sharing data in spreadsheets

Spreadsheets are powerful tools with many applications: collecting data, sharing data, visualizing data, analyzing data, reporting on data. Sometimes, the temptation to do all of these things in a single workbook is irresistible. But if your goal is to provide data to others for analysis, then features that are useful for, say, reporting are downright detrimental to the task of data analysis. To make things easier on your downstream analysts, and to reduce the risk of inadvertent errors that can be caused by spreadsheets, Karl Broman and Kara Woo have published a paper Data organization in spreadsheets chock-full of useful advice. To reiterate from their introduction:

Spreadsheets are often used as a multipurpose tool for data entry, storage, analysis, and visualization. Most spreadsheet programs allow users to perform all of these tasks, however we believe that spreadsheets are best suited to data entry and storage, and that analysis and visualization should happen separately. Analyzing and visualizing data in a separate program, or at least in a separate copy of the data file, reduces the risk of contaminating or destroying the raw data in the spreadsheet.

The paper provides a wealth of helpful tips for making data in spreadsheets ready for analysis:

Basic Data Practices: use consistent data codes and variable names, for example.

Naming Practices: don't use spaces in column names or file names, and make those names meaningful.

Dealing with Dates: avoid a common gotcha by using the ISO 8601 standard for dates, as shown in the XKCD comic to the right.

Representing missing data: don't use empty cells for missing data; use a hyphen or a unique code like NA (but not a number like -9999).

Don't overload the cells. Don't try and pack more than one piece of information into a cell (and don't merge cells, either). In particular, don't encode useful information with color or font; add another column instead.

Follow tidy data principles. Make the data rectangular. If you need multiple rectangles, use multiple files or worksheets. If your data still doesn't fit into that format, a spreadsheet probably wasn't the best place for it in the first place.

Don't make calculations in data files. You're preparing this data for analysis, so avoid the temptation to include formulas to create new data. Just provide the raw data.

You can find details, examples, and lots more useful advice in the complete paper, which I encourage everyone to read at the link below.

The American Statistician: Data organization in spreadsheets (2017), Karl W. Broman and Kara H. Woo (via Sharon Machlis)

Recent updates to the Team Data Science Process

It's been over a year since we first introduced introduced the Team Data Science Process (TDSP). The data, technology and practices behind Data Science continue to evolve, and the TDSP has evolved in parallel. Over the past year, several new facets have been added, including:

• The IDEAR (Interactive Data Exploration, Analysis and Reporting) framework, an open source extension to R and Python designed to standardize the process of data exploration and reporting;
• Guidance for use of Spark 2.0, including an end-to-end Spark v2.0 walkthrough;
• Guidance for use of in-database Python with SQL Server, including an end-to-end in-database Python tutorial;
• Instantiation of TDSP projects and templates within the new Azure Machine Workbench.

For an example of applying the TDSP to effective data science projects, check out Buck Woody's 10-part series walking through every stage of a typical data science project. 

As the practice of data science changes, the TDSP continues to evolve. The TDSP is an open project hosted on Github, and your contributions are welcome.

Cortana Intelligence and Machine Learning Blog: The Microsoft Team Data Science Process (TDSP) – Recent Updates

Two upcoming webinars

Two new Microsoft webinars are taking place over the next week that may be of interest:

AI Development in Azure using Data Science Virtual Machines

The Azure Data Science Virtual Machine (DSVM) provides a comprehensive development and production environment to Data Scientists and AI-savvy developers. DSVMs are specialized virtual machine images that have been curated, configured, tested and heavily used by Microsoft engineers and data scientists. DSVM is an integral part of the Microsoft AI Platform and is available for customers to use through the Microsoft Azure cloud. In this session, we will first introduce DSVM, familiarize attendees with the product, including our newest offering, namely Deep Learning Virtual Machines (DLVMs). That will be followed by technical deep-dives into samples of end-to-end AI development and deployment scenarios that involve deep learning. We will also cover scenarios involving cloud based scale-out and parallelization.

This webinar runs from 10-11AM Pacific Time on Thursday October 26, and is presented by Gopi Kumar, Principal Program Manager, Paul Shealy, Senior Software Engineer, and Barnam Bora, Program Manager, at Microsoft. Register here.

Document Collection Analysis

With the extremely large volumes of data, especially unstructured text data, that are being collected every day, a huge challenge facing customers is the need for tools and techniques to organize, search, and understand this vast quantity of text. This webinar demonstrates an efficient and automated end-to-end workflow around analyzing large document collections and serving your downstream NLP tasks. We’ll demonstrate how to summarize and analyze a large collection of documents, including techniques such as phrase learning, topic modeling, and topic model analysis using the Azure Machine Learning Workbench.

This webinar runs from 10-11AM Pacific Time on Tuesday October 31, and will be presented by Ke Huang, Data Scientist at Microsoft. Register here.

Announcing dplyrXdf 1.0

I’m delighted to announce the release of version 1.0.0 of the dplyrXdf package. dplyrXdf began as a simple (relatively speaking) backend to dplyr for Microsoft Machine Learning Server/Microsoft R Server’s Xdf file format, but has now become a broader suite of tools to ease working with Xdf files.

This update to dplyrXdf brings the following new features:

• Support for the new tidyeval framework that powers the current release of dplyr
• Support for Spark and Hadoop clusters, including integration with the sparklyr package to process Hive tables in Spark
• Integration with dplyr to process SQL Server tables in-database
• Simplified handling of parallel processing for grouped data
• Several utility functions for Xdf and file management
• Workarounds for various glitches and unexpected behaviour in MRS and dplyr

Spark, Hadoop and HDFS

New in version 1.0.0 of dplyrXdf is support for Xdf files and datasets stored in HDFS in a Hadoop or Spark cluster. Most verbs and pipelines behave the same way, whether the computations are taking place in your R session itself, or in-cluster (except that they should be much more scalable in the latter case). Similarly, dplyrXdf can handle both the scenarios where your R session is taking place on the cluster edge node, or on a remote client.

For example, here is some sample code where we extract a table from Hive, then create a pipeline to process it in the cluster:

rxSparkConnect()
sampleHiv <- RxHiveData(table="hivesampletable")

# this will create the composite Xdf 'samplehivetable'
sampleXdf <- as_xdf(sampleHiv)

sampleXdf %>%
    filter(deviceplatform == "Android") %>%
    group_by(devicemake) %>%
    summarise(n=n()) %>%
    arrange(desc(n)) %>%
    head()
#>     devicemake     n
#> 1      Samsung 16244
#> 2           LG  7950
#> 3          HTC  2242
#> 4      Unknown  2133
#> 5     Motorola  1524

If you are logged into the edge node, dplyrXdf also has the ability to call sparklyr to process Hive tables in Spark. This can be more efficient than converting the data to Xdf format, since less I/O is involved. To run the above pipeline with sparklyr, we simply omit the step of creating an Xdf file:

sampleHiv %>%
    filter(deviceplatform == "Android") %>%
    group_by(devicemake) %>%
    summarise(n=n()) %>%
    arrange(desc(n))
#> # Source:     lazy query [?? x 2]
#> # Database:   spark_connection
#> # Ordered by: desc(n)
#>     devicemake     n
#>          <chr> <dbl>
#> 1      Samsung 16244
#> 2           LG  7950
#> 3          HTC  2242
#> 4      Unknown  2133
#> 5     Motorola  1524
#> # ... with more rows

For more information about Spark and Hadoop support, see the HDFS vignette and the Sparklyr website.

SQL database support

One of the key strengths of dplyr is its ability to interoperate with SQL databases. Given a database table as input, dplyr can translate the verbs in a pipeline into a SQL query which is then execute in the database. For large tables, this can often be much more efficient than importing the data and running them locally. dplyrXdf can take advantage of this with an MRS data source that is a table in a SQL database, including (but not limited to) Microsoft SQL Server: rather than importing the data to Xdf, the data source is converted to a dplyr tbl and passed to the database for processing.

# copy the flights dataset to SQL Server
flightsSql <- RxSqlServerData("flights", connectionString=connStr)
flightsHd <- copy_to(flightsSql, nycflights13::flights)

# this is run inside SQL Server by dplyr
flightsQry <- flightsSql %>%
    filter(month > 6) %>%
    group_by(carrier) %>%
    summarise(avg_delay=mean(arr_delay))

flightsQry
#> # Source:   lazy query [?? x 2]
#> # Database: Microsoft SQL Server
#> #   13.00.4202[dbo@DESKTOP-TBHQGUH/sqlDemoLocal]
#>   carrier avg_delay
#>   <chr>       <dbl>
#> 1 "9E"        5.37 
#> 2 AA        - 0.743
#> 3 AS        -16.9  
#> 4 B6          8.53 
#> 5 DL          1.55 
#> # ... with more rows

For more information about working with SQL databases including SQL Server, see the dplyrXdf SQL vignette and the dplyr database vignette.

Parallel processing and grouped data

Even without a Hadoop or Spark cluster, dplyrXdf makes it easy to parallelise the handling of groups. To do this, it takes advantage of Microsoft R Server's distributed compute contexts: for example, if you set the compute context to "localpar", grouped transformations will be done in parallel on a local cluster of R processes. The cluster will be shut down automatically when the transformation is complete.

More broadly, you can create a custom backend and tell dplyrXdf to use it by setting the compute context to "dopar". This allows you a great deal of flexibility and scalability, for example by creating a cluster of multiple machines (as opposed to multiple cores on a single machine). Even if you do not have the physical machines, packages like AzureDSVM and doAzureParallel allow you to deploy clusters of VMs in the cloud, and then shut them down again. For more information, see the “Parallel processing of grouped data” section of the Using dplyrXdf vignette.

Data and file management

New in dplyrXdf 1.0.0 is a suite of functions to simplify managing Xdf files and data sources:

• HDFS file management: upload and download files with hdfs_file_upload and hdfs_file_download; copy/move/delete files with hdfs_file_copy, hdfs_file_move, hdfs_file_remove; list files with hdfs_dir; and more
• Xdf data management: upload and download datasets with copy_to, collect and compute; import/convert to Xdf with as_xdf; copy/move/delete Xdf data sources with copy_xdf, move_xdf and delete_xdf; and more
• Other utilities: run a block of code in the local compute context with local_exec; convert an Xdf file to a data frame with as.data.frame; extract columns from an Xdf file with methods for [, [[ and pull

Obtaining dplyr and dplyrXdf

dplyrXdf 1.0.0 is available from GitHub. It requires Microsoft R Server 8.0 or higher, and dplyr 0.7 or higher. Note that dplyr 0.7 will not be in the MRAN snapshot that is your default repo, unless you are using the recently-released MRS 9.2; you can install it, and its dependencies, from CRAN. If you want to use the SQL Server and sparklyr integration facility, you should install the odbc, dbplyr and sparklyr packages as well.

install_packages(c("dplyr", "dbplyr", "odbc", "sparklyr"),
                 repos="https://cloud.r-project.org")
devtools::install_github("RevolutionAnalytics/dplyrXdf")

If you run into any bugs, or if you have any feedback, you can email me or log an issue at the Github repo.