Revolutions

Text featurization with the Microsoft ML package

Last week I wrote about how you can use the MicrosoftML package in Microsoft R to featurize images: reduce an image to a vector of 4096 numbers that quantify the essential characteristics of the image, according to an AI vision model. You can perform a similar featurization process with text as well, but in this case you have a lot more control of the features used to represent the text.

Tsuyoshi Matsuzaki demonstrates the process in a post at the MSDN Blog. The post explores the Multi-Domain Sentiment Dataset, a collection of product reviews from Amazon.com. The dataset includes reviews from 975,194 products on Amazon.com from a variety of domains, and for each product there is a text review and a star rating of 1, 2, 4, or 5. (There are no 3-star rated reviews in the data set.) Here's one example, selected at random:

What a useful reference! I bought this book hoping to brush up on my French after a few years of absence, and found it to be indispensable. It's great for quickly looking up grammatical rules and structures as well as vocabulary-building using the helpful vocabulary lists throughout the book. My personal favorite feature of this text is Part V, Idiomatic Usage. This section contains extensive lists of idioms, grouped by their root nouns or verbs. Memorizing one or two of these a day will do wonders for your confidence in French. This book is highly recommended either as a standalone text, or, preferably, as a supplement to a more traditional textbook. In either case, it will serve you well in your continuing education in the French language.

The review contains many positive terms ("useful", "indespensable", "highly recommended"), and in fact is associated with a 5-star rating for this book. The goal of the blog post was to find the terms most associated with positive (or negative) reviews. One way to do this is to use the featurizeText function in thje Microsoft ML package included with Microsoft R Client and Microsoft R Server. Among other things, this function can be used to extract ngrams (sequences of one, two, or more words) from arbitrary text. In this example, we extract all of the one and two-word sequences represented at least 500 times in the reviews. Then, to assess which have the most impact on ratings, we use their presence or absence as predictors in a linear model:

transformRule = list(
  featurizeText(
    vars = c(Features = "REVIEW_TEXT"),
    # ngramLength=2: include not only "Azure", "AD", but also "Azure AD"
    # skipLength=1 : "computer" and "compuuter" is the same
    wordFeatureExtractor = ngramCount(
      weighting = "tfidf",
      ngramLength = 2,
      skipLength = 1),
    language = "English"
  ),
  selectFeatures(
    vars = c("Features"),
    mode = minCount(500)
  )
)

# train using transforms !
model <- rxFastLinear(
  RATING ~ Features,
  data = train,
  mlTransforms = transformRule,
  type = "regression" # not binary (numeric regression)
)

We can then look at the coefficients associated with these features (presence of n-grams) to assess their impact on the overall rating. By this standard, the top 10 words or word-pairs contributing to a negative rating are:

boring       -7.647399
waste        -7.537471
not          -6.355953
nothing      -6.149342
money        -5.386262
bad          -5.377981
no           -5.210301
worst        -5.051558
poorly       -4.962763
disappointed -4.890280

Similarly, the top 10 words or word-pairs associated with a positive rating are:

will      3.073104
the|best  3.265797
love      3.290348
life      3.562267
wonderful 3.652950
,|and     3.762862
you       3.889580
excellent 3.902497
my        4.454115
great     4.552569

Another option is simply to look at the sentiment score for each review, which can be extracted using the getSentiment function. 

sentimentScores <- rxFeaturize(data=data, 
                    mlTransforms = getSentiment(vars = 
                                     list(SentimentScore = "REVIEW_TEXT")))

As we expect, a negative seniment (in the 0-0.5 range) is associated with 1- and 2-star reviews, while a positive sentiment (0.5-1.0) is associated with the 4- and 5-star reviews.

You can find more details on this analysis, including the Microsoft R code, at the link below.

Microsoft Technologies Blog for Enterprise Developers: Analyze your text in R (MicrosoftML)

Probable more likely than probably

What kind of probability are people talking about when they say something is "highly likely" or has "almost no chance"? The chart below, created by Reddit user zonination, visualizes the responses of 46 other Reddit users to "What probability would you assign to the phase: <phrase>" for various statements of probability. Each set of responses has been converted to a kernel destiny estimate and presented as a joyplot using R. 

Somewhat surprisingly, the results from the Redditors hew quite closely to a similar study of 23 NATO intelligence officers in 2007 the 60's or 70's (the timeline of the study isn't clear — see comment below) . In that study, the officers — who were accustomed to reading intelligence reports with assertions of likelihood — were giving a similar task with the same descriptions of probability. The results, here presented as a dotplot, are quite similar.

For details on the analysis of the Redditors, including the data and R code behind the joyplot chart, check out the Github repository linked below.

Github (zonination): Perceptions of Probability and Numbers

3-D animations with R

R is often used to visualize and animate 2-dimensional data. (Here are just a few examples.)  But did you know you can create 3-dimensional animations as well? 

As Thomas Lins Pedersen explains in a recent blog post, the trick is in using the persp function to translate points in 3-D space into a 2-D projection. This function is normally used to render a 3-D scatterplot or wireframe plot, but if you instead capture its output value, it returns a transformation matrix. You can then use the trans3d function to with this matrix to transform points in 3-D space. Thomas demonstrates how you can pass the transformed 2-D coordinates to plot a 3-D cube, and even animate it from two slightly different perspectives to create a 3-D stereo pair:

Rendering 3-D images isn't just for fun: there's plenty of 3-D data to analyze and visualize, too. Giora Simchoni used R to visualize data from the Carnegie-Mellon Graphics Lab Motion Capture Database. This data repository provides the output of human figures in motion-capture suits performing actions like walking, jumping, and even dancing. Since the motion-capture suits include multiple sensors measured over a time-period, the data structures are quite complex. To make things simpler, Giora created the mocap package (available on Github) to read these motion data files and generate 3-D animations to visualize them. For example, here's the output from two people performing the Charleston together:

You can find complete details behind both animations, including the associated R code, at the links below.

Data Imaginist: I made a 3D movie with ggplot2 once - here's how I did it
Giora Simchoni: Lambada! (The mocap Package)

Packages to simplify mapping in R

Computerworld's Sharon Machlis has published a very useful tutorial on creating geographic data maps with R. (The tutorial was actually published back in March, but I only came across it recently.) While it's been possible to create maps in R for a long time, some recent packages and data APIs have made the process much simpler. The tutorial is based on the following R packages:

• sf, a package that provides simple data structures for objects representing spatial geometries including map features like borders and rivers
• tmap and tmaptools, packages for creating static and interactive "thematic maps" like choropleths and spatial point maps using a ggplot2-like syntax
• tigris, a package that provides shapefiles you can use to map the USA and its states, counties, and census tracts (very useful for visualizing US census data, which you can easily obtain with the tidycensus package)
• rio, a package to streamline the import of flat files from third-party data sources like the U.S. Bureau of Labor Statistics

With those packages, as you'll learn in the tutorial linked below, you can easily create attractive maps to visualize geographic data, and even make those graphics interactive with scrolling, zooming, and data pop-ups thanks to the capabilities of the leaflet package. All of these packages are now available on CRAN, so there's no need to install from Github as the tutorial then suggested, unless you want access to even newer in-development capabilities.

Computerworld: Mapping in R just got a whole lot easier

Because it's Friday: Total Eclipse in the Park

Here in the US, it's been Eclipse-mania for a the past week or so, culminating in the totality event across much of the country on Monday. My husband and I were lucky enough to catch total eclipse on a glorious day — thanks to the City of St Louis for putting together a great watching event in Jefferson Barracks Park, and for the weather gods for keeping the sky free of clouds. Here's a picture of us just a few minutes before totality — you can even see images of the soon-to-be-eclipsed sun on the ground, thanks to the camera obscura image through the small gaps in the trees.

Seeing the totality event with my own eyes through was truly awe-inspiring: if you ever have chance to see it yourself, do it: photos don't really do it justice. Nonetheless, Derek Muller got some pretty good shots in Oregon:

With some careful planning, Destin Sandlin also got some shots of the ISS transiting the eclipse in Wyoming:

That's all from us here at the blog for this week. Have a great weekend, and we'll be back again on Monday. See you then!

Tips and tricks on using R to query data in Power BI

In Power BI, the dashboarding and reporting tool, you can use R to filter, transform, or restructure data via the Query Editor. For example, you could use the mice package to impute missing values, or use the tidytext package to assign sentiment scores to text inputs. As Imke Feldmann explains, there are lots of useful tricks you can accomplish using R in the query editor, including:

• Passing multiple Power BI tables into the R script
• Parameterizing R scripts
• Modifying file names and output data types for use in Power BI
• Generating multiple table outputs from an R script

For the details on these tips, follow the link below. You can also find tips on using R in the Query Editor here.

The Biccountant: Tips and Tricks for R scripts in the query editor in Power BI

Calculating a fuzzy kmeans membership matrix with R and Rcpp

by Błażej Moska, computer science student and data science intern 

Suppose that we have performed clustering K-means clustering in R and are satisfied with our results, but later we realize that it would also be useful to have a membership matrix. Of course it would be easier to repeat clustering using one of the fuzzy kmeans functions available in R (like fanny, for example), but since it is slightly different implementation the results could also be different and for some reasons we don’t want them to be changed. Knowing the equation we can construct this matrix on our own, after using the kmeans function. The equation is defined as follows (source: Wikipedia):

$$w_{ij} = \frac{1}{ \sum_ {k=1}^c ( \frac{ \| x_{i} - c_{j} \| }{ \| x_{i} - c_{k} \| }) ^{ \frac{2}{m-1} } } $$

\(w_{ij}\) denotes to what extent the \(i\)th object belongs to the \(j\)th cluster. So the total number of rows of this matrix equals number of observation and total number of columns equals number of variables included in clustering. \(m\) is a parameter, typically set to \(m=2\). \(w_{ij}\) values range between 0 and 1 so they are easy and convenient to compare. In this example I will use \(m = 2\) so the Euclidean distance will be calculated.

To make computations faster I also used the Rcpp package, then I compared speed of execution of function written in R with this written in C++.

In implementations for loops were used, although it is not a commonly used method in R (see this blog post for more information and alternatives), but in this case I find it more convenient.

Rcpp (C++ version)

#include <Rcpp.h>
#include <math.h>
using namespace Rcpp;

// [[Rcpp::export]]
NumericMatrix fuzzyClustering(NumericMatrix data, NumericMatrix centers, int m) {  
/* data is a matrix with observations(rows) and variables, 
   centers is a matrix with cluster centers coordinates, 
   m is a parameter of equation, c is a number of clusters
*/
  
  int c=centers.rows();
  int rows = data.rows();
  int cols = data.cols(); /*number of columns equals number of variables, the same as is in centers matrix*/
  double tempDist=0;        /*dist and tempDist are variables storing temporary euclidean distances */
  double dist=0;      
  double denominator=0;    //denominator of “main” equation
  
  NumericMatrix result(rows,c);    //declaration of matrix of results
  
  for(int i=0;i<rows;i++){
    for(int j=0;j<c;j++){
      for(int k=0;k<c;k++){
        for(int p=0;p<cols;p++){
          tempDist = tempDist+pow(centers(j,p)-data(i,p),2);      
         //in innermost loop an euclidean distance is calculated.
          dist = dist + pow(centers(k,p)-data(i,p),2);              
/*tempDist is nominator inside the sum operator in the equation, dist is the denominator inside the sum operator in the equation*/
        }
        tempDist = sqrt(tempDist);
        dist = sqrt(dist);
        denominator = denominator+pow((tempDist/dist),(2/(m-1)));
        tempDist = 0;
        dist = 0;
      }
      result(i,j) = 1/denominator;    
// nominator/denominator in the  main equation
      denominator = 0;                           
    }
  }
  return result;
}

We can save this in a file with .cpp extension. To compile it from R we can write:

sourceCpp("path_to_cpp_file")

If everything goes right our function fuzzyClustering will then be available from R.

R version

fuzzyClustering=function(data,centers,m){

  c <- nrow(centers)
  rows <- nrow(data)
  cols <- ncol(data)
  result <- matrix(0,nrow=rows,ncol=c)  #defining membership matrix
  denominator <- 0
  
  for(i in 1:rows){
    for(j in 1:c){
      tempDist <- sqrt(sum((centers[j,]-data[i,])^2))  #euclidean distance, nominator inside a sum operator 
      for(k in 1:c){
        Dist <- sqrt(sum((centers[k,]-data[i,])^2))    #euclidean distance, denominator inside a sum operator 
        denominator <- denominator +((tempDist/Dist)^(2/(m-1))) #denominator of an equation 
      }    
      result[i,j] <- 1/denominator    #inserting value into membership matrix
      denominator <- 0           
    }
  }
  return(result);
}

Result looks as follows. Columns are cluster numbers (in this case 10 clusters were created), rows are our objects (observations). Values were rounded to the third decimal place, so the sums of rows can be slightly different than 1:

          1     2     3     4     5     6     7     8     9    10  
 [1,] 0.063 0.038 0.304 0.116 0.098 0.039 0.025 0.104 0.025 0.188  
 [2,] 0.109 0.028 0.116 0.221 0.229 0.080 0.035 0.116 0.017 0.051  
 [3,] 0.067 0.037 0.348 0.173 0.104 0.066 0.031 0.095 0.018 0.062  
 [4,] 0.016 0.015 0.811 0.049 0.022 0.017 0.009 0.023 0.007 0.031  
 [5,] 0.063 0.048 0.328 0.169 0.083 0.126 0.041 0.079 0.018 0.045  
 [6,] 0.069 0.039 0.266 0.226 0.102 0.111 0.037 0.084 0.017 0.048  
 [7,] 0.045 0.039 0.569 0.083 0.060 0.046 0.025 0.071 0.015 0.046  
 [8,] 0.070 0.052 0.399 0.091 0.093 0.054 0.034 0.125 0.022 0.062  
 [9,] 0.095 0.037 0.198 0.192 0.157 0.088 0.038 0.121 0.019 0.055 
[10,] 0.072 0.024 0.132 0.375 0.148 0.059 0.025 0.081 0.015 0.067

Performance comparison

Shown below is the output of Sys.time for the C++ and R versions, running against a simulated matrix with 30000 observations, 3 variables and 10 clusters.
The hardware I used was a low-cost notebook Asus R556L with Intel Core i3-5010 2.1 GHz processor and 8 GB DDR3 1600 MHz RAM memory.

C++ version:

  user    system  elapsed
  0.32    0.00    0.33 

R version:

  user    system  elapsed
  15.75    0.02     15.94 

In this example, the function written in C++ executed about 50 times faster than the equivalent function written in pure R.

Recreating and updating Minard with ggplot2

Minard's chart depicting Napoleon's 1812 march on Russia is a classic of data visualization that has inspired many homages using different time-and-place data. If you'd like to recreate the original chart, or create one of your own, Andrew Heiss has created a tutorial on using the ggplot2 package to re-envision the chart in R:

The R script provided in the tutorial is driven by historical data on the location and size of Napoleon's armies during the 1812 campaign, but you could adapt the script to use new data as well. Andrew also shows how to combine the chart with a geographical or satellite map, which is how the cities appear in the version above (unlike in Minard's original). 

The data behind the Minard chart is available from Michael Friendly and you can find the R scripts at this Github repository. For the complete tutorial, follow the link below.

Andrew Heiss: Exploring Minard’s 1812 plot with ggplot2 (via Jenny Bryan)

 

Gender roles in film direction, analyzed with R

What do women do in films? If you analyze the stage directions in film scripts — as Julia Silge, Russell Goldenberg and Amber Thomas have done for this visual essay for ThePudding — it seems that women (but not men) are written to snuggle, giggle and squeal, while men (but not women) shoot, gallop and strap things to other things.  

This is all based on an analysis of almost 2,000 film scripts mostly from 1990 and after. The words come from pairs of words beginning with "he" and "she" in the stage directions (but not the dialogue) in the screenplays — directions like "she snuggles up to him, strokes his back" and "he straps on a holster under his sealskin cloak". The essay also includes an analysis of words by the writer and character's gender, and includes lots of lovely interactive elements (including the ability to see examples of the stage directions).

The analysis, including the chart above, was was created using the R language, and the R code is available on GitHub. The screenplay analysis makes use on the tidytext package, which simplifies the process of handling the text-based data (the screenplays), extracting the stage directions, and tabulating the word pairs.

You can find the complete essay linked below, and it's well worth checking out to experience the interactive elements.

ThePudding: She Giggles, He Gallops

Highlights of the Data Science Track at Microsoft Ignite

I will be at the AI Summit in San Francisco next month, which means I can't make it to Ignite in Orlando this year. Which is a bit of a shame, because there's a fantastic Data Science track at Ignite. There are 25 sessions on offer, with presentations from my Microsoft colleagues on Microsoft R, Cognitive Toolkit, Bot Framework, the Team Data Science Process, and much more. There will also be applications of these technologies and systems presented by the Portland Trail Blazers, Enbridge, and Jack Henry and Associates.

• Big data machine learning with Microsoft R Server
• Patterns, Architecture, & Best Practices: Scaling Machine Learning Algorithms with Azure HDInsight
• Cert Exam Prep: Exam 70-773: Analyzing Big Data with Microsoft R
• How to build machine learning applications using R and Python in SQL Server 2017
• How to modernize analytics by migrating from SAS to the Microsoft platform

You can access these talks plus keynote presentations by Satya Nadella and Joseph Sirosh with the Data Science, Machine Learning and AI Pass. For the complete list of talks included, follow the link below.

Microsoft Ignite: Data Science, Machine Learning and AI Pass Session Catalog