Revolutions

R-based data maps in PowerBI

One of benefit of integrating R with PowerBI is access to rich array of data visulizations not present in the standard PowerBI loadout. R is practically unlimited in the types of graphics it can create (although the amount of programming required can vary from a few lines using an existing R package, to large custom functions for truly bespoke graphics). Some of the visualizations you can create with R include population pyramids, small multiples, annotated time series, calendar heat maps, rank plots and even emoji charts. But perhaps one of the biggest opportunities is the ability to plot data on a geographic surface with choropleths, map projections, topological maps and animated maps. 

If you'd like to learn how to use R maps with PowerBI, David Eldersveld from BlueGranite has put together a useful series of tutorials. In R Maps in Microsoft Power BI: Getting Started, Dave walks you through the steps of using R's maps and mapproj packages to create an interactive PowerBI dashboard to explore the (surprisingly numerous) airfields in the Great Lakes region, based on data provided by the FAA. 

In part 2 of the tutorial, Dave explores using R to create small multiples: repeated maps showing data varying by time or data subset (a great way of making comparisons). This example shows the airports by type of owner (public, private, Army or Air Force):

You can learn more about how to create charts like these in PowerBI by following the link below. You can also download the PowerBI PBIX files and the modified dataset to try them out yourself.

 

DataVeld: R Maps in Microsoft Power BI: Getting Started

Because it's Friday: It's all about that accent

If you thought there was only one British accent, well, I'm guessing you've never tried to have a conversation with a Scotsman in bar. Dialect coach Andrew Jack (who trains foreign actors playing British characters in many Hollywood movies) demonstrates the range of accents in this clip:

 

That's an impressive feat of switching from one accent to another! And if you're interested in being able tp play a foreign accent yourself, actress Amy Walker has a collection of tips:

 

That's all for this week. We look forward to seeing you back next soon: the blog will resume on Monday. Have a great weekend!

Japan's ageing population, animated with R

The US Census makes a number of its databases available to developers via the Census API. One of those databases is the International Data Base, in which the Census department provides historic demographic breakdowns (population by age and sex) for many countries, along with projections through 2050. Kyle Walker created the R package idbr (currently only available on GitHub) to make it easy to download these datasets using R, and used it to create the animation below showing Japan's demographic change since 1990. As you can see, the Census Bureau predicts that this ageing of the population will only intensify over time. Note the ever-declining population of babies being added at the bottom of the chart. In fact, Japan's population declined for the first time just this year.

Kyle used ggplot's geom_bar plot to visulize the population pyramid, and rendered it using the Economist theme from the ggthemes package. To run the code, you'll need to request a Census API key (you just need to provide your company and email address, and you'll have one in minutes), and install ImageMagick to support the animation package. You can see Kyle's original code below (I adapted it slightly to show the historical demographics since 1990 and to resize the animation for this blog post.)

For more on Kyle's idbr package, check out the GitHub repository linked below.

Github (walkerke): idbr, An R interface to the US Census Bureau International Data Base API (via the author)

Generating and Visualizing Multivariate Data with R

By Joseph Rickert

The ability to generate synthetic data with a specified correlation structure is essential to modeling work. As you might expect, R’s toolbox of packages and functions for generating and visualizing data from multivariate distributions is impressive. The basic function for generating multivariate normal data is mvrnorm() from the MASS package included in base R, although the mvtnorm package also provides functions for simulating both multivariate normal and t distributions. (For tutorial on how to use R to simulate from multivariate normal distributions from first principles using some linear algebra and the Cholesky decomposition see the astrostatistics tutorial on Multivariate Computations.)

The following block of code generates 5,000 draws from a bivariate normal distribution with mean (0,0) and covariance matrix Sigma printed in code. The function kde2d(), also from the Mass package generates a two-dimensional kernel density estimation of the distribution's probability density function. 

# SIMULATING MULTIVARIATE DATA
# https://stat.ethz.ch/pipermail/r-help/2003-September/038314.html
# lets first simulate a bivariate normal sample
library(MASS)
# Simulate bivariate normal data
mu <- c(0,0)                         # Mean
Sigma <- matrix(c(1, .5, .5, 1), 2)  # Covariance matrix
# > Sigma
# [,1] [,2]
# [1,]  1.0  0.1
# [2,]  0.1  1.0
 
# Generate sample from N(mu, Sigma)
bivn <- mvrnorm(5000, mu = mu, Sigma = Sigma )  # from Mass package
head(bivn)                                      
# Calculate kernel density estimate
bivn.kde <- kde2d(bivn[,1], bivn[,2], n = 50)   # from MASS package

R offers several ways of visualizing the distribution. These next two lines of code overlay a contour plot on a "heat Map" that maps the density of points to a gradient of colors.

# Contour plot overlayed on heat map image of results
image(bivn.kde)       # from base graphics package
contour(bivn.kde, add = TRUE)     # from base graphics package

This plots the irregular contours of the simulated data. The code below which uses the ellipse() function from the ellipse package generates the classical bivariate normal distribution plot that graces many a textbook.

# Classic Bivariate Normal Diagram
library(ellipse)
rho <- cor(bivn)
y_on_x <- lm(bivn[,2] ~ bivn[,1])    # Regressiion Y ~ X
x_on_y <- lm(bivn[,1] ~ bivn[,2])    # Regression X ~ Y
plot_legend <- c("99% CI green", "95% CI red","90% CI blue",
                 "Y on X black", "X on Y brown")
 
plot(bivn, xlab = "X", ylab = "Y",
     col = "dark blue",
     main = "Bivariate Normal with Confidence Intervals")
lines(ellipse(rho), col="red")       # ellipse() from ellipse package
lines(ellipse(rho, level = .99), col="green")
lines(ellipse(rho, level = .90), col="blue")
abline(y_on_x)
abline(x_on_y, col="brown")
legend(3,1,legend=plot_legend,cex = .5, bty = "n")

The next bit of code generates a couple of three dimensional surface plots. The second of which is an rgl plot that you will be able to rotate and view from different perspectives on your screen.

# Three dimensional surface
# Basic perspective plot
persp(bivn.kde, phi = 45, theta = 30, shade = .1, border = NA) # from base graphics package
 
# RGL interactive plot
library(rgl)
col2 <- heat.colors(length(bivn.kde$z))[rank(bivn.kde$z)]
persp3d(x=bivn.kde, col = col2)

Next, we have some code to unpack the grid coordinates produced by the kernel density estimator and get x y, and z values to plot the surface using the new scatterplot3js() function from the htmlwidgets, javascript threejs package. This visualization does not render the surface with the same level of detail as the rgl plot. Nevertheless, it does show some of the salient features of the pdf and has the distinct advantage of being easily embedded in web pages. I expect that html widget plots will keep getting better and easier to use.

# threejs Javascript plot
library(threejs)
# Unpack data from kde grid format
x <- bivn.kde$x; y <- bivn.kde$y; z <- bivn.kde$z
# Construct x,y,z coordinates
xx <- rep(x,times=length(y))
yy <- rep(y,each=length(x))
zz <- z; dim(zz) <- NULL
# Set up color range
ra <- ceiling(16 * zz/max(zz))
col <- rainbow(16, 2/3)
# 3D interactive scatter plot
scatterplot3js(x=xx,y=yy,z=zz,size=0.4,color = col[ra],bg="black")

<

The code that follows uses the rtmvt() function from the tmvtnorm package to generate bivariate t distribution. The rgl plot renders the surface kernel density estimate of the surface in impressive detail.

# Draw from multi-t distribution without truncation
library (tmvtnorm)
Sigma <- matrix(c(1, .1, .1, 1), 2)  # Covariance matrix
X1 <- rtmvt(n=1000, mean=rep(0, 2), sigma = Sigma, df=2) # from tmvtnorm package
 
t.kde <- kde2d(X1[,1], X1[,2], n = 50)   # from MASS package
col2 <- heat.colors(length(bivn.kde$z))[rank(bivn.kde$z)]
persp3d(x=t.kde, col = col2)

The real value of the multivariate distribution functions from the data science perspective is to simulate data sets with many more than two variables. The functions we have been considering are up to the task, but there are some technical considerations and, of course, we don't have the same options for visualization. The following code snippet generates 10 variables from a multivariate normal distribution with a specified covariance matrix. Note that I've used the genPositiveDefmat() function from the clusterGeneration package to generate the covariance matrix. This is because mvrnorm() will throw an error, as theory says it should, if the covariance matrix is not positive definite, and guessing a combination of matrix elements to make a high dimensional matrix positive definite would require quite a bit of luck along with some serious computation time.

After generating the matrix, I use the corrplot() function from the corrplot package to produce an attractive pairwise correlation plot that is coded both by shape and color. corrplot() scales pretty well with the number of variables and will give a decent chart with 40 to 50 variables. (Note that now ggcorrplot will do this for ggplot2 plots.)  Other plotting options would be to generate pairwise scatter plots and R offers many alternatives for these.

#Higher Dimensional Distributions
library(corrplot)
library(clusterGeneration)
mu <- rep(0,10)
pdMat <- genPositiveDefMat(10,lambdaLow=10)
Sigma <- pdMat$Sigma
dim(Sigma)
mvn <- mvrnorm(5000, mu = mu, Sigma = Sigma )
 
corrplot(cor(mvn), 
         method="ellipse",
         tl.pos="n",
         title="Matrix Correlations")
 

Finally, what about going beyond the multivariate normal and t distributions? R does have a few functions like rlnorm() from the compositions package which generates random variates from the multivariate lognormal distribution that are as easy to use as mvrorm(), but you will have to hunt for them. I think a more fruitful approach if you are serious about probability distributions is to get familiar with the copula package.

Explore New Zealand's Tourist Industry with R and Shiny

New Zealand's Ministry of Business, Innovation and Employment has just released the New Zealand Tourism Dashboard, an interactive application which allows NZ residents (and curious onlookers everywhere) to explore the economic impact of tourism in the far-flung nation. The dashboard is implemented using Shiny, and all of the graphics and analyses were created using the R language 

Each tab at the top of the window leads to a different Shiny application (or a collection of applications) to explore data related to New Zealand tourism. You can explore the types of attractions visited (museums, nature, or Maori activities?), which nationalities visit the most often (Australia, closely followed by China), the types of accommodations preferred by different nationalities (residents of Oceania and Africa tend to stay with friends) and what tourists spend money on (the Chinese spend more of their money on accommodations), and explore the performance of regions within New Zealand (here, Wellington):

You can explore all of the data and reports at the link below. By the way, it's only appropriate that New Zealand use the R language for an application like this: R was created in New Zealand!

New Zealand Ministry of Business, Innovation and Employment: The New Zealand Tourism Dashboard (via Peter Ellis)

Analysis of Xbox Usage Trends with R: Time Series Decompositions

by Daniel Moore
Director of Applied Statistics Engineering, Console Development
Microsoft

In Xbox Hardware, we are interested in the various ways that our hardware is used, and we are especially interested in how that usage changes over time. We employ several several time series analysis techniques that are helpful in getting a holistic view of usage of the Xbox console. When it comes to the actual data that we look at – it could be an individual game or app usage or it could be specific features of the console. For all of these, there are a few parts of the time series that we are interested in. The first is the trend of usage over time. Many games are very popular when they are first released and then lose popularity as they age. Some games remain steady in their popularity. The consoles themselves may show increased usage on holiday periods. All of these would be reflected in the overall trend of the data. We are also interested in the weekly cycle of usage. As you can imagine, use of a gaming console goes up on the weekend and down during the week. This is probably even more so among children than adults – as they have more time (and permission!) to play on non-school days than they do on school days.

We use R extensively to perform a time series analysis. In this post we’ll explore the initial analysis and decomposition of the time series into its component parts. Though some packages offer more complete time series analysis options, the base version of R has some good built in features for this initial analysis. The data for this example is the usage of a single game over more than a year of usage on the Xbox One. This is done with the following code below:

data<-read.csv("dataset.csv") # loads the data set in
plot(data, type="l") # plot the data first to get a look at it. Here, just a line plot showing weekly periodicity over a trend
tsdata<-ts(data, start=1, freq=7) # here we define a time series out of the data, with the first observation set at 1 and a weekly frequency..
decomposeddata<-stl(tsdata, s.window=7)
plot(decomposeddata) # this shows four panes of line charts – the data, the sinusoidal seasonal fluctuations, the trend, and the remainder

The object “decomposeddata” is of class “stl” with several components useful for time series analysis. This uses loess to decompose the time series and many smoothing and other settings are available, depending on specific needs and the analysis being performed.

To highlight something that can be illuminated with this, look at the seasonal component of the chart below.

You’ll notice a period (highlighted) where the seasonal fluctuations dropped significantly. This is the summer time when school is out and weekdays/weekends blend together for kids. It’s always fun to find those artifacts in the data!

Latest Redmonk and Tiobe language rankings for R

Analyst firm RedMonk have updated their (near-)biannual Programming Lanuage rankings as of January 2016, and the R language ranks at #13, unchanged since the last ranking in June 2015. Redmonk's rankings are based on number of projects in GitHub and number of questions tagged in StackOverflow, and the most recent data is visualized (using R's ggplot2 package) below:

In this latest RedMonk ranking, Python is likewise steady at #4, and Matlab falls one place to #18. See the entire list and a new visualization of ranking changes over time here: RedMonk Programming Language Rankings: January 2016.

Meanwhile, the Tiobe Index for February 2016, which ranks languages according to search engine results, places R at #17 (up 1 places from a year ago). In the same Tiobe rankings, Python places at #5, Matlab at #18, and SAS at #12.

For more R rankings from these and other sources, check out the popularity section of the blog.

Because it's Friday: Networking in Destiny

I play a fair bit of Destiny, a space-themed video game. Actually, a lot. (It's an amount my husband calls "too much".) I enjoy the game not just for its great story and space-age shooting gameplay, but also for the social interaction. It's a massively-multiplayer game: you're always online, and you'll continually run into other players in the game world. Some parts of the game even require teams of 3 or 6 people to complete. As a result, I often play with real-world friends, and my Xbox friends list is now full of people I've met online playing the game. This social aspect is quite a unique feature of Destiny, and has led to a large yet close-knit community. It's even produced things like this in-game tribute to a player who died of cystic fybrosis last year.

 

Only a truly social game where you can wander through the virtual world at will, join up with friends, and meet others along the way could sustain this kind of community spirit, and as it turns out a lot of work went on behind the scenes to make this a possibility. I just discovered this presentation by an employee of Destiny developer Bungie, describing the underlying console, networking and cloud infrastructure that makes a game like this possible. The concept is built around "bubbles", a simulation of a small part of the world, hosted by one of the players that inhabits it. 

The magic comes in how Destiny handles creating and deactivating bubbles as players move around the world, and what happens when a player suddenly quits, meaning that the bubble hosting has to be moved to another player while maintaining consistent state in the game world. If you're at all interested about the networking aspects behind online games, it's well worth a read.

That's all from us this week. Have a great weekend, and we'll see you back here on Monday!

 

Microsoft R Open 3.2.3 update

We've released a minor update to Microsoft R Open 3.2.3 to address issues that some people were experiencing. The update available now on MRAN fixes the following issues:

• The Windows R GUI (RGUI.exe) could crash when typing beyond the bounds of the visible window.
• The "R" command wasn't available from the Terminal in OS X El Capitan.
• TCL/TK wasn't working on some Linux platforms.
• Using MKL on Windows 7 resulted in a "0xc000007b error".

If you didn't experience any of these problems there's no need to upgrade, but if you did (or if you simply haven't installed MRO 3.2.3) yet, you can download MRO for Windows, Mac and Linux here. Note that to get the performance improvements, you'll need to download and update the MKL math libraries as well.

MRAN: Download Microsoft R Open

Bay Area useR Group Lightning Talks

by Joseph Rickert

Earlier this month the Bay Area useR Group (BARUG) held it annual lightning talk meeting. This is by far our most popular meeting format: eight, 15 minute talks (12 minutes speaking and 3 minutes Q & A while the next speaker is setting up) packed into a two hour time slot. The intensity seems to really energize the speakers and engaged the audience. 

Bradley Shanrock-Solberg kicked off the event with delightful example of an R Monte Carlo simulation based on his wildpoker package that you can find on CRAN. I have never seen a more prepared lightning talk presenter: high energy, a royal flush presentation and a four color printed hand out just in case you have trouble keeping up with him for the 12 minutes. In a series of well conceived plots Bradley showed how, for a number of different poker variations, the best hand changes as the game progresses. The number of players who start the game, the number who stay until the showdown, wildcards and many more contingent events dynamically change the value of your hand. Bradley is definitely the guy for your next trip to Vegas.

 

If you are a poker player, you will definitely want to check out his package and supplementary material: his paper on Winning More at Dealers Choice Poker and his example of why rules of thumb fail.

William Sundstrom, professor of Economics at Santa Clara University, gave an entertaining and thought provoking presentation on teaching R to undergraduate Econometrics students. One interesting observation that generated some discussion was that even though today's students are "digital natives" having grown up using intelligent devices of all kinds, many of them are nevertheless "digital naïfs". The following slide, a reprint of an email from one of Professor Sundstrom's students, captures some of their frustration. 

 

David Ouyang MD, a Stanford resident, a guy who sometime puts in 73+ hour work weeks, presented some explorations of the Epic electronic medical records data set he is analyzing in his spare time. The following plot shows the distribution of physician's interactions with the Epic system over the course of a day.

Dennis Noren, a long time BARUG member and contributor showed some results from a recommendation App he is building on top of The Movie database. The following slide from his presentation shows a Shiny dashboard he build to drive a parallel plot. The interactivity really makes the plot useful.

If you are thinking about betting on the Academy Awards you might want to consider Keith Everett's predictions based on his GLMNet Model.

Nelson Auner talked about Modern KPI Tracking in R. My favorite slide describes the behavior of Execs who don't work for companies that sell BI Software:

And that wasn't all of it! We also had a update on data.table from Matt Dowle himself, and introduction to exploring large data sets with Apache Spark from Hossein Falaki.

Many thanks to Earl Hubbell and the folks at Thermo Fisher Scientific who hosted the meeting.