Revolutions

The useR! 2016 Tutorials

by Joseph Rickert

Over the years I have seen several excellent tutorials at useR!conferences that were not only very satisfying "you had to be there" experiences but were also backed up with meticulously prepared materials of lasting value. This year, quite a few  useR!20i6 tutorials measure up to this level of quality. My take on why things turned out this way is that GitHub, Markdown, and Jupyter notebooks have been universally adopted as workshop / tutorial creation tools, and that having the right tools encourages creativity and draws out one's best efforts.

Jenny Bryan's tutorial Using Git and GitHub with R, Rstudio, and R Markdown and the tutorial by Andrie de Vries and Micheleen Harris: Using R with Jupyter notebooks for reproducible research are two superb, Escheresque self-referencing examples of what I am talking about. Bryan's tutorial which uses GitHub and R Markdown to teach GitHub and R Markdown is an impressive introduction to these two essential resources. And, the tutorial by de Vries and Harris makes very effective use of GitHub and Jupyter Notebooks. Moreover, this tutorial sets the gold standard for how to set up a system for interactive user participation. Harris and de Vries staged their tutorial on Microsoft's Azure Data Science VM. The Linux version of this VM comes provisioned with JupyterHub, a set of processes that enables a multi-user Jupyter Notebook server. Once the VM is loaded with the training materials, its only a matter of giving students a username and password to grant them immediate access to the interactive workshop materials. Have a look at notebook 06 to see how to set all of this up.

After seeing this, and comparing it to other tutorials where instructors wasted the better part of an hour trying to get students up and running with local copies of their course materials I can't see why everyone wouldn't opt for a cloud solution to this problem. When word gets out, the Data Science VM is going to be the standard for delivering technical workshops.

Computerworld's advanced beginner's guide to R

Many newcomers to R got their start learning the language with Computerworld's Beginner's Guide to R, a 6-part introduction to the basics of the language. Now, budding R users who want to take their skills to the next level have a new guide to help them: Computerword's Advanced Beginner's Guide to R. Written by Sharon Machlis, author of the prior Beginner's guide and regular reporter of R news at Computerworld, this new 72-page guide dives into some trickier topics related to R: extracting data via API, data wrangling, and data visualization. 

On the data wrangling front, the guide provides some recipes for handling messy data. You'll learn how to transform data and add the resulting data as a new column to a data frame. There's also an extended look at restructuring data: transforming "wide" data to "long" data, and vice versa.

For visualizing data, there's a basic intro to the ggplot2 package and its grammar of graphics. There's also an in-depth tutorial on creating choropleths: geographics maps with regions shaded by data values.

And as an in-dept example of importing data, you'll learn how to use the Google Analytics API to download and prepare data on traffic to your website.

Finally, don't miss the comprehensive index of R packages for data import, data wrangling, and visualization.

To download the 72-page PDF (free by providing your email address) visit Computerworld at the link below.

Computerworld Crash Course: Advanced Beginner's Guide to R

Apache Spark integrated with Microsoft R Server for Hadoop

by Bill Jacobs, Microsoft Advanced Analytics Product Marketing

They say that time is infinite.  Seem to me data is fast becoming the same.  Or perhaps it's becoming true that our thirst for speed is providing eternal job security to computer scientists who can deliver it.

Apache Spark, one of the Apache Foundation's fastest-growing open source projects, delivers new levels of speed to computing clusters, combining in-memory computing and efficient parallelization.  With Spark, Hadoop clusters and data lakes can achieve speeds far greater than available with Hadoop's MapReduce framework.

We're happy to announce this month that Microsoft has integrated support for Apache Spark into Microsoft R Server for Hadoop, bringing Spark's speed advantages within the reach of R users.

But how much faster is it?  <drum-roll please...>

To measure performance, we first pitted R Server for Hadoop running on MapReduce against R Server for Hadoop running R algorithms in Apache Spark on Hadoop. In our testing using R Server to move work into Spark boosted GLM speeds by roughly 6x over R Server on Hadoop MapReduce on the same cluster and the same data.  That's quite a testament to Spark's new distributed compute engine in-memory architecture.

Second, more pertinent to users of open source R and CRAN algorithms (and perhaps struggling with scale limits), we compared R Server on a five node Spark cluster to open source R with CRAN algorithms which can only run on a single server.  We expected faster, and we go it. R Server ran GLM 125 times faster on five times the hardware, showing the combined speed of R Server's parallelized algorithms and Spark's in-memory architecture.

Hadoop and Spark have a reputation of being "testy" to develop for, requiring fluency in programming languages including Java, Python or Scala skills.  For R users involved in things like risk analytics, insurance underwriting, ad optimization or fraud detection, speed is only useful if it's easily achievable.  We built R Server version 8 for Hadoop and Spark for this case.

For most R users, however, it's what you learn from the data that matters, often making effort to learn a new language a bridge too far.  And so, it is the intersection of speed and ease of use that matters the most.

Ease of use was treated as a top engineering priority for Spark integration.  As a result, using Spark requires only minimal script changes.  CRAN R users can add R Server's remote execution context settings and substitute ScaleR algorithms in order to run parallelized computation on large data sets in Spark.  For existing users of R Server for Hadoop, it's easier still, requiring them to change the context setting from MapReduce to Spark.  

The combined speed and simplicity of R Server on Spark are available to preview today in the Azure Cloud and will be generally available in Azure later this summer.  For use on premises, MapR, Hortonworks and Cloudera users can install Microsoft R Server for Hadoop, version 8.0.5, which will become available the last week of June 2016. 

If you're interested in learning more, here's a short video on Microsoft's Channel9 blog site describing our Spark support in Microsoft R Server version 8.0.5. 

If you'd like to test drive R Server on Spark and Hadoop in the Azure cloud, you can start for free at http://azure.microsoft.com.  With your Azure account, you can spin-up a cluster with R Server and Spark on HDInsight Hadoop in about the time it takes to make a decent lunch.

Livestreaming of useR! 2016 conference begins tomorrow, June 28

The useR! 2016 conference, the annual gathering of R users from around the world, is already underway at Stanford University. Today is a day of interactive tutorials, and the presentation program begins tomorrow.

But don't worry if you weren't able to make it to California, or if you missed out on getting a ticket for the conference. (You're not alone: registrations sold out within 2 weeks.) As a service to R users everywhere, Microsoft has sent down a camera crew to the conference, and will be livestreaming many of the sessions on Channel 9, the Microsoft video portal. Visit the R User Conference 2016 streaming page to explore the 200+ sessions that will be recorded, beginning with Rick Becker's keynote Forty Years of S at 9AM Pacific time on June 28.

Even if you can't catch the sessions live, all of the presentations will be preserved on Channel 9 to watch at any time. You'll also be download and watch any of the presentations in any medium you choose, as all the presentations will be made freely available under the Creative Commons Attribution-NonCommercial-NoDerivs 3.0 Unported license.

Several of the data science team from Microsoft, myself included, are at useR! 2016 this week, so if you are at the conference please come and say hi, and visit the Microsoft booth to pick up some R swag. You might also be interested to check out the talks from the Microsoft team on the livestream. (Click on the dates to add a link to your calendar.)

• Changing lives with Data Science at Microsoft (David Smith) June 28, 2016 from 1:45PM to 2:00PM
• R at Microsoft (David Smith) June 28, 2016 from 4:45PM to 5:00PM
• Exploring the R / SQL boundary (Gopi Kumar) June 28, 2016 from 11:45AM to 12:00PM
• Forecasting Revenue for S&P 500 Companies Using the baselineforecast Package (Konstantin Golyaev) June 28, 2016 from 10:30AM to 10:45AM
• A Lap Around R Tools for Visual Studio (John Lam) June 28, 2016 from 1:15PM to 1:30PM
• When will this machine fail? (Xinwei Xue) June 28, 2016 from 5:00PM to 5:15PM

Whether you're attending the conference in person or watching the sessions via livestream, I hope you enjoy user! 2016 as much as I'm looking forward to it!  

Channel 9: Livestreaming: https://aka.ms/user2016conference; Recordings: useR! 2016 international R User conference

Because it's Friday: A robot writes a movie

What happens when you take the scripts from dozens of sci-fi movies and TV series, and feed them (along with a couple of seed prompts) into a long short-term memory recurrent neural network? You get this bizarre screenplay, with dialogue for three characters (named H, H, and C -- one of the H's had to be renamed H2) and stage directions like

             C  
    Well, I have to go to the skull.  I
    don't know. 

He picks up a light screen and fights the security force of
the particles of a transmission on his face.

Director Oscar Sharp assembled a cast and crew (including Silicon Valley's Thomas Middleditch) and produced the screenplay as a short film. The results are ... well, just watch:

 

Ars Technica has a fascinating story on the making of the movie and the AI who wrote the screenplay. Benjamin (the name the AI gave itself) even chimes in in the comment section.

That's all from us this week. We'll be back next week, live from from the useR! conference in Stanford. See you there!

 

Amazon X-ray data provides insight into movie characters

I'm a regular user of Amazon Video: as someone who spends a fair bit of time on planes, it's great to be able to download some of my favourite shows (hello, Orphan Black and Vikings) and catch up on episodes during the trip. Amazon Video has a useful feature, too: if you forget the name of a character, or if you see an an actor and wonder what other things they've been in, you can press pause and see a list of the actors appearing in the scene:

The feature is called X-Ray, and I just learned from Curious Gnu that you can download the data files that provide the X-Ray data, as long as you have an Amazon Video subscription. The process isn't trivial (it involves using Developer Tools in Chrome to exploit a loophole in the streaming process), but at the end of the day you end up with a JSON file whcih you can easily read into R with the jsonlite package and then plot the data using any of R's visualization tools. For example, Curious Gnu created this visualization of the screentime of major Star Wars: The Force Awakens characters.

Of course, another way of collecting data like this is to analyze the movie script, but most scripts aren't readily available online (and even then, the format isn't exactly standardized). The advantage of using Amazon X-Ray JSON files is that the data is already structured, and are available for more movies and series ... for as long as Amazon leaves the downloading loophole open.

For more details on how to download Amazon X-Ray data and more examples of movie data visualizations from Curious Gnu, follow the link below.

Curious Gnu: Using Amazon's X-Ray to Visualize Characters' Screen Time (via Verena Haunschmid)

R, Stan and Bayesian Statistics

by Joseph Rickert

Just about two and a half years ago I wrote about some resources for doing Bayesian statistics in R. Motivated by the tutorial Modern Bayesian Tools for Time Series Analysis by Harte and Weylandt that I attended at R/Finance last month, and the upcoming tutorial An Introduction to Bayesian Inference using R Interfaces to Stan that Ben Goodrich is going to give at useR! I thought I'd look into what's new. Well, Stan is what's new! Yes, Stan has been under development and available for some time. But somehow, while I wasn't paying close attention, two things happened: (1) the rstan package evolved to make the mechanics of doing Bayesian in R analysis really easy and (2) the Stan team produced and/or organized an amazing amount of documentation. 

My impressions of doing Bayesian analysis in R were set in the WinBUGS era. The separate WinBUGs installation was always tricky, and then moving between the BRugs and R2WinBUGS packages presented some additional challenges. My recent Stan experience was nothing like this. I had everything up and running in just a few minutes. The directions for getting started with rstan are clear and explicit about making sure that you have the right tool chain in place for your platform. Since I am running R 3.3.0 on Windows 10 I installed Rtools34. This went quickly and as expected except that C:\Rtools\gcc-4.x-y\bin did not show up in my path variable. Not a big deal: I used the menus in the Windows System Properties box to edit the Path statement by hand. After this, rstan installed like any other R package and I was able to run the 8schools example from the package vignette. The following 10 minute video by Ehsan Karim takes you through the install process and the vignette example.

 

The Stan documentation includes four major components: (1) The Stan Language Manual, (2) Examples of fully worked out problems, (3) Contributed Case Studies and (4) both slides and video tutorials. This is an incredibly rich cache of resources that makes a very credible case for the ambitious project of teaching people with some R experience both Bayesian Statistics and Stan at the same time. The "trick" here is that the documentation operates at multiple levels of sophistication with entry points for students with different backgrounds. For example, a person with some R and the modest statistics background required for approaching Gelman and Hill's extraordinary text: Data Analysis Using Regression and Multilevel/Hierarchical Models can immediately beginning running rstan code for the book's examples. To run the rstan version of the example in section 5.1, Logistic Regression with One Predictor, with no changes a student only needs only to copy the R scripts and data into her local environment. In this case, she would need the R script:  5._LogisticRegressionWithOnePredictor. R, the data: nes1992_vote.data.R and the Stan code: nes_logit.stan.  The Stan code for this simple model is about as straightforward as it gets: variable declarations, parameter identification and the model itself.

data {
	int<lower=0> N;
	vector[N] income;
	int<lower=0,upper=1> vote[N];
	}
	parameters {
	vector[2] beta;
	}
	model {
	vote ~ bernoulli_logit(beta[1] + beta[2] * income);
	}

Running the script will produce the iconic logistic regression plot:

I'll wind down by curbing my enthusiasm just a little by pointing out that Stan is not the only game in town. JAGS is a popular alternative, and there is plenty that can be done with unaugmented R code alone as the Bayesian Inference Task View makes abundantly clear. 

If you are a book person and new to Bayesian statistics, I highly recommend Bayesian Essentials with R by Jean-Michel Marin and Christian Robert. The authors provide a compact introduction to Bayesian statistics that is backed up with numerous R examples. Also, the new book by Richard McElreath, Statistical Rethinking: A Bayesian Course with Examples in R and Stan looks like it is going to be an outstanding read. The online supplements to the book are certainly worth a look.

Finally, if you are a Bayesian or a thinking about becoming one and you are going to useR!, be sure to catch the following talks:

• Bayesian analysis of generalized linear mixed models with JAGS, by Martyn Plummer
• bamdit: An R Package for Bayesian meta-Analysis of diagnostic test data by Pablo Emilio Verde
• Fitting complex Bayesian models with R-INLA and MCMC by Virgilio Gómez-Rubio
• bayesboot: An R package for easy Bayesian bootstrapping by Rasmus Arnling Bååth
• An Introduction to Bayesian Inference using R Interfaces to Stan by Ben Goodrich
• DiLeMMa - Distributed Learning with Markov Chain Monte Carlo Algorithms Using the ROAR Package by Ali Zaidi

R 3.3.1 now available

Peter Dalgaard announced yesterday on behalf of the R core team that R 3.3.1, the latest update to the R language, is now available for download from your local CRAN mirror. As of this writing, binaries of R 3.3.1 are available for Windows and Linux; the Mac version should appear very soon.

This minor update, codenamed "Bug in Your Hair", makes a few small fixes to the R 3.3.0 release. Bugs fixed include mostly rarely-encountered cases like generating Gamma random numbers with zero or infinite rate parameters, and correctly matching text (with the match function) that only differed in the encoding. 

There are no new features in this update, and all R code and packages should work with R 3.3.1 just as they did with R 3.3.0. For a complete list of the fixes in R 3.3.1, follow the link below.

r-announce mailing list: R 3.3.1 is released

Exploring Global Internet Performance Data Using R

by Lourdes O. Montenegro

Lourdes O. Montenegro is a PhD candidate at the Lee Kuan Yew School of Public Policy, National University of Singapore. Her research interests cover the intersection of applied data science, technology, economics and public policy.

Many of us now find it hard to live without a good quality internet connection. As a result, there is growing interest in characterizing and comparing internet performance metrics. For example, when planning to switch internet service providers or considering a move to a new city or country, internet users may want to research in advance what to expect in terms of download speed or latency. Cloud companies may want to provision adequately for different markets with varying levels of internet quality. And governments may want to benchmark their communications infrastructure and invest accordingly. Whatever the purpose, a consortium of research, industry and public interest organizations called Measurement Lab has made available the largest open and verifiable internet performance dataset in the planet. With the help of a combination of packages, R users can easily query, explore and visualize this large dataset at no cost.

In the example that follows, we use the bigrquery package to query and download results from the Network Diagnostic Tool (NDT) used by the U.S. FCC. The bigrquery package provides an interface to Google BigQuery which hosts NDT results along with several other Measurement Lab (M-Lab) datasets. However, R users will need to first set-up a BigQuery account and join the M-Lab mailing list to authenticate. Detailed instructions are provided on the M-Lab website. Once done, SQL-like queries can be run from within R. The results are saved as a dataframe on which further analysis can be performed. Aside from the convenience of working within the R environment, the bigrquery package has another advantage: the only limitation to the size of the query results that can be saved for further exploration is the amount of available RAM. In contrast, the BigQuery web interface only allows export to .csv format of query results which are at or below 16,000 rows.

The following R script gives us the average download speed (in Mbps) per country in 2015.[1] The SQL-like query can be modified to return other internet performance metrics that may be of interest to the R user such as upload speed, round-trip time (latency) and packet re-transmission rates.

# Querying average download speed per country in 2015
require(bigrquery)
downquery_template <- "SELECT
connection_spec.client_geolocation.country_code AS country,  
AVG(8 * web100_log_entry.snap.HCThruOctetsAcked/ (web100_log_entry.snap.SndLimTimeRwin +
web100_log_entry.snap.SndLimTimeCwnd + web100_log_entry.snap.SndLimTimeSnd)) AS downloadThroughput,
COUNT(DISTINCT test_id) AS tests,
FROM
plx.google:m_lab.ndt.all
WHERE
IS_EXPLICITLY_DEFINED(web100_log_entry.connection_spec.remote_ip)
AND IS_EXPLICITLY_DEFINED(web100_log_entry.connection_spec.local_ip)
AND IS_EXPLICITLY_DEFINED(web100_log_entry.snap.HCThruOctetsAcked)
AND IS_EXPLICITLY_DEFINED(web100_log_entry.snap.SndLimTimeRwin)
AND IS_EXPLICITLY_DEFINED(web100_log_entry.snap.SndLimTimeCwnd)
AND IS_EXPLICITLY_DEFINED(web100_log_entry.snap.SndLimTimeSnd)
AND project = 0
AND IS_EXPLICITLY_DEFINED(connection_spec.data_direction)
AND connection_spec.data_direction = 1
AND web100_log_entry.snap.HCThruOctetsAcked >= 8192
AND (web100_log_entry.snap.SndLimTimeRwin +
web100_log_entry.snap.SndLimTimeCwnd +
web100_log_entry.snap.SndLimTimeSnd) >= 9000000
AND (web100_log_entry.snap.SndLimTimeRwin +
web100_log_entry.snap.SndLimTimeCwnd +  
web100_log_entry.snap.SndLimTimeSnd) < 3600000000
AND IS_EXPLICITLY_DEFINED(web100_log_entry.snap.CongSignals)
AND web100_log_entry.snap.CongSignals > 0
AND (web100_log_entry.snap.State == 1 OR (web100_log_entry.snap.State >= 5 AND web100_log_entry.snap.State <= 11))
AND web100_log_entry.log_time >= PARSE_UTC_USEC('2015-01-01 00:00:00') / POW(10, 6) 
AND web100_log_entry.log_time < PARSE_UTC_USEC('2016-01-01 00:00:00') / POW(10, 6)
GROUP BY country
ORDER BY country ASC;"
downresult <- query_exec(downquery_template, project="measurement-lab", max_pages=Inf)

Once we have the query results in a dataframe, we can proceed to visualize and map average download speeds for each country. To do this, we can use the rworldmap package which offers a relatively simple way to map country level and gridded user datasets. Mapping is done mainly through two functions: (1) joinCountryData2Map joins the query results with shapefiles of country boundaries; (2) mapCountryData plots the chloropleth map. Note that the join is best effected using either two or three-letter ISO country codes, although the rworldmap package also allows join columns filled with country names.

In order to make the chloropleth map prettier and more comprehensible, we can augment with a combination of the classInt package to calculate natural breaks in the range of download speed results and the RColorBrewer package for a wider selection of color schemes. In the succeeding R script, we specify the Jenks method to cluster download speed results in such a way that minimizes deviation from the class mean within each class but maximizes deviation across class means. Compared to other methods for clustering download speed results, the Jenks method draws a sharper picture of countries clocking greater than 25 Mbps on average.

require(rworldmap)
require(classInt)
require(RColorBrewer)
 
downloadmap <- joinCountryData2Map(downresult
                              , joinCode='ISO2'
                              , nameJoinColumn='country'
                              , verbose='TRUE')
par(mai=c(0,0,0.2,0),xaxs="i",yaxs="i")
 
#getting class intervals using a 'jenks' classification in classInt package
classInt <- classInt::classIntervals( downloadmap[["downloadThroughput"]], n=5, style="jenks")
catMethod = classInt[["brks"]]
#getting a colour scheme from the RColorBrewer package
colourPalette <- RColorBrewer::brewer.pal(5,'RdPu')
mapParams <- mapCountryData(downloadmap, nameColumnToPlot="downloadThroughput", 
                            mapTitle="Download Speed (mbps)", catMethod=catMethod, 
                            colourPalette=colourPalette,
                            addLegend=FALSE)
do.call( addMapLegend, c(mapParams, legendWidth=0.5, legendLabels="all", 
                         legendIntervals="data", legendMar = 2))

 

Looking at the map, we see that the UK, Japan, Romania, Sweden, Taiwan, The Netherlands, Denmark and Singapore (if we squint!) are the best places to be for internet speed addicts. Until further investigation, we can safely discount the suspiciously high results for North Korea since the number of observations are too low. In contrast, average download speeds in South Korea might be grossly underestimated when measured from foreign servers, as may be the case with NDT results, since most Koreans access locally hosted content. There are, of course, a number of caveats worth mentioning before drawing any conclusions regarding the causes of varying internet performance between countries. Confounding factors such as distance from the client to the test server, the client's operating system, and the proportion of fixed broadband to wireless connections will need to be controlled for. Despite these caveats, this tentative exploration already reveals interesting patterns in global internet performance that is worth a closer look.

[1]    Thanks to Stephen McInerney and Chris Ritzo for code advice.

Updates to the 'forecast' package for R

The forecast package for R, created and maintained by Professor Rob Hyndman of Monash University, is one of the more useful R packages available available on CRAN. Statistical forecasting — the process of predicting the future value of a time series — is used in just about every realm of data analysis, whether it's trying to predict a future stock price or trying to anticipate changes in the weather. If you're looking to learn about forecasting, a great place to start is the online book Forecasting: Principles and Practice (by Hyndman and George Athana­sopou­los) which walks you through the theory and practice, with many examples in R based on the forecast package. Topics covered include multiple regression, Time series decomposition, exponential smoothing, and ARIMA models.

The forecast package itself recently received a major update, to version 7. One major new capability is the ability to easily chart forecasts using the ggplot2 package with the new autoplot function. For example:

fc <- forecast(fdeaths)
autoplot(fc)
 

You can also add forecasts to any ggplot using the new geom_forecasts geom provided by the forecast package:

autoplot(mdeaths) + geom_forecast(h=36, level=c(50,80,95))

There have been several updates to the forecasting functions as well. The function for fitting linear models to time series data, tslm, has been rewritten to be more compatible with the standard lm function. It's now possible to forecast means (as well as medians) when using Box-Cox transformations. And you can now apply neural networks to time series data by building a nonlinear autoregressive model with the new nnetar function.

Those are just some of the highlights of the updates to the forecast package in version 7. For complete details, follow the links to Rob Hyndman's blog, below.

Hyndsight: forecast v7 and ggplot2 graphics ; Forecast v7 (part 2) (via traims)