Revolutions

Analysis of Trump's State of the Union Speech, with R

President Trump's State of the Union speech was last night, and it seemed to me it dragged on a bit. That's because it was apparently the slowest SOTU speech in history, based on average number of words spoken per minute. The chart below, based on these data, was created by Josh Katz. (The R code for the chart is here. Note the use of his needs package to load the required R packages.)

Peter Aldhous at Buzzfeed analyzed last night's State of the Union speech as well. Not only was the speech delivered slowly, the script was at a 9-year-old's reading level, following a long-term decline in the complexity of State of the Union speeches.

You can find more analysis of this and past SOTU speeches at the link below. The R code behind the analysis can be found here. (You may also want to check out this analysis of Trump's Twitter activity and the associated R code.)

Buzzfeed: “I Have The Best Words.” How Trump’s First SOTU Compares To All The Others.

 

Speed up R with Parallel Programming in the Cloud

This past weekend I attended the R User Day at Data Day Texas in Austin. It was a great event, mainly because so many awesome people from the R community came to give some really interesting talks. Lucy D’Agostino McGowan has kindly provided a list of the talks and links to slides, and I thoroughly recommend checking it out: you're sure to find a talk (or two, or five or ten) that interests you.

My own talk was on Speeding up R with Parallel Programming in the Cloud, where I talked about using the doAzureParallel package to launch clusters for use with the foreach function, and using aztk to launch Spark clusters for use with the sparklyr package. I've embdeded the slides below: it's not quite the same without the demos (and sadly there was no video recording), but I've included screenshots in the slides to hopefully make it clear.

Live Free or Dichotomize: Wrangling Data Day Texas Slides

Because it's Friday: Excel Painter

Excel often gets criticism when it gets used for applications it's not really designed for. In this instance it's being used in completely the wrong way, and the results are beautiful (via FlowingData):

That's all from the blog for this week. We'll be back next week, and in the meantime have a great weekend!

Speed up simulations in R with doAzureParallel

I'm a big fan using R to simulate data. When I'm trying to understand a data set, my first step is sometimes to simulate data from a model and compare the results to the data, before I go down the path of fitting an analytical model directly. Simulations are easy to code in R, but they can sometimes take a while to run — especially if there are a bunch of parameters you want to explore, which in turn requires a bunch of simulations.

When your pc only has four core processors and your parallel processing across three of the.#DataScience #RStats #Waiting #Multitasking pic.twitter.com/iVkkr7ibox

— Patrick Williams (@unbalancedDad) January 24, 2018

In this post, I'll provide a simple example of running multiple simulations in R, and show how you can speed up the process by running the simulations in parallel: either on your own machine, or on a cluster of machines in the Azure cloud using the doAzureParallel package.

To demonstrate this, let's use a simple simulation example: the birthday problem. Simply stated, the goal is to calculate for a room of \(N\) people the probability that someone in the room shares a birthday with someone else in the room. Now, you can calculate this probability analytically — R even has a function for it — but this one of those situations where it's quicker for me to write a simulation that it would be to figure out the analytical result. (Better yet, my simulation accounts for February 29 birthdays, which the standard result doesn't. Take that, distribution analysis!) Here's an R function that simulates 10,000 rooms, and counts the number of times a room of n people includes a shared birthday:

You can compare the results to the built-in function pbirthday to make sure it's working, though you should include the feb29=FALSE option for an apples-to-apples comparison. The more simulations (nsims) you use, the closer the results will be.

We want to find the number of people in the room where the probability of a match is closest to 50%. We're not exactly sure what that number is, but we can fund out by calculating the probability for a range of room sizes, plotting the results, and see where the probability crosses 0.50. Here's a simple for loop that calculates the probability for room sizes from 1 to 100 people:

Scraping a website with 5 lines of R code

In what is rapidly becoming a series — cool things you can do with R in a tweet — Julia Silge demonstrates scraping the list of members of the US house of representatives on Wikipedia in just 5 R statements:

library(rvest)
library(tidyverse)

h <- read_html("https://t.co/gloY1eErBn")

reps <- h %>%
html_node("#mw-content-text > div > table:nth-child(18)") %>%
html_table()

reps <- reps[,c(1:2,4:9)] %>%
as_tibble() pic.twitter.com/25ANm7BHkj

— Julia Silge (@juliasilge) January 12, 2018

Since Twitter munges the URL in the third line when you cut-and-paste, here's a plain-text version of Julia's code:

library(rvest)
library(tidyverse)

h <- read_html("https://en.wikipedia.org/wiki/Current_members_of_the_United_States_House_of_Representatives")

reps <- h %>%
 html_node("#mw-content-text > div > table:nth-child(18)") %>%
 html_table()

reps <- reps[,c(1:2,4:9)] %>% as_tibble()

And sure enough, here's what the reps object looks like in the RStudio viewer:

As Julia notes it's not perfect, but you're still 95% of the way there to gathering data from a page intended for human rather than computer consumption. Impressive!

Visualize your Strava routes with R

Strava is a fitness app that records your activities, including the routes of your walks, rides and runs. The service also provides an API that allows you to extract all of your data for analysis. University of Melbourne research fellow Marcus Volz created an R package to download and visualize Strava data, and created a chart to visualize all of his runs over six years as a small multiple.

Inspired by his work (and the availability of the R package he created), others also visualized bike rides, activity calendars, and aggregated route maps with elevation data. (You can see several examples in the Twitter moment embedded below.) If you'd like to download your own Strava data, all you need a Strava access token, a recent version of R (3.4.3 or later) and the strava package found on Github.

Because it's Friday: Principles and Values

Most companies publish mission and vision statements, and some also publish a detailed list of principles that underlie the company ethos. But what makes a good collection of principles, and does writing them down really matter? At the recent Monktoberfest conference, Bryan Cantrill argued that yes, they do matter, mostly by way of some really egregious counterexamples.

That's all from the blog for this week. We'll be back on Monday — have a great weekend!

Registration and talk proposals open Monday for useR!2018

Registration will open on Monday (January 22) for useR! 2018, the official R user conference to be held in Brisbane, Australia July 10-13. If you haven't been to a useR! conference before, it's a fantastic opportunity to meet and mingle with other R users from around the world, see talks on R packages and applications, and attend tutorials for deep dives on R-related topics. This year's conference will also feature keynotes from Jenny Bryan, Steph De Silva, Heike Hofmann, Thomas Lin Pedersen, Roger Peng and Bill Venables. It's my favourite conference of the year, and I'm particularly looking forward to this one.

This video from last year's conference in Brussels (a sell-out with over 1,1000 attendees) will give you a sense of what a useR! conference is like:

The useR! conference brought to you by the R Foundation and is 100% community-led. That includes the content: the vast majority of talks come directly from R users. If you've written an R package, performed an interesting analysis with R, or simply have something to share of interest to the R community, consider proposing a talk by submitting an abstract. (Abstract submissions are open now.) Most talks are 20 minutes, but you can also propose a 5-minute lightning talk or a poster. If you're not sure what kind of talk you might want to give, check out the program from useR!2017 for inspiration. R-Ladies, which promotes gender diversity in the R community, can also provide guidance on abstracts. Note that all proposals must comply with the conference code of conduct.

Early-bird registrations close on March 15, and while general registration will be open until June my advice is to get in early, as this year's conference is likely to sell out once again. If you want to propose a talk, submissions are due by March 2 (but early submissions have a better chance of being accepted). Follow @user!2018_conf on Twitter for updates about the conference, and click the links below to register or submit an abstract. I look forward to seeing you in Brisbane!

Update Jan 19: Registrations will now open January 22

useR! 2018: Registration; Abstract submission

Microsoft R Open 3.4.3 now available

Microsoft R Open (MRO), Microsoft's enhanced distribution of open source R, has been upgraded to version 3.4.3 and is now available for download for Windows, Mac, and Linux. This update upgrades the R language engine to the latest R (version 3.4.3) and updates the bundled packages (specifically: checkpoint, curl, doParallel, foreach, and iterators) to new versions. 

MRO is 100% compatible with all R packages. MRO 3.4.3 points to a fixed CRAN snapshot taken on January 1 2018, and you can see some highlights of new packages released since the prior version of MRO on the Spotlights page. As always, you can use the built-in checkpoint package to access packages from an earlier date (for reproducibility) or a later date (to access new and updated packages).

MRO 3.4.3 is based on R 3.4.3, a minor update to the R engine (you can see the detailed list of updates to R here). This update is backwards-compatible with R 3.4.2 (and MRO 3.4.2), so you shouldn't encounter an new issues by upgrading. 

We hope you find Microsoft R Open useful, and if you have any comments or questions please visit the Microsoft R Open forum. You can follow the development of Microsoft R Open at the MRO Github repository. To download Microsoft R Open, simply follow the link below.

MRAN: Download Microsoft R Open

A simple way to set up a SparklyR cluster on Azure

The SparklyR package from RStudio provides a high-level interface to Spark from R. This means you can create R objects that point to data frames stored in the Spark cluster and apply some familiar R paradigms (like dplyr) to the data, all the while leveraging Spark's distributed architecture without having to worry about memory limitations in R. You can also access the distributed machine-learning algorithms included in Spark directly from R functions. 

If you don't happen to have a cluster of Spark-enabled machines set up in a nearby well-ventilated closet, you can easily set one up in your favorite cloud service. For Azure, one option is to launch a Spark cluster in HDInsight, which also includes the extensions of Microsoft ML Server. While this service recently had a significant price reduction, it's still more expensive than running a "vanilla" Spark-and-R cluster. If you'd like to take the vanilla route, a new guide details how to set up Spark cluster on Azure for use with SparklyR.

All of the details are provided in the link below, but the guide basically provides the Azure Distributed Data Engineering Toolkit shell commands to provision a Spark cluster, connect SparklyR to the cluster, and then interact with it via RStudio Server. This includes the ability to launch the cluster with pre-emptable low-priority VMs, a cost-effective option (up to 80% cheaper!) for non-critical workloads. Check out the details at the link below.

Github (Azure): How to use SparklyR on Azure with AZTK