Revolutions

Some R Highlights from H20 World

by Joseph Rickert

H₂O.ai held its first H₂O World conference over two days at the Computer History Museum in Mountain View, CA. Although the main purpose of the conference was to promote the company's rich set of Java based machne learning algorithms and announce their new products Flow and Play there were quite a few sessions devoted to R and statistics in general.

Before I describe some of these, a few words about the conference itself. H20 World was exceptionally well run, especially for a first try with over 500 people attending (my estimate). The venue is an interesting, accommodating space with plenty of parking, that played well with what, I think, must have been an underlying theme of the conference: acknowledging contributions of past generations of computer scientists and statisticians. There were two stages offering simultaneous talks for at least part of the conference: The Paul Erdős stage and the John Tukey stage. Tukey I got, why put such an eccentric mathematician front and center? I was puzzled until Sri Ambati, H₂O.ai's CEO and co-founder remarked that he admired Erdős because of his great generosity with collaboration. To a greater extent than most similar events, H₂O World itself felt like a collaboration. There was plenty of opportunity to interact with other attendees, speakers and H₂0 technical staff (The whole company must have been there). Data scientists, developers and Marketing staff were accessible and gracious with their time. Well done!

R was center stage for a good bit the hands on training that that occupied the first day of the conference. There were several sessions (Exploratory Data Analysis, Regression, Deep Learning, Clustering and Dimensionality Reduction) on accessing various H2O algorithms through the h2o R package and the H2O API. All of these moved quickly from R to running the custom H₂O alogorithms on the JVM. However, the message that came through is that R is the right environment for sophisticated machine learning.

Two great pleasures from the second day of the conference were Trevor Hastie's tutorial on the Gradient Boosting Machine and John Chamber's personal remembrances of John Tukey. It is unusual for a speaker to announce that he has been asked to condense a two hour talk into something just under an hour and then go on to speak slowly with great clarity, each sentence beguiling you into imagining that you are really following the details. (It would be very nice if the video of this talk would be made available.)

Two notable points from Trevor's lecure where understanding gradient boosting as minimizing the exponential loss function and the openness of the gbm algorithm to "tinkering". For the former point see Chapter 10 of the Elements of Statistical Learning or the more extended discussion in Schapire and Freund's Boosting: Foundations and Algorithms.

John Tukey spent 40 years at Bell Labs (1945 - 1985) and John Chamber's tenure there overlapped the last 20 years of Tukey's stay. Chambers who had the opportunity to observe Tukey over this extended period of time painted a moving and lifelike portrait of the man. According to Chambers, Tukey could be patient and gracious with customers and staff, provocative with his statistician colleagues and "intellectually intimidating". John remembered Richard Hamming saying: "John (Tukey) was a genius. I was not." Tukey apparently delighted in making up new terms when talking with fellow statisticians. For example, he called the top and bottom lines that identify the interquartile range on a box plot "hinges" not quartiles. I found it particularly interesting that Tukey would describe a statistic in terms of the process used to compute it, and not in terms of any underlying theory. Very unusual, I would think, for someone who earned a PhD in topology under Solomon Lefschetz. For more memories of John Tukey including more from John Chambers look here.

Other R related highlights were talks by Matt Dowle and Erin Ledell. Matt reprised the update on new features in data.table that he recently gave to the Bay Area useR Group and also presented interesting applications using data.table from UK insurance company Landmark, and KatRisk (Look here for KatRisk part of Matt's presentation).

Erin, author of the h20Ensemble package available on GitHub, delivered an exciting and informative talk on using ensembles of learners (combining gbm models and logistic regression models, for example) to create "superlearners".

Finally, I gave a short talk Revolution Analytics' recent work towards achieving reproducibility in R. The presentation motivates the need for reproducibility by examining the use of R in industry and science and describing how the checkpoint package and Revolution R Open, an open source distribution of R that points to a static repository can be helpful.

 

Eggnog for Thanksgiving

It's Thanksgiving Day here in the US, so we're taking the day off to spend some time with our families and to eat far too much food. If you're in the US or celebrating Thanksgiving elsewhere, enjoy the day! And for everyone in this season of joy, here's a handy app to scale your eggnog recipe, however many people are around your table. 

The R code and the Shiny app is from Hadley Wickham. Enjoy!

The beautiful R charts in London: The Information Capital

If you've lived in or simply love London, a wonderful new book for your coffee-table is London: The Information Capital. In 100 beautifully-rendered charts, the book explores the data that underlies the city and its residents. To create most of these charts, geographer James Cheshire and designer Oliver Uberti relied on programs written in R. Using the R programming language not only created beautiful results, it saved time: "a couple of lines of code in R saved a day of manually drawing lines". 

Take for example From Home To Work, the graphic illustrating the typical London-area commute. R's ggplot2 package was used to draw the invidual segments as transparent lines, which when overlaid build up the overall picture of commuter flows around cities and towns. The R graphic was then imported into Adobe Illustrator to set the color palette and add annotations. (FlowingData's Nathan Yau uses a similar process.) 

Another example is the chart below of cycle routes in London. (We reported on an earlier version of this chart back in 2012.) As the authors note, "hundreds of thousands of line segments are plotted here, making the graphic an excellent illustration of R’s power to plot large volumes of data."

You can learn more from the authors about how R was used to create the graphics in London: The Information Capital and see several more examples at the link below. And if you'd like a copy, you can buy the book here.

London: The Information Capital / Our Process: The Coder and Designer

What's the most popular German car where you live?

by Tim Winke
PhD student in Demography and Social Sciences in Berlin

This post has been abstracted from Tim's entry to a contest that Dalia Research is running based on a global smarthpone survey that they are conducting. Tim's entry post is available as is all of the code behind it. - editor

When people think about Germany, what comes to their mind? Oktoberfest, ok – but Mercedes might be second or BMW or Porsche. German car brands have a solid reputation all over the world, but how popular is each brand in different countries?

There are plenty of survey data out there but hardly anyone collects answers within a couple of days from 6 continents. A new start-up called Dalia Research found a way to use smartphone and tablet networks to conduct surveys. It’s not a separate app but works via thousands of apps where targeted users decide to take part in a survey in exchange for an incentive.

In August 2014, they asked 51 questions to young mobile users in 64 countries including Colombia, Iran and the Ukraine. This is impressive – you have access to opinions of 32 000 people collected within 4 days from all over the world – 500 respondents in each country – about their religion, what they think about the Unites States, about where the EU has global influence or if Qatar should host the 2022 FIFA World Cup, and also: “What is your favorite German car brand?”.

Surprisingly, as the map below shows, BMW seems to be the most popular German car brand – and Volkswagen does not reach the pole position in any country.

 

The ggplot2 stacked barchart provides even more detail.

To see how I employed dplyr, ggplot2 and rworldmap to construct these plots as well as how to integrate the survey data with world development indicators from the World Bank please have a look at my original post.

 

Twitter's R package for detecting breakouts in time series

With so many more devices and instruments connected to the "Internet of Things" these days, there's a whole lot more time series data available to analyze. But time series are typically quite noisy: how do you distinguish a short-term tick up or down from a true change in the underlying signal? To solve this problem, Twitter created the BreakoutDetection package for R, which decomposes a time series into a series of segments of one of three types:

• Steady state: The time series follows a fixed mean (with random noise around the mean);
• Mean shift: The time series jumps directly from one steady state to another;
• Ramp up / down: The time series transitions linearly from one steady state to another, over a fixed period of time.

Given a univariate time series (and a few tuning parameters), the breakout function will return a list of breakout points: times when these state transitions are detected. It uses a non-parametric algorithm (E-Divisive with Medians) to detect the breakout points, so no assumptions are made about the underlying distribution of the time series.

Twitter uses this R package to monitor the user experience on the Twitter network and detect when things are "Breaking Bad". Data scientist Randy Zwitch used the package to identify the dates of blog posts or references on Hacker News from his blog traffic data. (He also compared the algorithm to anomaly detection with the Adobe Analytics API.) And the University of Louisville School of Medicine has also looked at using the package to identify past influenza outbreaks from CDC data:

For more information about the BreakoutDetection package, check out Twitter's blog post linked below. You can download the BreakoutDetection R package itself from GitHub.

Twitter Engineering blog: Breakout detection in the wild (via FlowingData)

 

Because it's Friday: Too Many Cooks

This is just ... bizarre. But if you've ever watched American sitcoms from the 80's and 90's, and you like dark and surreal comedy as much as I do, you'll enjoy most (but probably not all!) of the 11-minute sketch. It's impossible to explain, just take a look (warning: some slightly NSFW images):

 

Apparently Too Many Cooks originally aired as a sketch on US cable channel Adult Swim in the graveyard shift, and later became a viral hit on YouTube. I'm just glad to see there are so many people (about 3.5M views as of this writing) that enjoy unusual comedy that you'd think given most of the bland dross on TV. My personal theory is that great comedy has to be terrible sometimes: it's only with a high mean and a high variance that the truly great stuff emerges. Monty Python is the canonical example for me: a lot of their sketches simply don't work, but those that do are just sublime.

Have a great weekend -- we'll be back on Monday!

Ford uses R for data-driven decision making

Mike Cavaretta is Ford Motor Company’s Chief Data Scientist, and was tasked by the incoming CEO Alan Mulally to help change the culture so that "important decisions within the company had to be based on data". In a feature article at Dataconomy, he reveals that R is a big part of this revolution at Ford: 

On the statistical side, we did a lot of stuff in R. ... We’ve done a lot more with R and we’re currently evaluating Pentaho. So we’ve really moved from more point solutions for solving particular problems, to more of a framework and understanding different needs in different areas. For example, there may be certain times when SAS is great for analysis because we already have implementations, and it’s easier to get that into production. There are other times when R is a better choice because it’s got certain packages that makes that analysis a lot easier, so we’re working on trying to put all that together. 

Data Science is revolutionizing Ford's products, too. Among the applications Cavaratta describes:

• Breaking down data silos between analytics groups within Ford;
• Understanding how drivers are using electric cars, based on opt-in telemetry data from the cars themselves; and
• Supporting the development of features in vehicles by looking at social media chatter about them. (He gives a great example related to the "three-blink" turn signal in the Ford Fiesta.)

You can read more about Ford's use of data science and R at the link below, or explore more companies using R.

Dataconomy: How Ford Uses Data Science: Past, Present and Future (via @JulianHi)

Partying R Style with Sqor Sports, R on Azure, and data.table

by Joseph Rickert

We usually have a pretty good time at the monthly Bay Area useR Group (BARUG) meetings, but this month's meeting was a bit more of a party than usual. The very well connected PR team at Sqor Sports, our host company for the evening, secured San Francisco's tres trendy 111 Minna Gallery for the venue. There was a full bar, house music for the networking portion of the meeting, gourmet grilled cheese sandwiches complements of Revolution Analytics and drama — Matt Dowle, one of our speakers, was on a flight that was late getting in from London. 

Oh! and yes, there were three very engaging presentations — well worth standing around in the dark.

First up was Noah Gift, CTO or Sqor, a company with a mission to take sports marketing to a whole new level. They are creating a marketplace for athletes to build and promote their digital brands. Noah described how devilishly difficult it is to gather, clean and prepare the data. Correctly labeling social media data from several sources generated by different athletes with the same name poses a number of vexing challenges. 

One surprising aspect of the technology that Sqor is developing is what they call an Erlang to R bridge the replaces many tasks they formerly accomplished with Python. Noah indicated that they planning on placing this code in open source.

Below is a plot from Noah's presentation showing predictions from their R based machine learning algorithms.

Our second speaker was Stephen Elston who gave a virtuoso, live demo on using R on the Microsoft Azure Machine Learning cloud platform. Steve glided between the Azure workflow interface and running R scripts. He showed how to manipulate and transform data in both environments, go back and forth to run models in both Azure and R and visualize results in R. Slides for Steve’s talk are available as is some R code on Steve's github site. Studying the scripts will give you an idea of the features he presented.

Finally, just in from London, and still lucid at what would have been 4AM his time, Matt Dowle walked through a summary of new features of data.table v1.9.4 and v1.9.5. There were several data.table users present, and Matt made a few new converts with a series of impressively fast benchmarks against base R. In one demo, Matt showed data.table's forder() taking only 17 seconds to sort 40 million random numerics, a task that took R 7 minutes. According to Matt, the trick for getting this kind of performance is data.table's C-based implementation of radix sorting which works on numeric, character and integer types, with no range restrictions (recall that base::sort.list(...,method="radix") is limited to integers with range < 100,000).  

data.table's radix sorting, which scales linearly i.e. below the O(n log n) bound for comparison sorts, is based on two papers: one by Terdiman and the other by Herf. However, where both of these papers use the least significant digit, data.table uses the most significant digit to improve cache efficiency.

Matt also demonstrated data.table's new automatic indexes (You can now use == in i and data.table will automatically build a secondary key) as well as using dplyr syntax with data.table. Matt emphasized that this flexibility shows the power of R's object oriented design. Matt also claimed that both Python's pandas and the dplyr for R made the wrong choices in using hashing. Instead of hashing, data.table uses fast sorting based on the sort order vector which is an index in data.table

For more benchmark information be sure to visit Matt's github site. If you are new to data.table, I recommend starting with Matt's 2014 useR presentation which explains some the ideas underlying data.table as well as providing an introduction. 

The three types of Reddit posts, and how they make it to the front page

Todd Schneider's blog post on solving the traveling salesman problem with R hit the front page of reddit.com. This is a big deal: front-page placement on the popular social news site can drive a ton of traffic (in Todd's case, 1.3 million pageviews). But what factors determine which of reddit's contributed links make it to the front page? (There are 25 front-page slots, but more than 100,000 reddit posts on an average day.)

Todd set out to answer this question using the statistical language R, and reported his results on Mashable. He collected 6 weeks of data including 1.2 million rankings for about 15,000 posts, and looked for commonalities amongst those posts that made the top 25.

Now, you might expect that a post's front page ranking is determined by its score (the number of times it has been "liked" by a reddit user, most likely after having seen it in the "subreddit" special topic area where it was posted), and how long since it was posted (reddit's front page generally contains recent posts). But it turns out that not all subreddits are treated equally. Todd discovered that there are three different types of subreddits when it comes to how posts are promoted to the front page:

• "Viral Candy" subreddits like funny, gifs and todayilearned. Posts from this category dominate page one.
• "Page Two" subreddits", which includes Documentaries, Fitness and personalfinance. As the name suggests, posts in these subreddits almost never make it to page 1, but are often promoted to page 2.
• "The Rest", which includes food, LifeProTips, and sports. Todd's post was in this category, in the subreddit dataisbeautiful. Posts in these subreddits make a small but significant fraction of page 1 posts.

It seems that reddit's front page (and pages 2, 3 and 4 which follow) follow a well-defined mix of posts from each of the three categories, as you can see in the chart below:

Starting from the left of the chart above, you can see the #1 post (on page 1) is from one of the "Viral Candy" subreddits about 97% of the time, but that a "The Rest" post does occasionally make top billing. By contrast, posts from the "Page Two" subreddits almost never appear above #10, but dominate page two (ranks 26-50). There's a pretty consistent mix on pages 3 and 4: about 65% "viral candy", about 15% "page twos" and about 25% "the rest".

As for post scores, Todd noted that posts from "Viral Candy" and "The Rest" subreddits need high scores to get on page 1: about 3500-4500 and 3000-4000 respectively for the top slot. By contrast, posts in "Page 2" reddits only need scores in the 500-1500 range to hit the lower ranks of Page 1 (but are much more likely to appear on Page 2).

If you're interested in the details of what gets a post on reddit's front page, Todd's blog post has lots more information. And if you're an R user and want to do a similar analysis, Todd's data and R code are available on github.

Todd W Schneider: The reddit Front Page is Not a Meritocracy

Introduction to Revolution R Enterprise For Big Data Analytics on datacamp.com

by Jeremy Reynolds
Senior R Trainer, Revolution Analytics

Last week, Revolution Analytics released its first massive open, online course through a partnership with datacamp.com: Introduction to Revolution R Enterprise for Big Data Analytics. You can sign up for the free course here.

This course provides a look at some of the tools provided by the RevoScaleR package that ships with Revolution R Enterprise. The course and the interactive training framework provided by the platform allow you to get a feel for how you can manipulate, visualize, and analyze large datasets with RevoScaleR.

 

There are four “chapters” within the course:

• Chapter 1 introduces the RevoScaleR package. Within the chapter, we discuss the challenges associated with big data and how the functions and algorithms in RevoScaleR address them. We walk through an example in which we demonstrate the use of several core RevoScaleR functions, and we provide exercises in which you can use the RevoScaleR to create your first linear model on big data.
• Chapter 2 provides details for some of the RevoScaleR functions used to explore large datasets. We demonstrate how you can use these functions to summarize, cross-tabulate, and visualize variables in large datasets, and we provide hands-on exercises where you can practice with them.
• Chapter 3 covers the RevoScaleR functions used to manipulate and transform large datasets. We demonstrate how you can use these functions to perform simple and complex transformations. We provide a set of interactive exercises that allow you to practice creating transformed variables and to explore how chunking algorithms impact the ways in which you need to process data.
• Chapter 4 concentrates on the analysis of larte data sets. We demonstrate how to use RevoScaleR functions to build statistical and machine learning models on large data sets. Here, we cover linear and logistic regression, k-means clustering, and decision tree estimation. For each kind of analysis, there are hands-on exercises in which you can explore some of the flexibility and power associated with the functions.

We are very excited about our partnership with datacamp.com — the platform provides a unique, hands-on training environment in which you can practice in a live R environment, and you can “learn by doing.” We are diligently working to create more extensive content, including courses on the Fundamentals of the R Programming Language, Introductory Statistics with Revolution R Enterprise, Predictive Modeling, and Advanced R Programming. Stay tuned for more!

DataCamp: Introduction to Revolution R Enterprise for Big Data Analytics