Revolutions

Butler Analytics: Real Analysts use R

In an overview of several predictive analytics platforms (including SPSS, Oracle and SAS), Butler Analytics offers this 4.5/5 star review of Revolution R Enterprise:

Real analysts use R – well it sounds a bit macho, but actually there is some truth in it. R is the most widely used, and arguably the most powerful analysis software on the planet. Revolution Analytics has taken this Open Source wild child and turned it into something the enterprise can use with relevant support, training and enhanced productivity.

Revolution R Enterprise is built on open source R and has been enhanced for performance, productivity (through visual tools), and integration with enterprise data sources – and particularly Apache Hadoop for big data applications. Support and training services are bundled on top of the technology – something most organisations will require.

A community edition of Revolution R is available for free. It doesn’t come with the visual tools or database interoperability, but it is faster than the Open Source version.

Many large corporations use Revolution R Enterprise – it’s a bedrock of their analytical activity. Unlike more proprietary offerings you are unlikely to reach any dead ends using R – but you have to know what you are doing.

You can find the full review and reviews of nine other predictive analytics platforms at the link below.

Butler Analytics: 10 Enterprise Predictive Analytics Platforms Compared 

 

A great example of Simpson's Paradox: US median wage decline

Since 2000, the median US wage has risen about 1%, adjusted for inflation.

But over the same period, the median wage for:

• high school dropouts,
• high school graduates with no college education,
• people with some college education, and
• people with Bachelor’s or higher degrees

have all decreased. In other words, within every educational subgroup, the median wage is lower now than it was in 2000.

How can both things be true: overall wages have risen, but wages within every subgroup have fallen? This is a great example of Simpson's Paradox. In this particular case, the explanation lies in the changing educational profile of the workforce over the past 13 years: there are now many more college graduates (who get higher-paying jobs) than there were in 2000, but wages for college graduates collectively have fallen at a much slower rate (down 1.2%) than for those of lower educational attainment (whose wages have fallen precipitously, down 7.9% for high school dropouts). The growth in the proportion of college graduates swamps the wage decline for specific groups.

Simpson's Paradox isn't really a paradox at all, it just reflects the fact that your perception of events can change depending on your viewpoint. From the points of view of an economist looking at the national economy, the "headline" rate of overall median wages has slightly increased, largely due to changes in demographics. But from the point of view of a typical individual, wages have declined. The New Statesman says that "it's more accurate to say that wages have fallen in the last thirty years even though the headline figure shows otherwise", but it all depends on your perspective.

Tips for speeding up R with byte compilation

A byte-compiler for R code — which can improve the execution performance of R functions — was introduced in R 2.13.0, and was automatically applied to the bundled packages in R 2.14.0. Drew Dimmery provides some good advice for identifying targets amongst your own R functions for compilation:

I have some function that will be repeatedly executed n times. You’ll want to make n large enough that it takes some fairly significant chunk of time — like maybe 20 seconds or so. You will very likely see an improvement through compilation. The first replicate will be faster when uncompiled, but every subsequent iteration will tend to be faster through compilation. If you’re using a lot of external functions in this test function, don’t expect miraculous speed increases.

Drew also provides instructions to set environment variables for R to automatically compile packages at the highest level of optimization, which you can find at the link below.

Drew Dimmery: JIT compilation in R (via Pablo Barberá)

Because it's Friday: Get yer red hots

Given the summer heat, I thought I'd share something cool this week. Here's what happens when a red-hot nickel ball bearing meets a block of ice:

 

Make sure you've got the audio on: the sound is kind of terrifying. And if you put it in a glass of water, it almost sounds like it screams.

Keep cool and enjoy your weekend. See you next week!

Predictive Analytics: What Not to Do

If you're involved in any projects that involve the implementation of statistical models or data analysis in general, you should read ComputerWorld's excellent article, 12 Predictive Analytics Screw-Ups. It's an excellent collection of common mistakes that can occur, and their outcomes (or lack of outcomes) in some real-world examples. While some of the mistakes seem obvious in retrospect (using predictors that are just transformations of the outcome, failing to clean data, fitting nonsensical models), they can be all easy to make in practice.

Learning from what goes wrong is an excellent way of avoiding future mistakes, but too often we only hear about the sanitized successes. Check out the article at the link below.

ComputerWorld: 12 Predictive Analytics Screw-Ups

Revolution Analytics Supports the R Community

by Joseph Rickert

Early on, Revolution Analytics realized that R is more than just a tool for statistical computing — it is also the culture that has grown up around the use of the tool. The R culture is open and inclusive, competitive but also nourishing. There is a strong sense of community that encourages contribution and growth. We very much value R’s culture and community and we are doing our best to contribute. At UserR 2013 we displayed a poster that highlights some of our contributions.

Revolution Analytics sponsors R themed conferences such as useR! and we provide financial support to local R user groups around the world.  We believe in the “live event” and face-to-face encounters. We think that the personal bonds that are forged at these meetings are a great strength of R. In addition to providing a sense of occasion, they make possible the chance meeting,  the serendipitous exchange of ideas, and occasionally the opportunity for a kind word to encourage an R beginner.

On a daily basis we do our best to keep these relationships alive, by listing R user groups on our website, tracking community events, reporting on new developments and recognizing accomplishments in our blog.

We also look outward from the R community to the world of business, finance, industry and government and promote the use of R for analytics and data analyses of all kinds. We are R evangelists and work hard to explain and promote open source R to the press, to social media and to the general population. We do our best to present R, and the R community in a way that we hope all R developers and contributors will be pleased with.

Finally, we look for ways to make technical contributions to R itself. Revolution developers have authored and continue to maintain a number of open-source parallel computing packages on CRAN.  More recently, we have funded early stage R research, such as the beginning work on bigvis, and the RHadoop project which we hope will help to help establish R as a fundamental component of “Big Data” analytics.

We would very much like to hear your ideas about how you think Revolution Analytics might continue contribute to the R community in the future — let us know your thoughts in the comments.

What is R? A new video on the history, community and applications of R

We meet a lot of R users on our travels, and something we often hear from them is that while they're doing amazing things with R (incredible data visualizations, statistical analysis, and data science applications), their supervisors or peers may not know that the R language is involved, or that others could benefit from using it. It would be great, they tell us, to have something short and sweet to share that describes what makes R so special. 

So we decided to write a script, hire an animator and a voice-over artist, and create a 90-second video all about the history, community and applications of R for everyone to share. Here it is:

 

This video is free to share (under a Creative Commons license), and you can even download the source MP4 video file for offline viewing or to remix. We designed it for anyone who's unfamiliar with R but would be interested to know what it is and what it can do. Let us know what you think in the comments, and please share the link www.revolutionanalytics.com/what-is-r to spread the word about R.

Revolution Analytics: What is R?

Using survival models for marketing attribution

by Andrie de Vries

Prior to joining Revolution Analytics in March this year, I spent several years in the field of market research and survey analytics. During this period, I spent a few months consulting to a digital marketing agency based in London. My role was to help build their capability in building customer surveys and integrating these into social media sites, including Facebook and Twitter.

As a media agency, a substantial part of the business activity was to place client advertisements on websites, including social media. The worldwide online advertising market is estimated at $100B per year. This is a fast moving, competitive and innovative industry, where tiny improvements in the clickthrough rate (CTR) and conversion rate can lead to large improvements in the effectiveness of campaigns.

Given the large amounts of online advertising spend, and the competitive nature of the industry, it is increasingly important to make sure each advertising dollar is spent in the right way. In the marketing world, this question of where to spend money is often referred to as attribution, since the question is how to attribute value. In other words, how much did each interaction with an advertisement contribute to sales conversion?

But for many media agencies it still is a challenge to attribute value, and as a result it's not uncommon for companies to use the easiest of all models, where all value is attributed to the last click.

There has to be a better way, and when I was asked to build a statistical model for attribution, my immediate thought was to make use of logistic regression. My thought was that one should be able to isolate a clickstream (or impression stream) for people who did convert (i.e. click through the final ad) vs. those who didn't.

It turns out this is not a trivial problem. In my case, I only had information about the converters, and none about the non-converters. So a logistic regression was not appropriate. This baffled me, until I saw a reference to how Datasong (previously Upstream) used survival models for marketing attribution. This made me search for more information on the topic, and I came across a PhD thesis by John Chandler-Pepelnjak, published in 2010. His thesis investigated Modelling conversions in online advertising.

With my interest piqued, I investigated how to use survival analysis for marketing attribution.

During the last week, I presented (summarised and anonymized) results of my work at three occasions:

• The annual R user conference, UseR!2013 in Albacete, Spain.
• The London R user group, LondonR.
• As a webinar at Revolution Analytics (I've embedded the slides below, and you can watch a replay of the webinar here).

It was very satisfiying to share my ideas with people, and this lead to many new ideas and questions. In a follow-up blog post I plan to highlight some questions, insights and novel applications of survival analysis outside the traditional fields of medicine and engineering.

Bike sharing in 100 cities

Many cities around the world have bike sharing programs: pick up a bike at a docking station, ride it across town and drop it off at another session, and just pay for the time you use. (Even Albacete, the Spanish college town hosting last month's UseR conference, had one.) Most of these systems provide open data feeds of bike availability, and the data is available for all cities via the CityBikes API. Ramnath Vaidyanathan used this API to create an on-line application showing the real-time status of bike availability in over 100 cities. Here's London, for example:

Red dots indicate bikeshare stations with zero available bikes; green dots show stations with the most availability. (I took the snapshot above around midnight London time: it looks like all the city-center bikes have been taken to the outlying stations by commuters, and the overnight redistribution hasn't yet taken place.)

Ramnath used an R language script to download the data from the JSON API (using the httr package), and created the maps using the rCharts packages (which provides an interface to Leaflet maps). The interactive application, with city selection, pan and zoom, and pop-up detail for each station, was created using RStudio's Shiny package for R. It's easy to imagine other applications for this data, for example using statistical forecasting to optimize the overnight redistribution, or to determine the most popular bike routes in a city.

Click the map above to play around with the application yourself, or follow the link below for more details from Ramnath on how the application was created.

Ramnath Vaidyanathan: Visualizing Bike Sharing Networks

Because it's Friday: Reunion with Christian the Lion

I just got back from a three-week trip overseas, and one of my favourite things about coming home after being away is the super-enthusiastic welcome I get from our Jack Russels, Max and Easy. It reminded me of the clip below. Apparently it's been seen by 60 million people since it was first uploaded in 2008, but if you haven't seen it before and you're a pet owner, you'll immediately recognize the reaction of Christian the Lion as he's reunited with his former caretakers John and Ace a year after being released in Africa:

 

That's all for this week. Enjoy your weekend!