Revolutions

Steve Miller on R at Predictive Analytics World

At the Information Management blog, Steve Miller has provided two great reviews (here and here) of last week's Predictive Analytics World conference, including a recap of the Bay Area User's Group meeting featuring John Chambers. (My personal highlight from John's talk? A photograph of the very first sketch of what was to become the S system, which ultimately  begat the R language.) But I also wanted to point you to Steve's review of Mike Driscoll's jaw-dropping talk The Social Effect: Predicting Telecom Customer Churn with Call Data:

Mike Driscoll's: The Social Effect: Predicting Telecom Customer Churn with Call Data, was a good illustration of predictive analytics in a larger data warehousing and BI context. Driscoll and his team analyzed billions of calls, millions of records and thousands of defectors from a Greenplum DW looking for predictors of churn. Driscoll's a big proponent of the open source R Project for Statistical Computing to support his work flow of data munge, data model, and data visualize. And with a Ph.D. in Bioinformatics, he often thinks like an epidemiologist, in this case looking for indications of contagious churn behavior. Using several social network analysis packages available in R, Driscoll's team appears to have found that churn in an individual's social network of calling accounts in a given month is likely to lead to more churn in subsequent months, a clear indication of a network effect. That contagion is overwhelmingly the strongest signal the team found from the data. A next step is to work with marketing to intervene on early network churn with email campaigns to minimize losses from the affected networks. I'd love to see the results from those experiments.

Mike's talk was jaw-dropping for me for two reasons. Firstly, in innovation: using a caller's social network (defined by the people they call the most, information drawn from the call-log data) was an elegent and powerful way to predict probability of switching to another network. It seems intuitive -- if your friends switch to another provider, you're likely to, as well -- and it's always good to see intuition borne out by data. And secondly, in scale: we're talking about 10Gb of data here, analyzed in just a few hours using R. Not too long ago that type of computation would be reserved for high-tech computing grids and expensive software; these days, it's amazing what you can do with a hefty 64-bit workstation and open-source tools. Unfortunately Mike's slides aren't available just now (some great social network charts in there), but if they do become available I'll let you know.

Information Management: Predictive Analytics World – Take 2

Because it's Friday: Visualizing an email chain

We've all been there: someone sends an email to a mailing list with a Reply-To directing responses back to the mailing list. Before long, someone replies (unwittingly, to everyone) to ask to be taken of the list. And before long, the entire affair devolves into an endless cycle of requests to unsubscribe and pleas to stop mailing the entire list. This process is visualized beautifully in this timeline of emails to the Caltrans Performance Measurement System mailing list (click to see the enlarged original on FlowingData):

 Some of the classic r-help threads could also benefit from such a treatment.

FlowingData: Data Underload #8 – Unsolicited

http://flowingdata.com/2010/02/08/data-underload-8-unsolicited/

Solving Sudoku with Simulated Annealing

How long would it take you to solve this devlishly hard Sudoku puzzle (from Le Monde)?

 

You could do it the old-fashioned way -- with a pencil -- but Xi'an decided to solve it by programming a simulated annealing solver in R. The algorithm works by first guessing a solution at random -- filling in the empty cells above with random digits between 1 and 9. Then it "scores" this solution by counting the number of digits duplicated in all the rows, columns and blocks. Next it evaluates a number of candidate new solutions by tweaking one of the free digits, and scores those. The algorithm then selects one of the candidate solutions at random for the next step, weighted by the change in the score.

The code to do all this , which you can use to solve your own Soduku puzzles if, say, you've lost your pencil, is at Xi'an's blog.

Xi'an's og: Sudoku via simulated annealing

Object types in R: The fundamentals

If you're a self-taught R programmer, you've probably grappled with the different kinds of objects you can use in the language. When should you use a list instead of a vector? What's the difference between a factor and character vector? These questions are easier to answer when you have some of the basics of R's object types down pat, and Chris Bare lays out the fundamentals quite nicely in his blog post The R Type System. An excerpt:

Because the purpose of R is programming with data, it has some fairly sophisticated tools to represent and manipulate data. First off, the basic unit of data in R is the vector. Even a single integer is represented as a vector of length 1. All elements in an atomic vector are of the same type. The sizes of integers and doubles are implementation dependent. Generic vectors, or lists, hold elements of varying types and can be nested to create compound data structures, as in Lisp-like languages.

He goes on from with useful descriptions and examples of matrices, arrays, data frames, factors and more. Well worth checking out if you want to understand how R's object types tick.

Digithead's Lab Notebook: The R type system

Reminder: useR! 2010 abstracts due Monday

Don't forget, if you're planning to attend the R user conference useR! 2010 and are going to present a talk (and if not, why not?), abstracts are due for submission this coming Monday, March 1. That's also the deadline for early-bird registrations, so if you haven't registered yet, now is the time.

useR! 2010: The R User Conference

Slides from "R Productivity Environment" webinar

Thanks to everyone who attended for the great turnout at this morning's live webinar, 7 Ways to Increase your R Productivity. I really appreciate all the feedback and questions, seems like a lot of people are interested in a code editing and debugging environment for R. 

If you missed the webinar and want to learn about REvolution R Enterprise and the R Productivity Environment, the replay will be live at the link below tomorrow. The slides are there for download now.

REvolution Computing: Webinar: 7 Ways to Increase Your R Productivity

Speeding up R code: A case study

On his Psychology and Statistics blog, Jeromy Anglim tells how he was analyzing some data from a skill acquisition experiment. Needing to run a custom R function across 1.3 million data points, Jeromy estimated it would take several hours for the computation to complete. So, Jeromy set out to optimise the code.

First, he used the Rprof function, which inspects your R functions as they run, and counts the amount of time spent in each sub-function. This is a useful tool to identify the parts of your functions that are ripe for optimisation, and in this case (with some help from the system.time function to time a specific section of the code) he learned that most of the time wasn't taken performing actual calculations: most time was actually spent selecting the subset of the data to analyze!

And thus a solution was born: rather than repeatedly selecting from the large data frame in an iterative loop, he instead split the data frame into its constituent parts once, and then looped over the parts. This reduced the analysis time from hours down to just a couple of minutes. As the end of his case study, Jeromy shares some valuable lessons learned about optimising R functions:

• R is very fast most of the time.
• A single slow command can be the cause of a slow analysis.
• system.time is a very useful function.
• Optimisation can proceed from theory or from experimentation. 
• Optimisation proceeds from diagnosing the cause of the problem to exploring solutions.
• Optimisation is about orders of magnitude. Focus on saving hours before tackling saving seconds.

Jeromy Anglim's Blog: Psychology & Statistics: A Case Study in Optimising Code in R

Where did all the bankers go?

When Lehman Brothers, Bear Stearns and Merrill Lynch went kablooie in the financial crisis, what happened to all their employees? Thanks to the magic of LinkedIn data, their Chief Scientist DJ Patil can answer that question: they went to the surviving banks:

 

It's a great, if tantalizingly incomplete visualization -- I'd love to see this with "Other (non-bank) employers" (even if it is just a "handful" of the set, as Patil mentions) and "Unemployed" as additional leaf nodes.

LinkedIn blog: Where did all the people go from the collapsed financial institutions?

How to call C++ from R with ease

At last night's meeting of the ACM Student Chapter at the University of Chicago, DIrk Eddelbuettel gave an invited guest lecture, "Programming with Data: Using and Extending R". I wasn't there myself, but Dirk has already posted his slides, and they're a treat. After a backgrounder on R itself (BTW, I'm flattered he referenced my Introduction to R talk here), Dirk dives into some case studies on various ways to call C, C++ and Fortran code from R. Most of the time is spent on the Rcpp interface (by Dirk and Romain François), which greatly simplifies the process of writing C++ code to be called from R. You can even "inline" C++ code directly in your R program, freeing you from the need to write complete C++ source files, and compile and link the code yourself. Dirk presents some impressive performance gains from using this technique. 

[Update 9:42am: Attribution of Rcpp]

Dirk Eddelbuettel / Thinking Inside the Box: U of C ACM talk

Newspaper flubs probability calculation

That headline's right up there with "Dog Bites Man" for shock value, but the Daily Express in the UK isn't one to let mere probability stand in the way of a sensational headline like "Mum beats odds of 50 million to one to have 3 babies on same date". As Ben Goldacre helpfully explains, the probability is actually a modest 133,000-to-one (50M-to-one is the probability of having four babies all on the same date, or three babies all on a prespecified date). The article was published over a week ago, but there's no correction in the online content, other than a comment from an observant reader. Another one for the "Film at 11" vault, like the Bulgarian Lottery coincidence.

Bad Science: Guns don’t kill people, puppies do