Revolutions

Because it's Friday: Evolution of a research paper about Reddit

Computer Science PhD student Tim Weninger wrote a 10-page paper for the World Wide Web conference looking at how Reddit users interact on the discussion pages of the social news site. During the process, he saved 463 revisions of the paper in a source-code control system. Then, he wrote a computer program to animate each revision of the paper. The result: a film showing how a typical research paper evolves from an outline to a complete thesis:

 

Note how the paper starts with just a title, outline and figures (which look as though they were generated using R). Then the narrative is added and edited. The reference section (automatically generated by LaTeX) grows over time as citations are added to the narrative. The title of the paper also goes though several iterations over its lifetime, before settling on "An Exploration of Discussion Threads in Social News Sites: A Case Study of the Reddit Community".

Tim shares how he made the video in the YouTube comments:

I wrote a computer program that converted the pdf files into images with the page layout, and then used another program called ffmpeg to stitch the pages together at 10 frames per second.

It's a lovely visualization of the creative process. I'd love to see a similar video for some of my own writings, but sadly I'm not as diligent about checking in documents as I am for code.

Real-Time Predictive Analytics with Big Data, and R

Can R be used for real-time applications? Absolutely! The key is in setting up an technology stack that can support real-time interactions with models developed in R ... and a clear understanding of what "real-time" really means, and its implications in the context of Big Data.

I explained how this works in yesterday's webinar, Real-Time Predictive Analytics with Big Data, From Deployment to Production. I described the four layers of the analytics stack above, and outlined a process for deploying real-time predictive analytics applications based on R:

1. Data Distillation
2. Model development
3. Model validation and deployment
4. Real-time model scoring
5. Model refresh

At the end of the presentation I also included what I hope are more useful definitions of "real time" and "big data" than the buzz-words alone. I've embedded the video replay; you can also download it and the slides from the Revolution Analytics webinar page linked below.

 

Revolution Analytics Webinars:  Real-Time Predictive Analytics with Big Data, From Deployment to Production

The tools in an R package developer's toolbox

Yihui Xie is the creator of several popular R packages, including knitr, animation and cranvas. In an interview with The Setup, he shares some of the software and hardware he uses in his data-to-day work, including (of course) R:

For programming and data analysis, I primarily use R since I'm a statistician. I have created a bunch of R packages including animation, knitr, formatR, Rd2roxygen, R2SWF, fun and cranvas, etc. I use other R packages like ggplot2, gWidgets, roxygen2, Shiny and XML. Emacs was my editor before I switched to RStudio; for small tasks, I just use the default editor gedit. I also use sed, awk, grep and shell scripts frequently.

(Data Scientist John Myles White gave a similar interview back in July.) Yihui's interview also includes the above photo of him sporting a Revolution Analytics "I ♥ R" T-shirt — if you'd like one of your own, check with your local R user group!

The Setup: Yihui Xie (via Karthik Ram)

Hadley's guide to high-performance R with Rcpp

Hadley Wickham has written a comprehensive tutorial for the Rcpp package, which makes it easy to create C++ code embedded in R programs. Hadley explains why you might want to do this in the introduction:

Sometimes R code just isn't fast enough - you've used profiling to find the bottleneck, but there's simply no way to make the code any faster. This chapter is the answer to that problem. You'll learn how to rewrite key functions in C++ to get much better performance, while not taking too much longer to write. The key to this magic is Rcpp, a fantastic tool written by Dirk Eddelbuettel and Romain Francois (with key contribution by Doug Bates, John Chambers and JJ Allaire), that makes it dead simple to connect C++ to R. It is possible to write high performance code in C or Fortran. This might produce faster code than C++ (but probably not), but it will take you much much longer to write. Without Rcpp, you must sacrifice many helpful wrappers and master the complex C internals of R yourself. Rcpp is currently the best balance between speed and convenience, and any other approach will be much more painful.

No knowledge of C++ is required, so if you have some R code — especially tight loops — that you need to speed up, you should definitely check it out the whole tutorial.

This guide comes from a book on writing reproducible code that Hadley's working on. You can browse the other sections of the book on GitHub — even though the book isn't yet complete there's a wealth of useful information there for any serious R programmer.

Hadley Wickham (github): Rcpp

Using R in the Human Resources department

Ajay Ohri, author of R for Business Analytics, was recently interviewed about using R instead of Excel for human resources management. Human Resources is an increasingly analytics-driven field, with predictive modeling now used to prioritize applicants for interview and to organize workspaces, to give just two examples. As Ajay points out, R gets over the limitations of spreadsheets:

There is a lot of data out there and it’s stored in different formats. Spreadsheets have their uses but they’re limited in what they can do. The spreadsheet is bad when getting over 5000 or 10000 rows – it slows down. It’s just not designed for that. It was designed for much higher levels of interaction.

In the business world we really don’t need to know every row of data, we need to summarise it, we need to visualise it and put it into a powerpoint to show to colleagues or clients.

In a similar vein, the social network Facebook uses R for a variety of internal human-resources analytics applications. (The fact that Facebook employees use Facebook to collaborate internally provides a wealth of data to work with.) These applications include:

• Selecting peer reviewers during performance review season
• Setting up optimally constructed teams, and even optimizing seating charts to improve interactions and productivity
• Automatically filtering internal news feeds of employee content
• Suggesting new colleague interactions, and giving managers insight into employee interactions.

You can read more about how Facebook uses R for HR applications in Eric Sun's presentation (PDF), Criss-Crossing the Org Chart: Predicting Colleague Interactions with R.

Applications open for 2013 John M Chambers Statistical Software Award

In 1998, John Chambers won the ACM Software Systems award for the S language, which the ACM heralded as having "forever altered how people analyze, visualize, and manipulate data". The R language is "not unlike S", and John continues pushing the frontiers of statistical computing as a member of the R Core Group. Ang his legacy continues in another way: he donated his ACM prize to a fund for the John M. Chambers Statistical Software Award. Recipients of the award include authors of several R packages, including Michael Kane (for the bigmemory package), Yihui Xie (animation), Hadley Wickham (reshape and ggplot), and Deepayan Sarkar (for lattice, now a part of standard R). 

Applications are now open to individuals or teams for the 2013 prize, which offers $1000 in cash and a travel allowance to the JSM conference in Montreal, where the award will be presented. The official announcement hasn't yet been updated on the web, so I've included the announcement after the jump below.

Happy Thanksgiving!

We're taking a break today and tomorrow at the blog for the Thanksgiving holiday. (We'll be back to usual programming on Monday.) Wherever you are in the world, please enjoy the holiday and/or the weekend with your families. Happy Thanksgiving from everyone at Revolution Analytics.

Upcoming Webinar: Real-time, big-data analytics

A quick heads-up that I'll be presenting another brand-new webinar on Thursday next week (November 29). In Real-time Big Data Analytics: From Deployment to Production, I'll review the process of making predictive models work in real-live operational environments. I'll also tackle those ubiquitous buzz-words "real-time" and "big data", and the fact that they can mean very different things in different parts of the process. Here's the talk description:

Real-time Big Data Analytics: From Deployment to Production

As the Big Data market has evolved, the focus has shifted from data operations (storage, access and processing of data) to data science (understanding, analyzing and forecasting from data). And as new models are developed, organizations need a process for deploying analytics from research into the production environment. In this talk, we'll describe the five stages of real-time analytics deployment:

• Data distillation
• Model development
• Model validation and deployment
• Model refresh
• Real-time model scoring

We'll review the technologies supporting each stage, and how Revolution Analytics software works with the entire analytics stack to bring Big Data analytics to real-time production environments.

I'll be taking live questions at the end of the talk, and you can register your seat at the link below.

Revolution Analytics Webinars: Real-time Big Data Analytics: From Deployment to Production

R User Conference in Spain: Call for Tutorials

I'm really looking forward to useR! 2013 (the international conference for R users), and not just because it's being held in Spain next year (July 10-12). The program is already coming together, with a great lineup of invited speakers, including R-core member Duncan Murdoch and prolific package authoR Hadley Wickham. You too can be part of the program, by presenting a tutorial: 

We invite R users to submit proposals for three hour tutorials on special topics regarding R. The proposals should give a brief description of the tutorial, including goals, detailed outline, justification of why the tutorial is important, background knowledge required and potential attendees. The proposals should be sent before December 15, 2012 to useR-2013_at_R-project.org.

If you'd prefer to give a talk, the call for abstracts will be announced soon.

University of Castilla-La Mancha: The R User Conference 2013

Data Scientist among best new jobs in USA

According to CNN Money. Video Game Designer, Social Media Manager and jobs in the green energy sector also made the list. The Harvard Business Review agrees: last month, they called 'Data Scientist' the "sexiest job of the 21st century".